--- title: "A three-pass AI code review that kept finding real bugs" canonical: https://dxdev.com/blog/three-pass-ai-code-review-kept-finding-real-bugs/ datePublished: 2026-05-18 --- I sent a feature scope to a read-only AI reviewer with the repo mounted. The first pass came back with six objections, every one citing a file and line I could open and check. I fixed them. The second pass found two more things wrong. I fixed those. The third pass found three more. None of the eleven were hallucinated. By the end I had a design that was actually correct, and an answer to something I'd been skeptical about: re-running a design through an AI reviewer produces compounding signal, not the same vague pushback reworded. Here's the run that convinced me. ## The setup The ticket was a custom-columns feature: custom roster columns for an ASP Classic SaaS that's been running for years. Admins wanted to attach arbitrary extra columns to a team roster (a jersey number, a position, whatever), have them render on the public roster, the admin roster, CSV exports, print views, and sort correctly. The codebase is classic ASP, JScript flavor, deep. The kind of code where the right design depends on how three or four existing subsystems actually behave, not on how you'd build it greenfield. So before writing the implementation I wrote a scope doc and handed it to Codex running at `effort=high` with the production codebase mounted read-only. Not "review this diff." Review this *plan* against the real code. Find where my assumptions don't match what's already there. The three passes ran at roughly 325K, 454K, and 518K tokens. The reviewer graded each one with a verdict: REVISE FIRST, then REVISE AGAIN, then MINOR FIXES. That progression is the whole story. It wasn't stuck. Each pass was materially different from the last because each one was reviewing a different, less-wrong design. ## Pass one: six objections, two that mattered most Two of the six were the kind of thing that would have shipped a quietly broken feature. One was an ID-scheme bug. My plan keyed custom columns as `c1`, `c2`, `c3`. Looked fine. But the reviewer pointed at three real code paths in the league-linking and team-packaging routines and showed that those positional keys silently rebind their meaning when a team gets linked to a league or detached from one. I'd designed the IDs to be stable across a *rename*. The actual requirement was that they stay durable across a change of *owner*, because the ownership operations in those files reshuffle exactly the thing my keys depended on. That's not a nitpick. That's a corruption bug that would only show up after a customer linked their team to a league months later. The other was a prefs-cascade mismatch. My design assumed custom columns would inherit via a nested merge, team values layered over league defaults. The reviewer pointed at the prefs layer in the codebase and showed that the set and get-types helpers are flat, top-level overwrite. There is no nested merge in this system. My sparse-override inheritance model was describing a mechanism that didn't exist. I could verify both in under a minute each because the objections weren't "consider whether ownership changes affect your IDs." They were "here is the file, here are the lines, here is the operation that breaks your assumption." That specificity is what a reviewer that actually read the code gives you. ## Pass two: a collision-math sign-off and one thing still impossible I rewrote the ID scheme to 16-hex-character `c_` IDs and sent it back. Pass two did the collision math instead of hand-waving it. 64 bits of entropy: the move-pair collision probability works out to about 9/2^64, roughly 4.9e-19. Even minting a million IDs, the birthday-style risk is about 2.7e-8. Verdict: operationally fine, use a full GUID if you want it to round to zero. I'll take a reviewer that runs the numbers over one that says "consider collision risk" every time. But the same pass found a feature I'd written into the plan that the codebase simply cannot support. I'd specced a "copy from former league" option. The reviewer pointed at the league-edit handler and showed the code overwrites the parent league in place and never persists the old one. There is no "former league" to copy from. The data isn't kept. Not a bug in my code, a bug in my plan: I'd promised a capability the storage model makes impossible. Better to learn that from a reviewer than from a support ticket. ## Pass three: a render path I'd missed and a syntax error The third pass is where I expected diminishing returns. It found three more real things. One was an under-scoped surface. I'd treated the non-responsive roster as just another render path I could gate with a simple prefix check. The reviewer showed it isn't a sibling template at all. It's a SQL-backed table assembled through a column-builder helper, so a prefix check alone is wrong there. It needed a real column-definition strategy, and I'd have found that out the hard way when the old-layout roster rendered garbage. Another was a flat-out syntax bug in my own pseudocode. I'd written a `case` on a string prefix in a switch. That isn't valid JScript. You can't `case` on a prefix. It has to be a prefix check *before* the switch. Small, but it's the kind of thing that means the author hasn't mentally compiled the code, and the reviewer had. The third pushed back on a migration step I'd called safe. My plan deleted the legacy custom-columns pref whenever a team saved a new-system config. The previous pass had signed off on that. This pass corrected it: both renderers still honor that pref as a full-layout override, so deleting it on every save is a one-way migration, not a neutral cleanup. If that's the behavior I want, the plan has to say so on purpose, not slip it in as housekeeping. After that, MINOR FIXES. The well ran dry at the point where the design was actually right, not before. That's the tell I now trust: when a fresh pass returns only nits, you're done, not when you get tired of asking. ## What shipped Server-side landed on a branch, seven commits, and I verified it live end to end: the custom column renders in the admin roster, the visitor roster, the CSV export, the print view, and sorting by custom-column ID works. The legacy positional-column data clears on save as intended. The client-side work (settings UI, per-player edit popup, sort dropdown) got parked for the next session because it needs a JS build-process discovery I didn't want to rush. But the part that the three passes hardened, the data model and the render contract, shipped correct. ## Related - [Separate Your Build Model From Your Review Model: Codex as Adversarial Reviewer](separate-build-model-from-review-model-codex): the discipline behind using a read-only reviewer against the actual codebase - [Letting a Second AI Review Your Toolkit Design, and Where I Overruled It](second-ai-design-review-and-where-i-overruled-it): a parallel multi-pass review and where reviewer judgment diverged from the author's - [All Seven Reviewers Passed. The Writing Was Worse. Here's What Went Wrong.](reviewer-over-correction-loop-feels-worse-stop-sign): the failure mode when multi-pass review overcorrects without a stop signal - [Seven AI Reviewer Personas With a Cost Ledger, and a Budget Rule That Fires Them](seven-ai-reviewer-personas-cost-ledger-budget-rule): the cost accounting behind deciding when multi-pass review is worth running - [Codex flagged my ONLINE=ON, my own probe said it was fine, and the server proved us both wrong](codex-flagged-online-index-standard-edition): a case where the AI reviewer caught something the author's own verification missed