Before You Trust an AI Reviewer's Verdict, Grep the One Claim You Can Check

Manus’s review of the memory-truncation draft came back in about a minute: SHIP, three notes attached, and note one told me to soften a sentence that supposedly still read as dramatic. It quoted the line directly: “The promise that it would be read was the lie.” Specific, exact, easy to act on. I almost did.

The draft in question was the second pass on a post about a 200-line memory truncation bug. It carried a v2 changelog at the top, an HTML comment listing exactly what got cut between v1 and v2 before either reviewer saw it again. Line 18 of that comment reads: Cut "the promise... was the lie" drama (both reviewers). Both Manus and Codex had already flagged that phrase once, on v1, and it had already been removed.

So when the same phrase showed up quoted in Manus’s v2 review as something still needing a fix, that was checkable in about five seconds. I ran grep -n "the promise" against the actual draft file. One hit. Line 18. Inside the changelog comment, not the body. Zero matches in prose.

What that one grep bought me

My first instinct was to trust the review, since it was a quoted string, not a vague vibe. I opened the draft and started scanning the body near “The lesson worth keeping” for anything that sounded like it, found nothing close, and almost talked myself into rewriting a nearby sentence anyway on the theory that Manus meant something adjacent and I was just missing the exact match. That would have been an edit against a phantom problem, on a section a separate Codex pass had already scored 8/10 for voice with only two low-severity notes, neither of which was this.

Running the grep instead of guessing took less time than the false edit would have and it settled the question outright: the phrase existed exactly once in the file, and it lived in a comment block explicitly marked strip before publish, describing a change already made. Manus wasn’t reading stale content. It was pattern-matching the literal text of the changelog note I’d told it to ignore (“ignore the HTML change-summary comment and frontmatter”) and reporting the string it found there as if it were live prose. The instruction to skip that block didn’t stop the model from treating its contents as fair game for a quote.

Codex’s review of the same draft, run separately, made the same kind of checkable claim in a different spot: a low-severity note flagging that the frontmatter’s author field still carried my real first name instead of the pseudonymous byline dx. That one held up. Grep confirmed it, and it went in the fix list.

Same review pipeline, same shape of claim, two different outcomes. One quoted string was real and useful. One quoted string was a hallucinated read of a section that was off-limits by design. Both looked identical in the review output: bolded location, quoted text, a severity tag. Nothing about the format told me which one to trust.

The habit that generalizes past this one post: when an LLM reviewer quotes exact text as evidence, that quote is a claim about a file you already have on disk. Grep it before you act on it. It costs nothing, and it is the only step in the whole pipeline that tells you whether the model read your file or read its own idea of your file.

dxdev LESSON · paste into your AI coding agent
LESSON: Grep Every Quoted String In An AI Reviewer's Feedback Against The Actual File
SOURCE: dxdev.com/blog/2026-09-11_verify-the-ai-reviewers-checkable-claims
WHAT HAPPENED: Manus reviewed the v2 draft of a memory-truncation post and flagged a specific quoted line as still reading as dramatic, even though that exact line had already been removed and both reviewers had flagged it once before on v1. Instead of rewriting a nearby sentence to satisfy the note, the author ran grep against the draft file and found the phrase existed only inside an HTML changelog comment marked to be stripped before publish, not in the live prose. A separate Codex review of the same draft made a structurally identical quoted claim -- that the frontmatter author field still had a real name instead of the pseudonym -- and that one checked out under grep and got fixed. Both claims looked identical in format: bolded location, quoted text, severity tag, but one was a hallucinated read of an ignored section and the other was accurate.
THE RULE: When an LLM reviewer quotes exact text as evidence for a claim, treat that quote as a testable assertion about the file on disk, and grep for it before acting, since the review's formatting gives no signal about whether the model read the live content or its own stale idea of it.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. For every reviewer note that quotes exact text, grep that exact string against the current file before editing anything in response to it
      2. If a phrase only matches inside a comment, changelog block, or other section explicitly marked off-limits to the reviewer, treat the note as invalid rather than adjusting nearby prose to compensate
      3. Confirm zero matches in the live body before concluding a flagged phrase was already removed, rather than assuming based on where in the document you expect it to be
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.