Why the retro scanner's 46 never meant 46 failures

The retro scanner reported 46 recurring failures. Its next run reported 33. Nothing in the underlying work had improved between those two numbers. The counter had been wrong.

We use retros to turn repeated agent and workflow failures into corrections. The loop only helps if the measurement answers a narrow question: what did we actually have to fix because the same problem kept returning? On this pass, the scanner was answering a different question without saying so. It was counting routine skill boilerplate changes as corrections.

That made the report worse than useless. It gave us a precise number that looked like evidence, while mixing maintenance work with the failures we were trying to understand.

The number that looked plausible

The bug did not announce itself as a crash or an empty result. The scanner worked. It found changes. It produced a total. A total of 46 even had the uncomfortable shape of a real result, large enough to suggest our process was leaking the same mistakes over and over.

A change to standard skill boilerplate is not necessarily a correction. It can be routine maintenance of the instructions that support the system. Counting it as a response to a recurring failure means the metric counts activity instead of learning.

That distinction is easy to blur when a retro system starts from changed material. A change tells you something was edited. It cannot, by itself, tell you why. The scanner had collapsed two distinct states into one bucket: a change made after a recurring failure, which is evidence of a pattern worth tracking, and routine skill boilerplate, which is maintenance and evidence of nothing.

So the 46 was not a count of failures. It was a count of things that satisfied a bad definition.

Checking the gauge before fixing the process

The retro-driven pass still found six real recurring failure patterns to fix. That work was valuable. It also exposed the scanner problem, because the reported total did not cleanly describe the corrections we could account for.

The diagnostic path was short. We held the actual work constant and inspected the category the scanner assigned to each hit. The routine boilerplate was inflating the same counter as the real corrections. The failure was in classification, and the retro itself was fine.

Our first move was the easy one: leave the rule alone and treat 46 as a rough directional signal. That kept the scanner running, and it also threw away the reason for having one. A noisy total cannot tell us whether a failure pattern is returning, whether the correction rate is falling, or whether a batch of maintenance just made the graph jump.

We also could have lowered the threshold or hidden the boilerplate after the fact. Neither repairs the definition. They only make the number look more comfortable. The fix had to separate an actual correction from routine skill boilerplate at the point where the scanner decides what belongs in the report.

After that change, the same retrospective read 33. The difference of 13 was the size of the measurement error.

Reading 46 to 33 correctly

A drop from 46 to 33 normally sounds like improvement. Reading it that way here would recreate the bug in prose. We had not eliminated 13 recurring failures. We had stopped attributing 13 routine boilerplate changes to failures in the first place.

That gives the remaining 33 more weight. They are closer to the set of changes that should drive follow-up work, revisions to the operating skills, and another retro later. We logged two follow-up items from the pass, including the scanner defect itself, because a reporting bug inside the feedback loop is production work. It changes what we believe about the system.

A score needs a defensible noun. Before we use a number to tell us what keeps going wrong, we ask what each row in that number actually means. A scanner that counts boilerplate as learning adds noise, and it also teaches the system a false story about where the work is.

When a report gives you a failure count, find out what each row represents before you act on it. A drop after a definition change is a measurement fix and not an improvement.

dxdev LESSON · paste into your AI coding agent
LESSON: A metric that counts things that changed is not the same as a metric that counts things that were a correction to a recurring failure
SOURCE: dxdev.com/blog/the-gauge-that-lied-about-itself
WHAT HAPPENED: A retrospective scanner reported 46 recurring failures on one run and 33 on the next, with nothing in the underlying work actually improved in between. The scanner had been silently counting routine skill-boilerplate maintenance edits as corrections alongside genuine fixes for recurring failures, because a file being changed is evidence something was edited, not evidence of why it was edited. Rather than accept 46 as a rough directional signal, or quietly lower a threshold to make the number look more comfortable, neither of which fixes the underlying definition, the classification itself was corrected at the point the scanner decides what belongs in the report, separating a genuine correction from routine maintenance that happened to touch the same material. The resulting 33 was explicitly not read as thirteen fixed failures; it was the size of the measurement error being removed, which made the remaining 33 more trustworthy, not the drop itself.
THE RULE: A plausible, precise-looking number from a self-audit tool is not evidence on its own; before trusting a metric to tell you whether a problem pattern is improving, ask what each unit it counts actually is, a genuine correction versus routine maintenance that merely touched the same files, because a measurement bug inside a feedback loop teaches the system a false story about where the real work is.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Any recurring-metric or trend report, such as a failure count or a correction count, derived from something changed without distinguishing why it changed.
      2. Any metric drop being read as improvement without first checking whether the classification or measurement logic itself changed between the two readings being compared.
      3. Any self-audit or retro tool whose own reporting logic has never been audited with the same rigor as the process it is measuring.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.