506 open tickets, 429 candidates, and a verifier that would not agree

Eight of the 54 tickets our classifier called fixed or obsolete came back from an independent verifier with file:line evidence that the work was only partly done. If we had closed on the first pass, eight tickets with live work in them would have gone to Done.

The sweep

The backlog had 506 open tickets. After exclusions (epics, 24 tickets for one integration, 14 tickets for a parked app, anything labelled desk or window), 429 were candidates. The question was which were already fixed in code, which had a premise that was gone, and which were a contained fix an executor could ship in under two hours.

The pipeline was scripts and CLI calls:

1. dump_open.py pulls every open ticket with plain-text description and the last 3 comments. Read-only JIRA.
2. git_evidence.py finds every ticket key named in a commit message across four repos. chunk2.py attaches up to 25 commits per ticket and splits the set into 10 chunks.
3. One sonnet per chunk classifies each ticket and writes one line per ticket to class_<K>.jsonl: {key, class, bet, evidence, left, dup_of}, with class one of KEEP, PARTIAL, SMALL_FIX, FIXED, OBSOLETE.

The result summed to 429: 255 KEEP, 66 PARTIAL, 54 SMALL_FIX, 40 FIXED, 14 OBSOLETE.

Classifiers were told to choose KEEP on thin evidence.

Small fixes hiding in the buckets

We ended up shipping 41 small fixes, and some of them had been classified PARTIAL or KEEP. They turned out small once an executor looked. The class arithmetic does not close: the 40 small-fix closes include tickets classified PARTIAL or KEEP.

The verifier prompt

The 54 FIXED and OBSOLETE claims were split across 2 files and handed to fresh sonnets with a different brief. The brief told the verifiers to try to disprove each claim. Each verdict was CONFIRMED, REJECTED or a third value for tickets needing a human decision, plus a why.

The saved files total 42 CONFIRMED, 8 REJECTED, 4 needing a human decision. The lane manager’s brief said 41 confirmed, 7 rejected, 6 left for a human decision. The files do not show that split. The verifier split is unreconciled. The record follows the files.

Each of the 8 rejections came with file:line evidence that an ask was only partly met:

• A cron retirement where only the catch-up half was retired.
• An email dispatch that was built on the desktop, not on the machine the ticket named.
• A shadow-validation week that had not been run.
• A funnel with only 4 of 7 funnel events.
• A --cancel flag where the sid-only requirement was not met.
• A send path where keyless sends were still allowed.
• Two tickets for the parked app, which was parked by decision and not gone.

Each rejected ticket stayed open with a “partly done, staying open” comment where that applied.

The ones a verifier cannot settle

Four came back needing a human decision. Two were tickets where the code had shipped, but the ticket was waiting on a choice that an agent had already made. The other two were tickets for the parked app, where closing is a product decision. All of them stayed open with a comment saying why.

What was applied

• 34 FIXED tickets closed by close_fixed.py, each with an evidence comment and the verifier’s note, then Done. A further 2 closed as Won’t do on a hub ruling.
• 7 tickets deleted as obsolete. delete_obsolete.py archives the JSON and comments to deleted_hq/ first, and overrides its 7-day recent-write guard only when every recent write was goals-lane tagging. The reasons were concrete: a duplicate of a ticket that stays canonical, an approval guard that a commit deleted, a local path that no longer exists, a dashboard component archived on 2026-09-02.
• 41 small fixes shipped and closed (close_w1.py 21, close_w2.py 6, close_w3.py 14), each with its commit hash in the Done comment. 6 tickets got a “partly done” comment instead of closing.

Every follow-up commit also got an Opus review. As reported by the lane manager, that found one BLOCKER and nine HIGHs, all fixed before the ticket was closed. Those counts are as reported and were not re-derived.

What we would not claim

No live JIRA query was made when writing this up, so open counts are what the scripts left. Several Done comments say a fix was not checked against live systems. And the classifier evidence is a claim as of 2026-10-03, so it ages with the code.

Takeaway

One classifier was not enough. The verifier prompt was “try to disprove”, not “check”, and 54 closure claims became 8 rejections with file:line evidence. Four tickets needed a human decision and stayed open. The verifier split is unreconciled, and the counts here follow the saved files.

dxdev LESSON · paste into your AI coding agent
LESSON: Have A Second Reviewer Try To Disprove Every Closure Claim Before You Close Anything
SOURCE: dxdev.com/blog/2026-10-03_one-classifier-is-not-enough
WHAT HAPPENED: A backlog sweep ran 10 sonnet classifiers over 429 candidate tickets, using commit evidence from four repos, and produced 40 FIXED and 14 OBSOLETE claims. Those 54 claims were split across 2 files and given to fresh sonnets with a different brief, which was to try to disprove the claim rather than check it. The saved verifier files show 42 CONFIRMED, 8 REJECTED and 4 needing a human decision. Each rejection cited file:line evidence, for example a cron retirement where only the catch-up half was retired, a funnel with 4 of 7 events instrumented, and a send path where keyless sends were still allowed. The 4 human-decision cases were not code questions, so they stayed open with a comment explaining why. The rejected tickets stayed open instead of going to Done.
THE RULE: Before closing work on an automated classifier's say-so, have an independent pass whose brief is to disprove the claim, and require it to cite file:line evidence. Route anything that is a decision rather than a code question to a human.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Every automated FIXED or OBSOLETE claim is reviewed by a separate pass whose prompt says to try to disprove the claim, not to confirm it.
      2. Each rejection or confirmation from the verifier carries file:line evidence that can be checked against the code.
      3. Tickets that turn on a product or ownership decision have a separate verdict that leaves them open for a human, and each one has a comment saying why.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.