---
title: "A Pass That Only Proves the Thing Exists"
canonical: https://dxdev.com/everyday-ai/a-pass-that-only-proves-existence/
datePublished: 2026-10-12
---
An independent audit of 10 of our 163 "pass" verdicts found 4 sound, 6 weak and 0 wrong. The 6 weak ones all had the same problem. The check had proved that a file or a change record existed, but the claim being checked was about something more: behaviour, a deploy, a measurement, test results, or a stated blocked state.

Here is how we got there. We keep a queue of units, each one a claim that some piece of work was done. An AI agent was working through that queue, checking each claim against real evidence and marking it pass or fail. The brief was explicit that anything it could not verify was to be left unconfirmed with a one-line reason, and never marked confirmed on plausibility. Its first batch marked 34 as pass, refuted none, and left 15 of the oldest 49 unconfirmed. The reply to that batch was "good batch. Keep going on the rest of the queue." About 870 units remained, so it kept going and added 127 more passes. Its report at that point read 161 confirmed, 0 refuted. The audit counted 163 pass verdicts, and the rework that followed covered all 163.

## What a pass was actually resting on

The agent's own reports describe the gap. The records only hold roughly the first 200 characters of each claim, so some units could only be checked on the part it could see. It also said that a commit or ticket existing doesn't prove the behaviour or the test result. Each pass carried an evidence note listing what was checked and what wasn't, and the audit's follow-up instruction was aimed at the notes that said something was not checked, not re-run or not re-measured.

## Going back over all 163

The instruction after the audit was to go back over all of them. Where an evidence note said something was not checked, not re-run or not re-measured, the agent was to do the real check if it was read-only and cheap. Otherwise it was to record a superseding verdict that was not a pass, with the gap quoted.

The result: 62 stay as pass. 59 were re-recorded as pass after a real check. 42 are now fail. The agent's summary was that the audit was right, and that many of its passes only showed a commit or file existed.

The real checks were ordinary ones:

- It ran the full test suite for one of our APIs on a fresh copy of current `origin/main`. The result was 2301 passed, 1 skipped, 0 failed, once `PUBLIC_CURSOR_SECRET` was set to a dummy value. Without it, 296 tests fail with 503s because that variable is missing.
- It ran the targeted tests for the units it had previously checked by commit alone. 411 passed, 0 failed.
- It looked at what is live. The production API-key dialog bundle matches `origin/master` and contains the 1-year expiry option, the focus-return flag and the new hint text. The production newsletter popup serves the fix. A production schedule script is byte-identical to its tag. dxdev.com `/about` and `/now` carry the final copy.

The full-suite run is a result for current main, not for the original commits. A few units have no test named for their change, and the agent said so in those units' notes.

## Why the 42 are labeled the way they are

The CLI's verify command only accepts `pass` or `fail`, so each downgrade had to be recorded as a `fail`. Each one is marked "DOWNGRADE, NOT REFUTED" and quotes the gap. None was disproved. The latest verdicts per unit now stand at 121 pass and 42 fail. Any metric that counts fails as refutations will count all 42 of them, so it needs to filter on the "DOWNGRADE, NOT REFUTED" wording.