A Pass That Only Proves the Thing Exists
What stays with me is how reasonable each step looked. A good batch got a ‘keep going’, and the audit later showed many of those passes proved only that a file existed.
A Pass That Only Proves the Thing Exists
An independent audit of 10 of our 163 ‘pass’ verdicts found 6 weak ones, 4 sound and none wrong. An AI agent had marked work as done after seeing that a file or change record existed, which does not prove the work behaves. The agent rechecked all 163: 62 stay pass, 59 passed a real check, and 42 were downgraded, none disproved. If you let AI sign off on work, ask what evidence it checked. A receipt is not a result.
One Failed Health Check Is Not a Reason to Kill It
One missed 1-second check was enough for our automation to kill and relaunch a busy but healthy signed-in browser. The tool treated a single slow answer as proof the browser was stuck. An AI assistant wrote 6 tests that failed first, then added rules: retry for 8 seconds, spare the browser if it is in use, and spare it after a recent success. A truly silent browser is still repaired after 15 minutes. Any monitor you run should retry before it kills something that is working.
The Gap Checker Called a Deliberate 0.1 Missing
Our gap checker flagged a deliberate Story Points value of 0.1 as missing, and the ticket asked whether that was real. It was real once, because the old check flagged any value under 1. A later commit already fixed it to flag only blank or zero, so the agent changed no code. It confirmed this against the live ticket and tests, and left one stale record for the manager to clear. Before paying for a fix, check whether the problem still exists.
Reply and tell me which of your own automated checks you would want audited for passes that only prove something exists.
Not sure where AI would actually help in your business? Our AI workflow readiness assessment is the place to start.
Curious how it is built behind the scenes? Subscribe to Follow the Build, our technical weekly.
Ben Downey