Roughly 21 percent of the approvals were guessed

Forty-three windows, closed by hand

One day, we closed forty-three agent windows manually, one at a time, with no batch action and no shortcut. That number kicked off the post-mortem. We asked a session to reconstruct what had happened that day, and it counted something else first. About half of everything we’d typed into any session that day was some version of “where are you.” Those were status checks. They weren’t instructions or corrections, just us asking sessions where they stood.

The first read on that pattern is trust. We don’t trust the agents to tell us when something’s done, so we check constantly, and the checking generates the mess we then clean up by hand. It’s a plausible story. It’s also the wrong layer to fix, and the post-mortem only found that out because it refused to stop there and pulled the actual instrumentation instead.

What “delivered” actually meant

Under those windows was a delivery step that placed a window onto our screen and logged it as done. If we weren’t at the keyboard when a session finished, the window still went up and still counted as a successful handoff. Then it sat there in an empty room until we came back, noticed it, and closed it. Every one of those was a false positive. The metric said “delivered,” and the ground truth was that nobody was there.

It’s the same shape as a monitor that looks clean because it excludes the thing that would make it dirty. The fix was presence detection. Sessions now check whether we’re actually at the keyboard, and if we’re not, they hand back a link instead of placing a window nobody will see. The “success” event only fires when someone is there to receive it.

The watcher that had been lying for two days

Digging into the attention data to size the problem turned up a second, uglier finding. The watcher that tracks whether we’re at the keyboard had been running stale code for two days. It wasn’t down and it wasn’t erroring. It was silently executing an old build the whole time. Any conclusion drawn from the last two days of attention data had to be thrown out before it could serve as a baseline for anything, including the post-mortem that was using it.

The same audit showed a third instrument estimating instead of measuring. The trust ladder is meant to track how often we approved a session’s output. It was guessing at roughly 21 percent of our approvals rather than counting them. A number that looked like ground truth was partly invented.

None of the three was the same bug, but they shared a failure mode. Each system reported a clean signal that wasn’t derived from the event it claimed to represent. If we’d fixed the top-line symptom by adding more automation to close windows, the dashboard would have looked better and we’d never have touched why the windows were empty.

The fixes that shipped were boring by comparison:

• Presence-aware delivery replaced blind window placement.
• The desk queue stopped hoarding requests from sessions that had already closed. Those requests were another source of phantom backlog inflating the “still pending” count.
• The watcher went back onto live code.
• The trust ladder started counting real approvals instead of estimating them.

A full handoff went onto the ticket for whoever picked up the thread next.

The same shape, the same day

A separate ticket filed in that session covered an order-dependent test hiding inside the suite’s own baseline. Its pass or fail outcome depended on what ran before it, and it lived inside what we treated as “green.” It was a different subsystem with an identical shape. The baseline you diff your fix against has to be verified as a baseline before the diff means anything. A metric that improved because the thing measuring it changed underneath you is noise dressed as a fix, and the only way to tell the difference is to open the instrument itself before you believe what it reports.

What to check before you trust a number

Before acting on an operations metric, check that it is counted from the event it claims to represent rather than estimated. Also check that the tool producing it is running current code. Here a “delivered” count, an attention watcher and an approvals figure each failed that check.

dxdev LESSON · paste into your AI coding agent
LESSON: A delivery metric that fires on placed rather than seen will report success on a handoff nobody actually received
SOURCE: dxdev.com/blog/verify-fix-metrics-baseline-shift
WHAT HAPPENED: A pattern of forty-three manually closed agent windows in one day, alongside roughly half of all typed input that day being pure status-check questions asking where a session stood, was first read as a trust problem, not trusting agents to report their own completion, prompting constant manual checking. The actual mechanism was a delivery step that counted a session as successfully delivered the moment it placed a window on screen, regardless of whether anyone was actually at the keyboard to see it, so every session finishing while the operator was away still logged as a success and sat unseen until manually noticed and closed. The same investigation, digging into presence data to size the problem, found the presence-detection watcher had silently been running two-day-old stale code the entire time, not crashed and not erroring, just executing an outdated build, meaning any attention data from that window was unusable as a baseline until it was thrown out. A third instrument, meant to count real approvals, was found to be estimating roughly a fifth of its numbers rather than counting them, making a figure that looked like ground truth partly invented. A separate, same-day ticket found an order-dependent test hiding inside what the team treated as a clean baseline, the identical shape of bug, an instrument reporting a signal not actually derived from the event it claims to measure, recurring in a completely different subsystem.
THE RULE: Before trusting that a fix improved a metric, verify the instrument producing that metric was not itself stale, broken, or estimating during the measurement window; a metric reporting a clean signal that is not actually derived from the real event it claims to represent will make an unrelated fix look like it solved the problem when the real defect was in what was being measured, not in the top-line symptom.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Any success or delivered event that fires when an action is taken, such as placing a window or sending a notification, rather than when its intended recipient actually receives or acts on it.
      2. Any metric, dashboard, or baseline being used to evaluate a fix's impact without first confirming the instrument producing that metric was running current code and actually measuring, rather than estimating, the event in question.
      3. A recurring manual-cleanup pattern, such as closing windows or clearing a queue, addressed by adding more automation to do the cleanup, without first checking why the items needing cleanup were generated in the first place.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.