The Usage Dashboard Said 60 Percent and Nothing Was Running

The alert said 60 percent. Our usage dashboard for 2026-09-10 showed spend at nearly double the trailing week’s daily average. We started hunting for a runaway agent, a stuck loop, a session that forgot to stop. That cost about twenty minutes before the bug surfaced: the number we were reading was broken.

The wrong turn: chasing a runaway that wasn’t there

The work log for that stretch was full of legitimate candidates. A session log entry from that window reads: “Found the runaway: an orphaned issue-tracker rewrite sweep from a closed session was burning credits and popping console windows.” That was a real incident, and it primed us to assume the quota spike had the same shape. So we went looking for its twin. We grepped the wrapper-cost logs for any process still tagged active past its session’s end. We checked ops-cli background for zombie PIDs. We cross-referenced scheduled task registrations against the headless-migration tracking list, in case something had started re-spawning windows and credits with it.

Nothing. Every session in the window closed cleanly. No orphaned sweep, no duplicate cron, no process holding a lock past its lifetime. We had spent real time proving that the issue-tracker sweep from three days earlier wasn’t happening again, while the report we were reading never described spend accurately in the first place.

What the rollup was actually doing

The rollup script read each session’s transcript and summed usage blocks per file. It had two bugs, and both were in how the read happened. Nothing an agent did was involved.

Double-counting. Usage blocks were read on every file scan, and a long-running session’s transcript gets rewritten incrementally as it goes. With no stable identity per usage event, a block already counted on a prior scan got counted again on the next one if the file had grown. There was no dedup key at all. The script summed whatever blocks were present in the file each time it ran.

Misattribution by mtime. The rollup bucketed cost by file modification time, where the timestamp inside the message was the right key. A session that started at 11 PM and kept writing to its transcript past midnight had its early spend filed under the day the file was last touched. The sessions in that day’s log ran long: 18h12m, 16h17m, 15h14m, several starting mid-afternoon and running to 5 or 6 AM. Crossing midnight was the rule for them.

Put together: a session’s early spend got filed on day N, and some of its later blocks got recounted on a subsequent scan and filed on day N+1 by mtime. Day N+1 picked up both its own real spend and a phantom slice of the day before. That is how an ordinary day reads as a 60 percent spike without a single extra token being spent.

The fix

We rewrote the rollup around a SQLite-backed store instead of a re-scan-and-sum:

• Usage blocks are deduped by message.id, so a block that already has a row never gets summed twice, however many times the source file is rescanned.
• Buckets are keyed by the message’s own timestamp.
• The day boundary moved to 6am to 6am instead of midnight to midnight. A working day in this shop routinely starts in the afternoon and ends past 3 or 4 AM, and midnight was just the boundary the naive timestamp math picked by default.
• Each file gets an incremental per-file cursor, so a rescan picks up only new blocks instead of re-reading and re-deriving the whole file.
• The wrapper-cost log entries fold into the same rows as the primary transcript usage, so a session’s total isn’t split across two places that could drift independently.

It shipped as ops-cli usage-rollup {backfill,update,status,days,sessions,messages}, backfilled 89 days back to 2026-06-13. A new test suite (253 lines) pins the dedup and bucketing behavior directly, so a future rewrite can’t reintroduce the double-count silently.

Same week, a different layer

The same week, a separate investigation turned up a measurement problem in hardware with the identical failure shape. The machine had been running slow, and the assumption going in was the usual one: some single process is pegging the CPU, so find it and kill it. What we found instead: “Found the machine was creation-bound rather than compute-bound, with no single hot process: a dozen ordinary things each spawning handfuls of processes across thirteen live sessions, while every diagnostic tool failed silently under exactly the load it was meant to measure.” The process queue was sitting around 300. We fixed the safety guards that were failing open and collapsed five processes per edit down to one. The queue dropped to near zero, and no single culprit was ever identified because there wasn’t one.

In both incidents the thing reporting on the system failed precisely under the load the system was generating, and the failure looked at first glance like the problem it was supposed to catch. A quota tool that double-counts under real session volume and a diagnostic tool that goes silent under real process load are the same bug wearing different clothes. Once you run agents at volume, your instrumentation fails under the same conditions as everything else, and it has to be debugged and tested like everything else.

When a cost figure jumps, check how the number is counted before you go looking for a process to kill.

dxdev LESSON · paste into your AI coding agent
LESSON: Debug The Meter When Its Failure Can Mimic The Incident
SOURCE: dxdev.com/blog/quota-crisis-was-a-measurement-bug
WHAT HAPPENED: The investigation first looked for orphaned agents, duplicate scheduled work, and zombie processes because a previous runaway sweep had created a similar incident. Those checks found that all sessions had closed cleanly. The usage rollup repeatedly summed previously seen usage blocks and assigned costs according to when a transcript file was last modified instead of when each message was created. The team replaced the rescan and sum approach with SQLite storage, message ID deduplication, message timestamp buckets, incremental cursors, and a 6 AM day boundary. A separate investigation that week found diagnostics failing silently under high process load, again making measurement failure look like the system problem itself.
THE RULE: When an alert is produced by instrumentation operating under the same load as the system, treat the measurement path as a possible source of the incident before acting on the alert. Design usage and diagnostic tools so repeated reads, delayed writes, and high load cannot silently distort what they report.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Each recorded usage event has a stable unique identifier, and repeated ingestion cannot add its cost more than once.
      2. Daily cost buckets use each event's own timestamp rather than the source file's modification time.
      3. Instrumentation has tests that exercise its reporting behavior under the same session volume or process load it is expected to observe.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.