The Second Claude Account Was Missing Three Tools Without an Error

Over 60 percent of a week’s Claude quota vanished in a single day. We went looking for a runaway loop and didn’t find one. We found two ordinary causes stacked together: a wide spread of daytime sessions, and a large unattended overnight run we hadn’t been watching.

That morning we built a usage-and-outcome tracking page, so the next spike shows up as a chart instead of a guess. But a tracking page only tells you what already happened. The bigger problem underneath was that the whole setup ran on one account. If that account runs dry mid-afternoon, every session and every scheduled job behind it just stops, silently, until someone notices a stale ticket.

A second account, and a diff to check it

The businesses we run share this setup, so the obvious move was a second account. We did not assume the rest of the tooling would carry over untouched. We asked the new login whether it still had the same skills and access, then diffed the two account configs side by side.

The diff found a real gap. The new login was missing the browser-control tool and project trust, and neither was reported as an error. Trust settings and tool grants live per-account, not per-launcher. We copied the config over by hand and verified that a fresh process could reach all three connected tools before trusting it with anything real. That was a separate side task of about 22 minutes, outside the failover work itself.

Wiring the failover

With two working accounts, the goal was for sessions and scheduled jobs to report which account they’re spending, so the usage guards could watch both. Alerting would route by business instead of dumping everything into one channel. We split the work into four per-business sweeps and wired in automatic failover: when an account’s quota guard trips, the jobs on it hop to the other account without anyone touching a config file.

We did not verify that part by reading the code. We ran it as a drill.

What the drill found

The test was simple. We forced one account into a dry state during the pre-close window, when the job load is heaviest, and watched which jobs picked up the failover. Nine of fourteen paged over cleanly. The other five didn’t move.

The five that stayed dark never reached the failover path at all. Four of them were alert jobs that exited with a success code no matter what happened inside. The check that decides “this account is dry, fail over” never ran, because nothing ever triggered it. The fifth job had the same shape. A separate script sitting next to all of them was reading a cached copy of which account was currently active instead of checking fresh, so even a clean failover elsewhere could still look stale to it.

The gap is now an open item on the task itself. After a drill like that it is tempting to say most of it worked and move on, but a business going dark during a busy week with no page at all is not a failure mode that tolerates “most.” The fix is mechanical. Make the four alert jobs and the fifth surface their real exit code instead of swallowing it, fix the script reading the stale account state, and re-run the same drill against all fourteen before calling it done.

The tracking page showed us what had already burned. The drill showed us what would have burned next, and nothing would have paged.

dxdev LESSON · paste into your AI coding agent
LESSON: A Drill Doesn't Count Until You Watch Every Job, Not Most of Them
SOURCE: dxdev.com/blog/failover-tested-incomplete
WHAT HAPPENED: A second account was added as a backup so that automatic scheduled jobs could fail over to it the moment the first ran dry, instead of silently stopping. The failover logic itself worked. A drill that forced the first account dry during the busiest part of the day found nine of fourteen jobs picked it up cleanly. The other five stayed dark, not because failover was broken, but because those five jobs exited with a success code even when they had actually failed, so the code path that decides "this needs to fail over" never ran in the first place. A sixth related script was reading a cached copy of which account was currently active, so even a correct failover elsewhere could look stale to it.
THE RULE: Building the failover path is not the same as proving every job actually reaches it. A job that swallows its own failure (exits 0 regardless of outcome) will never trigger a failover, a retry, or a page, no matter how correct that downstream logic is, because nothing ever calls it. Before calling a self-healing or failover system done, force the failure condition for real, under the load it is meant to survive, and check every job that is supposed to be covered, not a sample of the newest ones. "Nine of fourteen" is not a passing grade when the failure mode is a business unit going dark with no page at all.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any scheduled job or cron whose wrapper always exits 0, or swallows a caught exception, regardless of whether the underlying task actually succeeded.
  2. Any failover, retry, or alerting logic that depends on a specific exit code or signal, where you have not verified every caller that is supposed to produce it actually does.
  3. Any config or state check (e.g. "which account/seat is active") that reads a cached or previously-loaded value instead of the current source of truth, especially in a script that runs infrequently.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.