One seat ran out of weekly quota, and our ticket bot kept calling it for hours. Nothing in its logs read as a quota failure, because as far as the bot could tell, no failure had happened.

What the classifier read

The bot shells out to claude -p for each ticket comment, using a seat, which is a logged-in Claude account with a weekly allowance. When a call fails, the retry logic classifies why, so it can back off on rate limits and rotate on exhaustion. Roughly:

result = run_claude(prompt, seat)
if result.returncode != 0 or result.stderr:
    kind = classify(result.stderr)   # "quota", "rate_limit", "transient"
else:
    kind = "ok"

We had written classify against stderr because that is where errors go. We modeled quota exhaustion as an error, the way an HTTP 429 is an error. It is not one. When a seat hits its weekly limit, the usage-limit message is printed on standard output as if it were the model’s reply, and standard error stays empty. So the call went through the “ok” branch, or through the generic transient bucket when the exit status was nonzero. Either way classify never saw the word quota. The usage-limit text got treated as the reply to the ticket, or as a flaky call worth another attempt on the same seat.

Found by accident

No failing check caught this. A separate session was investigating why the agents were spending a maxed-out account without knowing it, and the retry history showed the same seat being hit again and again for hours with no rotation. The watchdog that was supposed to catch a stuck seat was not running at all. That is a second bug, and it meant the first one had no backstop.

The wrong turn was ours. We built the monitor, tested it against the failures we could imagine, and shipped it. Every test we wrote produced stderr, because we wrote the fixtures. The suite passed the whole time the seat was burning retries, since it only proved the classifier handled the shape of failure we already believed in.

Splitting the streams

The diagnosis took one move once we stopped trusting the classifier’s inputs: run the call by hand against the exhausted seat with stdout and stderr captured separately. Stdout held the limit message. Stderr was empty. Every layer above that had been faithfully processing an empty string.

The fix has two parts:

  • classify now scans both streams, and a usage-limit message on stdout is a quota failure regardless of exit status or stderr.
  • The bot now picks a seat with quota left before each call, instead of finding out afterward.

We verified it live on the running bot, not just in tests. A refused issue-tracker comment also now keeps its detail instead of being binned, so the next odd failure leaves evidence.

The sampler that reported success

The pacing sampler is the other half. It polls each seat’s usage so routing knows who has headroom. Its failure mode was the same mistake in a different costume: when a seat dropped out of what the sampler could see, it reported success on the seats it could see. An invisible seat and a healthy seat produced the same green output.

We changed it to fail loudly. If a seat that routing knows about has no reading, the run errors and says which one. The pacing tests went to 42 passing, and we added cases where a seat is missing from the sampler’s view, which is the case we had never written a fixture for.

That same morning, a seat went invisible to routing for real. The sampler said so, by name, instead of averaging over the gap. We had not staged it. It was the first live proof the check could fail.

What is still open

The burn rate is still a problem. Picking a seat with quota is a floor. Several unattended jobs draw on a small pool of seats, and one heavy day can take a seat to its ceiling. We already had one hard ceiling event that pushed the platform work onto another account. We filed a multi-seat epic for it, and it is not solved.

A check only covers the failure you imagined when you wrote it, and we had imagined an error on stderr. The vendor chose a different exit path, and our fixtures, written from our own assumption, agreed with us. Now a seat with no reading fails the run, and silence from a component is the first thing we page on.

If you run automated jobs against a vendor account, find out what happens when that account hits its usage limit, and how you would know. For us the limit message arrived on standard output while the classifier read only standard error, and the watchdog was not running.