Error 21211 and a Kill Switch Checked Once Per Recipient

Thirty-four tests were green when we stopped trusting them.

We were preparing new safety-net code for a live SMS integration. The queue already had idempotency, a kill switch, retry behavior, and stalled-job recovery. The change closed the window where a recipient is accepted for sending but the job is not yet marked sent. That window stays small until a process crashes, a worker stalls, or a provider rejects one recipient while others are in flight. In a live queue, the edges are the system.

We wrote the tests first, and 34 passed. We had a passing suite, a disciplined-looking change set, and every usual reason to call the work finished.

We did not touch production. We handed the passing suite to a second model and asked it to attack the tests instead of admiring them.

What the 34 covered

The implementation had three requirements. It had to record an accepted message before a crash could create an avoidable duplicate send. It had to stop if the kill switch engaged mid-job. And it had to tell a retryable provider failure apart from a hard error such as 21211.

The first suite covered the happy sequence and the obvious fault paths: retries, kill-switch behavior, and a simulated crash. It went green at 34 of 34.

That count showed the code met the cases we had anticipated. It said nothing about whether we had anticipated the right cases. A green suite can hide a wrong model of time, an incomplete classification, or a safety check that never runs in the worker that matters.

So we made the verification adversarial. The second model got the diff and the tests, with one job: find the path our passing suite had not made impossible.

The fake Redis had no clock

The verifier found the problem in our test double for Redis. fakeRedis did not model time-based TTL expiry the way the real store does. That matters because the safety net uses a short-lived key to control recovery after an interrupted send. A crash test that only manipulates values, without advancing a clock, can look convincing while never proving the key expires under the conditions production will see.

Once the verifier pointed at it, the diagnosis was short. The crash test looked like it tested expiration: it created the short-TTL state and then ran recovery. But the fake store never moved forward in time. The test asserted a sequence against a static key-value model, with no expiry involved.

We changed fakeRedis to model real TTL expiration and added _advance so the crash test could drive the clock deliberately. That moved the test from “a key was present when recovery ran” to “the key expired after the configured interval, then recovery took the expected loss-safe path.” It also surfaced a configuration rule: SHORT_TTL_MS < stallInterval.

If the short TTL outlives the stalled-job interval, the recovery logic can make a loss-safety promise it cannot keep. We added assertTtlInvariant() to enforce the relationship in code, in place of a comment or a remembered deployment setting.

The residual duplicate window

The other hard edge was the gap between accepting a recipient and marking it sent. We narrowed it by retrying markSent three times. If all three attempts fail, the worker emits a mark_sent_failed alert. That does not make the duplicate window disappear. We chose to accept a narrow residual window, monitor it, and make the unrecorded state visible.

We also considered treating every failure around markSent as retryable. That lost because a provider hard error needs a different outcome. Error 21211 is classified as non-retryable when it is thrown in the catch path. Retrying it only adds noise and keeps a bad recipient in a cycle that will not resolve.

We considered checking the kill switch only once, before the job begins, too. That lost because an operator can engage it after the first recipient. The final path checks it per recipient, so a mid-job switch stops the rest of the batch instead of waiting for the next job.

Each choice made the code more explicit about its limits. The markSent retries reduce one dangerous window. The alert exposes the residual one. The per-recipient kill switch makes an operator action take effect mid-execution. The non-retryable classification keeps a known bad destination from looking like transient infrastructure trouble.

The verifier also found two comments that had gone stale after the changes. We fixed them. In queue code, a stale comment tells the next person a safety boundary is somewhere it no longer is.

Handoff

After the fixes, the suite reached 39 of 39 and ran cleanly 20 times. We sent the revised result back through verification. The second pass confirmed the design we chose: the narrow duplicate window stays accepted and monitored, and is not falsely declared eliminated.

One production-protection pass is still left. worker.js is not yet wired to use these modules, and startup must call assertTtlInvariant(). Until both are in place, the hardening lives in tested code but not in the live worker path. That is why we stopped at the boundary instead of treating 39 green tests as permission to cut over.

The first 34 tests told us the implementation matched our plan. The verifier found the place where our plan had failed to model production time. The final 39 are better because one of them is a test we would have shipped without.

A green suite shows that the code meets the cases its authors thought of. Before changing a live queue, have a second reviewer attack the tests, and check that test doubles model time and expiry the way the real store does. Here, that review found a gap 34 passing tests had not exposed.

dxdev LESSON · paste into your AI coding agent
LESSON: A green test suite proves the code handles the cases you thought to test, not that you tested the right cases
SOURCE: dxdev.com/blog/adversarial-verifier-before-touching-prod
WHAT HAPPENED: A safety-net change to a live SMS queue passed 34 of 34 self-written tests covering retries, kill-switch behavior, and a simulated crash. Rather than treat that as finished, a second model was handed the diff and the tests with one job: find the path the passing suite had not made impossible. It found that the fake Redis test double did not model real time-based TTL expiry, so a crash-and-recovery test could look convincing while never actually proving a key expired under the condition the real production queue would see. The fix made the fake store model real TTL expiration with a controllable clock advance, which also surfaced a real invariant, the short TTL must be shorter than the stall-recovery interval, that got enforced with a runtime assertion instead of being left as a comment or a remembered deployment setting. Two more edge decisions were made explicit rather than papered over: a retry-then-alert path for the risky window between accepting a message and marking it sent, which accepts and monitors a narrow residual duplicate-send window rather than falsely claiming to eliminate it, and a kill switch checked per recipient rather than once at job start, since an operator can engage it mid-batch.
THE RULE: A green test suite proves the code meets the cases you thought to test, not that you thought of the right cases; handing the passing suite to an independent model with the explicit job of attacking it, rather than admiring it, is what surfaces a wrong model baked into the test itself, such as a fake store that does not model real time, which no amount of re-reading your own tests will catch.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Any test suite treated as sufficient once it reaches full green, with no adversarial second pass explicitly tasked with finding what the suite does not make impossible.
      2. Any test double or mock, a fake cache, a fake queue, a fake clock, that does not model a real, time-dependent behavior, such as TTL expiry, ordering, or retries, that the production system actually depends on.
      3. Any safety-critical numeric relationship, such as a TTL shorter than an interval or a timeout shorter than a retry window, enforced only by a comment or a remembered deployment setting rather than an explicit runtime assertion.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.