Why a Simulated Crash Passed 34 Tests Without Proving Expiry

Our messaging queue had a gap where a message could be accepted for sending but not yet marked as sent. If the system crashed at the wrong moment, the same message could go out twice. We were changing the safety-net code underneath that live queue to close the gap. When the fix is right, nobody notices it, and that is exactly the kind of change where “it looks fine” tells you the least.

We wrote the tests before the fix, the way you’re supposed to. There were 34 of them, covering retries, a manual kill switch, and a simulated crash partway through. All 34 passed. That is usually where a piece of work gets called finished, because a fully green suite feels like proof.

The passing suite and the change went to a second, independent reviewer with one instruction: don’t check whether this looks reasonable, try to find whatever the passing tests failed to rule out. The job was to attack the work.

What the reviewer found

The reviewer found something the 34 tests could never have caught, because the gap was in the tests themselves. The stand-in we used for the real storage system, the one that simulated the crash, didn’t behave like the real thing when it came to values that expire after a set time. A test could look completely convincing: crash simulated, recovery triggered, verdict green. It still never proved that a value would expire under the real timing conditions the live system would face. The suite was answering a slightly different question than the one that mattered. Reading the tests more carefully would not have revealed that, because the blind spot sat one layer beneath them.

A second problem under the first

Finding that surfaced something else we fixed properly instead of just noting. One safety window had to stay shorter than a separate recovery window. That had been true by luck, not by design. We turned it into an active check that fails loudly if it ever stops being true. Before, only a comment reminded anyone.

What the green suite did and didn’t tell us

A fully green suite tells you your code does what you designed it to do, in the situations you thought to check. It says nothing about the situations you didn’t think of. It also says nothing about whether the stand-ins simulating a real failure behave like the real thing.

dxdev LESSON · paste into your AI coding agent
LESSON: A full pass on your own tests proves you tested what you thought of, not that you thought of the right things
WHAT TO SHARE: Describe a recent change that passed all of its own tests before shipping, and what those tests actually check.
ASK YOUR AGENT TO:
1. Review the change and its tests specifically looking for what the tests do NOT make impossible, rather than confirming what they already cover.
2. Flag any test that simulates a real-world condition, timing, load, a failure, in a simplified way that might not match how it actually behaves.
3. Identify any important rule or relationship the change depends on that is only enforced by a comment or an assumption, not an active check.
RETURN: A short table of gap found, why the original tests missed it, and the smallest fix.
BOUNDARY: Review and analysis only. No agent deploys a change, modifies a live system, or approves a release; a person decides what to do with what's found.