The 5.28 Second Triage That Said Cause Unclear While the Site Stayed Down

At 3:41 PM the automated triage came back in 5.28 seconds, 76 tokens, thirteen cents: “cause unclear: health check failed after deploy AND rollback also failed; underlying app failure not shown in this log.” That was the whole verdict. The cheap low-effort pass we run on every CI alert had nothing to add, and for about half an hour after that, the site was down.

The pipeline is supposed to make this a non-event. A deploy ships, a health check hits a known endpoint, and if that check fails, the pipeline rolls back to the last good release automatically, with nobody paged at 3 AM. We had trusted that path enough that we hadn’t looked at it closely in months. That afternoon was the first time the rollback itself became the thing that needed rolling back.

What “cause unclear” was hiding

The triage only had the health check response and the deploy log to work from. It never saw the app’s own runtime output, so “unclear” really meant “not visible from here.” We didn’t stop at its answer. We pulled the rollback build alongside the forward deploy build that had just failed and compared them line by line.

The forward deploy runs a template substitution step that writes a production memory setting into the runtime config before the process starts, well above the framework default. The rollback path doesn’t run that step. It rebuilds the previous release from source through a separate script, one that predates the memory tuning and was never updated when we added it. So every rollback for months had been building with the out-of-the-box ceiling instead of the production one. It never mattered, because rollback rarely ran and the pages it served on the way back up were light.

This time they weren’t. The health check endpoint touches a code path that allocates enough to clear the default ceiling under real load. The rebuilt rollback hit the memory cap, fataled, and returned a 500 on its own health check. The pipeline read that as “rollback unhealthy,” which under our rules means retry the rollback. That rebuilds the same undersized process again. The pipeline was executing its rules correctly, in a loop, against a target that could never pass.

The fast fix was obvious: copy the memory setting into the rollback script too. We considered it and rejected it. It leaves two build paths that can still diverge the next time either one changes, which is the bug we had just found, with the value patched for now and the mechanism left intact.

We also considered widening the health check’s retry window, on the theory that a slower rollback would eventually catch up. It wouldn’t have. The rebuilt process was never going to come up healthy at that memory ceiling no matter how long we waited. A longer timeout would only have stretched the half hour before anyone saw that the loop wasn’t going to resolve itself.

What we shipped instead: rollback no longer rebuilds from source at all. It re-points the running service at the last known-good build artifact, the same one the forward deploy already produced and tagged, config and all. There is nothing left to drift, because there is nothing left to rebuild.

The same shape, elsewhere

This wasn’t an isolated find. The same week we caught a pre-push guard that had silently never been able to pass on a Windows checkout. We also spent an afternoon consolidating one repo down from 49 branches and eight stray clones, because nobody could say with confidence which of them held real work. Different systems, same shape: a path that exists for safety, runs rarely, and drifts out of sync with the path people actually watch.

Any code path whose whole job is to run after something else has already gone wrong gets the least production traffic and the least pressure keeping it honest. That is what we took from it, and it is narrower than “check your rollback.”

So ask whether your rollback is rebuilt from the exact same artifact as the release it replaces. If it goes through a separate build path, it is a second deploy pretending to be a rollback, and it can drift out of sync with the path people watch.

dxdev LESSON · paste into your AI coding agent
LESSON: A rollback built through a separate code path from the forward deploy will silently drift out of sync with it, and it is not really a rollback until it is
SOURCE: dxdev.com/blog/deploy-rollback-config-drift
WHAT HAPPENED: An automated rollback, triggered when a deploy's health check failed, itself failed, and the automated triage pass returned an inconclusive verdict. Manually diffing the rollback build against the forward deploy build line by line found that the forward deploy applied a production memory-tuning step the rollback's separate rebuild-from-source script had never received, since that script predated the memory tuning and was never updated when it was added. Every rollback for months had quietly been building with the out-of-the-box memory ceiling instead of the production one, unnoticed because rollback rarely ran and its past loads were light. This time real load hit that ceiling, the rebuilt rollback's own health check failed under its own undersized memory limit, and the pipeline correctly read that as an unhealthy rollback and retried it, executing a loop against a target that could never pass. The obvious quick fix, copying the memory setting into the rollback script too, was explicitly rejected for leaving two build paths that could still diverge the next time either one changed, the same underlying bug with the current value fixed and the mechanism still intact. The actual fix eliminated the rebuild step entirely: rollback now re-points the running service at the exact tagged build artifact the forward deploy already produced, so there is nothing left to drift because there is nothing left to rebuild.
THE RULE: Any code path whose entire job is to run only after something else has already failed is, by construction, the path with the least traffic and the least pressure keeping it honest, so it will silently drift out of sync with the path people actually watch unless it is built from the literal same artifact rather than a separate, parallel process; patching the current symptom of that drift leaves the underlying two-paths-that-can-diverge mechanism intact for the next change.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Any rollback, failover, or disaster-recovery path built through a separate build or deploy script from the primary path it is meant to restore, rather than reusing the exact same tagged artifact.
      2. Any safety or recovery path exercised rarely enough that a configuration or behavior drift between it and the primary path could go unnoticed for an extended period.
      3. A proposed fix to a discovered drift that copies the current correct value into the second path, rather than eliminating that second path's independent existence entirely.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.