The Safety Mechanism That Became the Failure: Why Your Rollback Rebuild Matters

Cause unclear

At 3:41 PM the automated triage came back in 5.28 seconds, 76 tokens, thirteen cents: “cause unclear: health check failed after deploy AND rollback also failed; underlying app failure not shown in this log.” That’s the whole verdict. The cheap low-effort pass we run on every CI alert had nothing to add, and for about half an hour after that, the site was down.

The pipeline is supposed to make this a non-event. A deploy ships, a health check hits a known endpoint, and if that check fails, the pipeline automatically rolls back to the last good release without anyone getting paged at 3 AM. We’d trusted that path enough not to look at it closely in months. The failure that afternoon was the first time the rollback itself became the thing that needed rolling back.

What “cause unclear” was hiding

We didn’t take the triage’s word for it. The automated pass only had the health check response and the deploy log to work from, not the app’s own runtime output, so “unclear” really meant “not visible from here.” We pulled the rollback build alongside the forward deploy build that had just failed, line by line.

The forward deploy applies a template substitution step that writes a production memory setting into the runtime config before the process starts, well above the framework default. The rollback path doesn’t run that step. It rebuilds the previous release from source through a separate script, one that predates the memory tuning and was never updated when we added it. So every rollback for months had quietly been building with the out-of-the-box ceiling instead of the production one. It never mattered, because rollback rarely got exercised and the pages it served on the way back up were light.

This time it wasn’t light. The health check endpoint itself touches a code path that allocates enough to clear the default ceiling under real load, so the rebuilt rollback hit the memory cap, fataled, and 500’d its own health check. The pipeline read that as “rollback unhealthy,” which under our rules means retry the rollback, which rebuilds the same undersized process again. It wasn’t stuck. It was correctly executing a loop against a target that could never pass.

Why we didn’t just patch the number

The fast fix is obvious: copy the memory setting into the rollback script too. We considered it and rejected it, because that leaves two build paths that can still diverge the next time either one changes. That’s exactly the bug we’d just found, just with the value fixed for now and the mechanism left intact.

We also considered widening the health check’s retry window, on the theory that a slower rollback would eventually catch up. It wouldn’t have. The rebuilt process was never going to come up healthy at that memory ceiling no matter how long we waited, so a longer timeout only would have made the half hour longer before anyone realized the loop wasn’t going to resolve itself.

What we shipped instead: rollback no longer rebuilds from source at all. It re-points the running service at the last known-good build artifact, the same one the forward deploy already produced and tagged, config and all. There’s nothing left to drift, because there’s nothing left to rebuild.

The pattern, not just the incident

This wasn’t an isolated find. The same week we caught a pre-push guard that had silently never been able to pass on a Windows checkout, and we spent an afternoon consolidating one repo down from 49 branches and eight stray clones because nobody could say with confidence which of them held real work. Different systems, same shape: a path that exists for safety, gets exercised rarely, and drifts quietly out of sync with the path people actually watch.

What we took from it was narrower than “check your rollback.” Any code path whose whole job is to run when something else has already gone wrong is, by construction, the path with the least production traffic and the least pressure keeping it honest. If it isn’t rebuilt from the exact same artifact as the thing it’s supposed to replace, it isn’t a rollback. It’s a second deploy pretending to be one.

dxdev LESSON · paste into your AI coding agent
LESSON: A rollback built through a separate code path from the forward deploy will silently drift out of sync with it, and it is not really a rollback until it is
SOURCE: dxdev.com/blog/2026-08-27_deploy-rollback-config-drift
WHAT HAPPENED: An automated rollback, triggered when a deploy's health check failed, itself failed, and the automated triage pass returned an inconclusive verdict. Manually diffing the rollback build against the forward deploy build line by line found that the forward deploy applied a production memory-tuning step the rollback's separate rebuild-from-source script had never received, since that script predated the memory tuning and was never updated when it was added. Every rollback for months had quietly been building with the out-of-the-box memory ceiling instead of the production one, unnoticed because rollback rarely ran and its past loads were light. This time real load hit that ceiling, the rebuilt rollback's own health check failed under its own undersized memory limit, and the pipeline correctly read that as an unhealthy rollback and retried it, executing a loop against a target that could never pass. The obvious quick fix, copying the memory setting into the rollback script too, was explicitly rejected for leaving two build paths that could still diverge the next time either one changed, the same underlying bug with the current value fixed and the mechanism still intact. The actual fix eliminated the rebuild step entirely: rollback now re-points the running service at the exact tagged build artifact the forward deploy already produced, so there is nothing left to drift because there is nothing left to rebuild.
THE RULE: Any code path whose entire job is to run only after something else has already failed is, by construction, the path with the least traffic and the least pressure keeping it honest, so it will silently drift out of sync with the path people actually watch unless it is built from the literal same artifact rather than a separate, parallel process; patching the current symptom of that drift leaves the underlying two-paths-that-can-diverge mechanism intact for the next change.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Any rollback, failover, or disaster-recovery path built through a separate build or deploy script from the primary path it is meant to restore, rather than reusing the exact same tagged artifact.
      2. Any safety or recovery path exercised rarely enough that a configuration or behavior drift between it and the primary path could go unnoticed for an extended period.
      3. A proposed fix to a discovered drift that copies the current correct value into the second path, rather than eliminating that second path's independent existence entirely.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.