---
title: "The Guard That Works But Can't Undo"
canonical: https://dxdev.com/blog/2026-09-18_live-guard-rollback-untested/
datePublished: 2026-09-18
---
One delete against a self-managed customer domain came back blocked, and that was my proof the guard worked. It was also the last thing I checked before I noticed the guard existed only on a branch nobody had merged.

## What the guard does

Our platform has a cleanup worker that removes origin content for domains we host. Some customers manage their own domains, and for those the platform has a flag, `selfmanaged`, set to true. The worker was never supposed to touch origin content for `selfmanaged` domains. Nothing enforced that. A bad cleanup pass could delete a live customer site off the origin box.

The fix is a fail-closed check in the cleanup path. Before any origin delete, the worker looks at the domain. If `selfmanaged` is true, it refuses. If the flag can't be read, it also refuses. Missing data gets the same answer as a protected domain, because the alternative is a delete that runs on a guess.

## The wrong turn: deploying from the held branch

The runbook fixes for this work were sitting on a held hotfix branch. I built the guard there, deployed it to the origin box from that branch, and verified it. The log entry for that session reads like a clean finish: built, deployed, verified, proven live by a blocked delete.

The cost was that production was now running code master did not contain. Any later deploy from master would replace the guard with the version that predates it, and it would do so quietly. No error, no diff anyone would look at, just a protection that stops existing on the next release. The guard was permanent on the box and temporary in the repo.

I chose this order for a reason. I wanted the protection live before anything else, and a merge needs review and a wait. But I had made the safety fix depend on a merge I hadn't done. The session summary flags it as the open item: the guard is running on prod, its commit isn't on master, and the rollback drill hasn't been run.

## Why the rollback drill matters more than the guard

A guard that blocks deletes has a failure mode the original problem lacked. If it misfires, it blocks a delete that should have happened, and it does so silently. Now the undo path matters. If we have to pull the guard back out, I need to know that reverting the commit leaves the origin box in a state I understand, and I hadn't checked that. Everything I knew about undoing it came from reading the diff, not from doing it.

So the situation that night was this. The failure I had fixed was unlikely and severe. The failure I had introduced was a revert I hadn't rehearsed, and one deploy from master could trigger it without anyone asking.

## What I did about it

The same night, I added two things around the guard:

- A monitoring doc for the cleanup worker, written alongside the guard, listing where its alerts don't fire yet. The worker has alert gaps, and a blocked delete is exactly the sort of event that should page someone. Right now it only shows up if you go looking.
- API access to our third-party uptime-monitoring service, so an alert can be traced back to what triggered it rather than found by accident.

The log then shows the guard merged to master and develop, and the vault's monitoring doc slimmed to a pointer at the canonical one in the platform repo. That closes the revert risk. It does not close the rollback drill, which I still hadn't run.

## What I'd do differently

Merge first, or deploy from the branch that will still exist after the next release. If the hotfix branch has to be the source, the merge to master goes on the same checklist as the deploy, not in a later session. The verification I did was real. A blocked delete on a live customer domain is better evidence than any unit test. But it verified the box, and the thing that could undo the guard was the repo.

I'd also run the rollback drill before calling a guard done. Blocking a delete proves it works. Reverting it cleanly proves you're allowed to keep running it.