The job was routine. A background process had been creating orphaned records, meaning entries left behind with nothing attached to them. We fixed the process and cleaned up the ones that had already piled up. Standard practice for a change like that is to write a paired plan for undoing it, in case anything needs to be reversed later. We wrote one. Then we tried to double-check that plan before ever running it for real.

The check that wasn’t checking

We wanted to confirm the undo plan was written correctly without actually running it, so we turned on a feature built for exactly that: look over the steps, don’t execute them. We ran it against the live system and treated it as completely harmless.

It wasn’t. That feature suppresses execution for one particular kind of step. Everything else runs exactly as if the check had never been turned on, and most of an undo plan is made of that “everything else.” So the “safety check” ran the plan for real. The cleanup we had just finished reverted to its messy state, as if none of it had happened.

Why it didn’t become an incident

We were already running verification checks as part of the same pass, and they came back wrong immediately. There was no gap between the mistake and the moment we saw it. We restored everything that had been undone, redid the cleanup, and checked the end state against what it was supposed to look like. It matched.

While putting it back together, we found a second problem sitting in the undo plan that had not fired yet, because nothing had needed the real undo before. One step assumed nothing new had been added since the original change ran, and that assumption was already wrong the same day. We fixed that too, before it could matter.

What the name promised

We did not skip a safety step. We added one specifically to avoid running something risky before we were sure it was right. What we got wrong was trusting what the feature’s name implied instead of checking what it covers. “Won’t execute anything” sounds like a blanket promise. It is a promise about one specific case and silent about everything else, including the exact kind of step our plan was mostly made of. We never confirmed it covered the two plain drop statements (commands that delete things outright) sitting in that plan.

Against a system real people depend on, that is where the risk hid. It was not in the step we added on purpose. It was in the one line of that step we never tested before we trusted it.

Before anyone runs a “dry run” or “preview” mode against a live system, find out what exactly it suppresses, and test it somewhere that isn’t live first. And keep verification checks running alongside any change to a live system, since they are what caught this one right away.