The last commit we made on a destructive recovery script was three lines that left the script unable to do anything, and that was the whole point. A recovery tool that sits one accidental page-hit away from rewriting production is a liability you forgot to put away.

Here’s the setup. A customer’s entire site got deleted: an organization with 327 units under it. The production app stores every organization’s site as rows keyed by username across hundreds of per-category tables. There’s no RESTORE DATABASE button for that, because you don’t want to restore the whole multi-tenant database to recover one tenant. So we hand-built a recovery engine in the legacy application at an internal admin path. It resurrects one account from a dated backup DB, row by row, table by table, with the foreign keys remapped in a second pass.

It worked. The organization came back. And then we almost left the gun loaded.

What the file looked like after the successful run

When the run finished, the file on disk read:

var RECOVERY_USERNAME = "DEMO_ORG_2024";
var RECOVERY_PHASE = "run"; // (plan or run)

A real production username was hardcoded in it, and the phase was set to "run", the writing mode. The file sits at a URL. Anyone or anything that hit that internal admin path would re-execute a live, writing recovery against a real customer account: a second full write pass against the target organization, on top of the data we’d just carefully restored.

The script is a web page, so the risk is concrete. A stray GET from a bookmark, a crawler, a fat-fingered URL, a quick visit to check that the file rendered, and a destructive operation starts against production with no confirmation step. The most dangerous moment for an operator script is the minute right after it succeeds. That’s when it’s fully armed and you’ve stopped paying attention to it.

Two constants, both pointing the wrong way

RECOVERY_USERNAME held a real prod identity. A destructive tool sitting idle should never have a real target baked in. If the default target is a real account, then “accidentally run this” and “accidentally run this against a live customer” are the same accident. The resting target should be something that can’t resolve to anything, so that even if the script fires, it fires at nothing.

RECOVERY_PHASE = "run" was the other problem. The script has two phases. "plan" is read-only: it counts rows, enumerates every unit, writes nothing, and dumps a diffable plan file. "run" writes. Leaving it on "run" meant the safety was off by default.

One part of the original design made disarming cheap. EDIT_DATABASE, the flag every write in the script checks before it touches the database, is derived from RECOVERY_PHASE, not a separate boolean set by hand. There is one source of truth. The phase can’t say “plan” while a forgotten EDIT_DATABASE = true two hundred lines down keeps the write path hot. Flip the phase and the entire write path goes with it.

If N booleans all have to agree for a dangerous thing to happen, sooner or later one of them gets left disagreeing in the dangerous direction. Derive them from one and there’s exactly one switch to think about.

The end-of-day commit

The fix was a small commit with the message “disable recover.” Three lines:

- var RECOVERY_USERNAME = "DEMO_ORG_2024";
+ var RECOVERY_USERNAME = "USERNAME_GOES_HERE";


- var RECOVERY_PHASE = "run"; // (plan or run)
+ // CHANGE TO "run" for LIVE RUN
+ var RECOVERY_PHASE = "plan"; // (plan or run)

After that commit, the script at rest targets nothing real and writes nothing. To make it dangerous again, the next operator (probably us) has to put in a real username and flip the phase to "run". That takes two separate edits, with the arming comment right beside them. No single keystroke and no single stray request turns it back into a live weapon.

The placeholder matters as much as the phase flip. "USERNAME_GOES_HERE" is not a valid account, so even if the phase somehow got left on "run", the script would be aiming at a username that resolves to nothing. Defense in depth, two cheap constants deep.

Before you close the laptop

When a destructive operations script finishes its job, check what it is pointed at and what mode it is left in. At rest it should target nothing real and write nothing, and arming it again should take two separate, deliberate edits.