The last commit I made on a destructive recovery script wasn’t a feature. It was three lines that made the script unable to do anything, and that was the whole point. A recovery tool that’s one accidental page-hit away from rewriting production isn’t a tool. It’s a liability you forgot to put away.

Here’s the setup. A customer’s entire sports site got deleted: a basketball league with 327 teams under it. The production app stores every org’s site as rows keyed by username across hundreds of per-sport tables. There’s no RESTORE DATABASE button for that, because you don’t want to restore the whole multi-tenant database to recover one tenant. So I hand-built a recovery engine in Classic ASP (JScript) at an internal admin path that resurrects one account from a dated backup DB, row by row, table by table, with the foreign keys remapped in a second pass.

It worked. The league came back. And then I almost left the gun loaded.

The state right after a successful run

When the run finished, the file on disk looked like this:

var RECOVERY_USERNAME = "DEMO_ORG_2024";
var RECOVERY_PHASE = "run"; // (plan or run)

The script had a real production username hardcoded into it, and the phase was set to "run", the writing mode. The file sits at a URL. Anyone, or anything, that hit that internal admin path would re-execute a live, writing recovery against a real customer account. That would not be a dry run; it would be a second full write pass against the target org, on top of the data I’d just carefully restored.

That’s not a hypothetical. It’s an ASP page. A stray GET from a bookmark, a crawler, a fat-fingered URL, a “let me just check that file rendered,” and you’ve kicked off a destructive operation against production with zero confirmation step. The most dangerous moment for an operator script is the minute right after it succeeds, because that’s exactly when it’s fully armed and you’ve stopped paying attention to it.

The two constants that almost shipped armed

Two things were wrong with the at-rest state.

RECOVERY_USERNAME held a real prod identity. A destructive tool should never have a real target baked in when it’s sitting idle. If the default target is a real account, then “accidentally run this” and “accidentally run this against a live customer” are the same accident. Decouple them. The resting target should be something that can’t resolve to anything, so that even if the script fires, it fires at nothing.

RECOVERY_PHASE = "run" was the other problem. The script has two phases. "plan" is read-only: it counts rows, enumerates every team, writes nothing, dumps a diffable plan file. "run" writes. Leaving it on "run" means the safety is off by default.

Here’s the part of the original design I’m proud of, because it made disarming cheap. EDIT_DATABASE, the flag every single write in the script checks before it touches the database, is not a separate boolean you set by hand. It’s derived from RECOVERY_PHASE. One source of truth. You cannot half-arm this thing. You can’t have the phase say “plan” while some forgotten EDIT_DATABASE = true two hundred lines down keeps the write path hot. Flip the phase and the entire write path goes with it.

That’s the design lesson hiding inside the safety lesson: if you have N booleans that all have to agree for a dangerous thing to happen, you will eventually leave one of them disagreeing in the dangerous direction. Derive them from one. Then there’s exactly one switch to think about.

The end-of-day commit

The fix was a small commit, message “disable recover.” It’s a three-line change:

- var RECOVERY_USERNAME = "DEMO_ORG_2024";
+ var RECOVERY_USERNAME = "USERNAME_GOES_HERE";


- var RECOVERY_PHASE = "run"; // (plan or run)
+ // CHANGE TO "run" for LIVE RUN
+ var RECOVERY_PHASE = "plan"; // (plan or run)

After that commit the resting state of the script is: targets nothing real, writes nothing. To make it dangerous again, the next operator (probably future me) has to put in a real username and flip the phase to "run". Two separate edits, on purpose, with the arming comment staring right at them. There is no single keystroke and no single stray request that turns this back into a live weapon.

The placeholder matters as much as the phase flip. "USERNAME_GOES_HERE" is not a valid account. So even in the failure case where the phase somehow got left on "run", the script would be aiming at a username that resolves to nothing. Defense in depth, two cheap constants deep.