The Recovery Script That Stayed Armed After 327 Units Came Back

All 327 units were back, and I still couldn’t close the page and call the recovery finished.

Then we looked at the two lines near the top of the file and saw that we had left the page loaded and ready to fire.

We call it a recovery script, which is just a page that puts lost information back where it belongs. In this case, a customer’s whole site had been deleted. The job was to bring back one organization without rolling back everybody else’s work. The page copied the organization’s saved information from an older backup, piece by piece, then connected the pieces back together.

The first thing we tried was the obvious thing: bring the site back. That worked. The organization came back. But treating that successful run as the end of the job did not work. The page was still sitting at an internal address with a real customer account named in it. It was also set to its writing mode.

That second detail matters. The page had two settings. Plan meant, “Show what you would do.” It counted the rows of information, listed the units, and wrote down a plan without changing anything. Run meant, “Go ahead and change the records.” The plan was a rehearsal. The run was the actual work.

When we finished restoring the organization, the page was still set to run. A bookmark, a mistyped address, or someone checking whether the page opened could have started the same writing job again against the same real account. There was no extra button asking, “Are you sure?” The page would have treated an ordinary visit as permission.

No customer data was hit a second time. We caught it first. That still doesn’t mean there was no cost. The recovery was no longer something we could safely forget about. Until we fixed those two lines, one accidental visit could have turned a careful restoration into a second full pass over a real customer’s records. The work that had seemed finished still needed an end-of-day cleanup step.

How ordinary the trigger was

The first run needed deliberate attention. The next one could have started through the same small act people take all day: opening a link. That gap between “we mean to do this” and “we happened to visit this” is where risky tools need to protect people.

Three lines

The fix was only three lines. We replaced the real account name with USERNAME_GOES_HERE, which cannot point to a real account. Then we changed the setting back to plan and left a comment saying that changing it to run was for a live job.

That did two useful things. First, if the page opened by accident, it had nothing real to aim at. Second, even if someone wanted to use it for a real recovery later, they had to make two separate choices: type a real account name and switch from plan to run. One forgotten setting would not be enough.

We also made sure the page had one master switch. Its “is writing allowed?” decision came from the plan or run setting, instead of from another separate yes or no box buried lower in the file. That matters because two switches can disagree. A page can look safe at the top while a forgotten setting elsewhere still lets it make changes. One switch is easier to see, easier to test, and harder to leave in the wrong position.

Plenty of ordinary work has a version of this problem. A payroll sheet may still point to last month’s list. A mass email tool may have real addresses loaded after the test is over. A piece of equipment may be left ready for the next person who walks up to it.

The fix took three lines and no redesign: a placeholder where the real account name had been, the setting flipped back to plan, and one write switch instead of two that could disagree with each other.

For anything left plugged in when a job looks finished, check whether it is aimed at nothing, and whether starting it again for real takes two separate deliberate choices instead of one ordinary visit.

dxdev LESSON · paste into your AI coding agent
LESSON: A recovery tool left armed and pointed at a real account is a second incident waiting for an ordinary visit
SOURCE: dxdev.com/everyday-ai/league-was-back-but-i-wasn-t
WHAT HAPPENED: A recovery page had two settings, a plan mode that only rehearses the change and a run mode that writes it, plus a hardcoded real customer account name. After the recovery finished successfully and restored 327 teams, the page was left with the real account name still in place and the mode still set to run. Nothing about an ordinary page visit (a bookmark, a mistyped URL, someone checking whether the page loaded) would have asked for confirmation before writing again against that same real account.
THE RULE: A tool capable of writing to production must be idle-safe by default: when nobody is actively using it, it should be pointed at nothing and set to change nothing, and that has to be true on disk, not just true in the operator's memory.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any admin, recovery, or migration script with a real account id, customer name, or environment name hardcoded rather than a clearly-fake placeholder, with no automatic reset after the job that used it finishes.
  2. Any write-capable tool whose permission to write is controlled by more than one setting or flag, so closing one gate can still leave another one open.
  3. Any page or script reachable by a plain GET request or an ordinary page load that can trigger a write, with no explicit confirmation step in between.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.