By the time we opened the backup, a deleted site had already turned into a recovery involving 327 units.

At first the problem sounded small. A customer’s site was gone, and the usual picture is a few pages restored from a saved copy. That was not the picture here. The site was a large web of information spread across many locations, events, results, news and statistics. Every unit belonged to the same organization, with records pointing back and forth to each other. Restoring it meant writing new copies into the live system customers use every day, while making sure every internal reference still pointed to the right new place. Getting that wrong once would be bad. Getting it wrong the same way 327 times could turn a recovery into a much larger cleanup.

So we did not start by changing anything live. We made the recovery walk through the whole job with its hands in its pockets first. It read the backup, found all 327 units, counted everything it would need to bring back, and wrote down every decision. The result was a 683-line plan file. It said what the recovery would do, unit by unit, without doing any of it yet.

That list mattered because a message like “everything looks good” would have been useless. A list can be checked against the result afterward, line for line. If a unit was promised in the plan but missing from the recovery, the gap would show. If seven events were expected but fourteen appeared, that would show too.

That second example was not imaginary. An early test exposed exactly that mistake. One unit had seven events in the backup, but the target ended up with fourteen. They were duplicates, copies of the same events under new labels. The first version of the recovery had no protection against being run twice, and we had not thought to add one until the plan-versus-result comparison put the mismatch in front of us. Missing it would have cost us time at the worst possible moment, with a customer waiting and a manual cleanup in the live system hanging over the work.

Fixing the double run

The fix was a label. Each new event was marked with the label of the event it came from. Before adding an event, the recovery checked whether it had already brought over the one with that label. If it had, it reused the existing one instead of making a duplicate. A second run could now be checked in seconds by counting, instead of by digging through a pile of nearly identical records.

There was one more design choice that kept the risky part from getting loose. The recovery had a single setting with two positions: plan or run. Plan meant read and write the list. Run meant make the changes. There was no separate switch for whether the script was allowed to write. That question was answered entirely by which position the one setting was in, so there was nothing to forget to also flip back.

A practice run on a throwaway copy

The plan file proved the walk through the backup was sane. It could not prove the writes were correct, because in plan mode nothing gets written. So before touching the real customer’s account, we ran one full recovery against a disposable copy of the database, made specifically so it could be thrown away. There the recovery was allowed to write, and three plain checks ran against the result afterward:

  • Did the event count match the backup?
  • Did every new record carry its original label?
  • Were there any duplicate pairs that should have been only one?

Only after one account passed all three, on a copy that didn’t matter, did the recovery run against the real one.

When the real run finished, it produced a second file: 988 lines, one line per unit recovered. Now there were two documents describing the same job from two sides. One said what should happen. The other said what did happen. Comparing them made “did this actually work” something you could answer by reading.

Putting it back on safe

The last part came after the recovery had already worked, and it’s the part we were most tempted to skip because the job was done. The script was still pointed at a real account, and its setting was still on run. So the final step was resetting the target to a placeholder and the setting back to plan. Using it live again would now take two deliberate choices instead of one leftover one.

If you run something like this

You don’t need 683 lines. Most bulk changes don’t touch 327 of anything. But the same three questions apply to any job that repeats one change across many records, whether that’s a data cleanup, a bulk email, a price update, or a batch of refunds:

  1. Can it tell you what it’s going to do before it does it, as a list you could actually read, rather than a count you have to trust?
  2. Can it tell the difference between running for the first time and running again, or will a second run double the damage?
  3. Does turning it off take one step, or does someone have to remember two separate ones?

The 327-unit recovery answered yes to all three, and the second question is the one that caught a mistake before it multiplied 327 times.

Before any bulk change runs against live records, ask for a readable plan of what it will do, and compare that plan with what it did afterward. Ask whether a second run is safe, and whether it has been tested on a disposable copy first. Then make sure the tool is returned to its safe setting when the job is done, so that using it live again takes a deliberate choice.