The Rollback Nobody Had Ever Tried Against Four Expired Test Sites

The migration checklist had one item still unticked: “rollback exercised once for real”. The plan promised that any customer address we had moved could be moved back. Nobody had ever tried it.

What we found when we looked

We were moving customers’ web addresses from an old server to a new setup in phases. A fallback existed on paper. The first job was working out what had actually been tested. One backup route, the recovery of a middle piece, had been run. Moving an address back to the old server had not. It was a paragraph.

The fallback only works if the old server still holds a credential for that address. Take that away and a visitor gets an error page instead of the site.

We went looking for something safe to rehearse on. We believed three of our own test sites were in the right state, and we had believed it for long enough that it cost us several rounds of work before we checked. They were not. An earlier tidy-up had already removed everything from the old server for all three, so they had no way back at all. The sites that did qualify were four expired test sites that we look after ourselves, each still holding its full set. We picked one that was half moved.

The rehearsal

The rehearsal had four steps, each checked before the next:

1. Finish the move.
2. Prove the way back still exists, without changing anything.
3. Move it back and confirm that both the registrar’s own records and the public internet agreed.
4. Put it back exactly as we found it.

The old server kept serving the site securely through the whole thing, and the test site ended the rehearsal exactly as it started. Run the same step-two check on a site that had already been tidied up and it failed straight away. That contrast made the case: cleaned up means no way back, kept means an instant way back.

What changed in the plan

The written procedure now begins with the read-only check. If it fails, the way back is gone and the procedure says to stop. We also filed a ticket for a waiting period before any tidy-up, because the credential that makes going back possible is the exact thing the tidy-up deletes.

The label that stood the AI down

One more thing went wrong during the rehearsal. The AI assistant saw a status reading “disabled” next to the browser and decided it had no browser to use, so it stood down. The label only meant that one feature was switched off, and the browser itself worked. We renamed the status to say what it actually turns off.

The AI ran each step, checked the result after every one and wrote the procedure down from what really happened. It did not decide which site was safe to rehearse on, and its confident memory about which test sites qualified was wrong until it was checked. The status label showed how a single word can talk an AI out of a tool it needs.

We chose the subject and the moment. A real customer’s site was never in play, and we would not have handed that choice to a tool. Deciding whether the plan was now proven enough to rely on was also ours.

Before you rely on a fallback, use it once on something safe, and find out what it needs in order to work. Then make sure the cleanup that comes later does not remove it.

dxdev LESSON · paste into your AI agent

LESSON: Rehearse the way back, and find out what it depends on

WHAT TO SHARE:
A de-identified description of a change you plan to roll out in stages, the fallback you would use if it went wrong, and any cleanup step that happens after the change.

ASK YOUR AGENT TO:
1. List what the fallback needs to still exist for it to work.
2. Say whether any planned cleanup step would remove one of those things, and when.
3. Suggest a safe subject to rehearse the fallback on, and one read-only check that tells you before you start whether the fallback still works for that subject.
4. List what should be recorded before, during and after the rehearsal so you can show it left things exactly as found.

RETURN:
The dependencies of the fallback, the cleanup step that threatens them, a rehearsal subject, and the read-only check.

BOUNDARY: Do not run the rehearsal or touch anything live. A person picks the subject and decides when to go.