A customer’s site has been offline for a day, sometimes two. They made the DNS change we asked for, they know DNS “takes time to propagate”, and they are waiting and refreshing, as the instructions said.
Waiting will never fix it, because propagation is not the problem. An SSL certificate needs to be reissued on our side. Until someone notices, nobody is acting and both of us keep waiting.
That was one ticket out of roughly forty that our domain migration generated in six weeks. On paper the migration was simple: move thousands of customer domains onto Cloudflare, batch by batch, canary first. Nearly every small batch unearthed some unexpected, unique nuance, and it took us embarrassingly long to see what they had in common.
the greatest hits
A sample from the collection:
- Staff screens showing “Awaiting DNS” on domains that had already migrated, because the status was a stored guess from record-creation time that nobody recomputed.
- Bulk cleanup that left old certificate renewals running against retired infrastructure.
- New customer domains that could fail setup without anyone, including the program, noticing.
- A mobile-only DNS resolution error that existed on exactly one browser.
- A secondary domain whose redirect stopped while its primary worked fine.
- An automation worker that stalled for 22 hours and nothing complained.
Each one got a ticket, a fix, sometimes a staff doc update. Forty times.
forty bugs, then forty missing states
That was the wrong turn, and it cost us about thirty tickets of patching one symptom at a time. Somewhere around the thirtieth the pattern stopped hiding: these were not forty bugs. They were forty missing states in a state machine nobody had written down.
A migrating domain lives in a lattice of partial truths: DNS moved or not, certificate issued or pending or needs-reissue, origin answering or not, redirects intact or broken, cleanup done or half-done. Once we started writing the states down, most of the forty tickets collapsed into a handful of rows:
| domain state | live check that defines it | next action | owner |
|---|---|---|---|
| DNS not moved | nameservers still at old registrar | update DNS records | customer |
| cert pending | DNS moved, no valid cert at the Cloudflare edge | reissue the certificate | us |
| origin dark | cert valid, origin returns 5xx | fix the origin binding | us |
| redirect broken | primary resolves, secondary 404s | repoint the redirect | us |
| cleanup half-done | site live, old renewal still firing | retire the old cert job | automation |
Three audiences were each inferring their own version of that lattice. Staff read screens that guessed. Customers read emails written before their specific situation existed. The automation read database flags set by whatever step ran last.
When all three agree, nobody notices the machinery. When they diverge, you get the customer patiently waiting for propagation while the real state, visible to no one, is “cert needs reissue, our move.”
the fix is a sentence
The canonical state must be computed from live checks, and every state gets exactly one next action and one owner.
Live checks, because the stored answer drifts the moment the world moves and a live answer cannot: where does DNS actually resolve, is the cert actually valid, does the origin actually answer. One next action with one owner, because the whole failure mode is two parties each believing it’s the other’s turn. “Waiting on you: update these records” and “waiting on us: reissuing your certificate” are different sentences, and showing the right one is the whole game.
Staff tools, customer status pages, and the worker all read that same state. Training becomes mostly a matter of trusting the board. The stuck-domain detector falls out for free: any domain sitting in any state longer than that state should take pages a human. That is the alarm that would have caught the cert-reissue customer on hour one instead of day two.
Here is where we are. Pieces of this exist: the board reads live DNS checks now, and several screens got rewired from stored status to live verification. The full consolidation is still queued, sitting behind the current batch. But the lens already changed how we triage. When the next batch surfaces its inevitable surprise, our first question is which state we forgot to name.
The next migration we run will start with the state machine on paper, incomplete and wrong in ways the batches will correct. Forty tickets bought us that. Every surprise after this one lands as a row we add.
Before a migration starts, write down the states a domain can be in, the live check that defines each state, and who owns the next action. Wherever a stored status or a customer email stands in for a live check, two parties can each believe it is the other’s turn, and a customer ends up offline waiting.