What a Second Unannounced Restart Twenty Minutes Later Revealed

The runbook said the site would be down for about three minutes during the reboot. The box was a production machine that had been putting off a restart for weeks, with a backlog of staged operating-system patches that don’t take effect until it restarts. The plan for that reboot went through two rounds of correction before it ran, and a third round right after, because the box didn’t behave the way any version of the plan expected.

The three-minute figure had never been checked

The earliest draft of the maintenance plan stated that number the way routine downtime windows usually get written. It read like a fact. It was closer to a placeholder, because nobody had checked it against this specific box.

Re-checking the box directly, ahead of scheduling the window, turned up the real picture: a two-month backlog of staged patches, several rounds deep, still waiting to apply. A box with that much staged work can sit mid-reboot applying updates for a long stretch on both the way down and the way back up. The three-minute figure got struck and replaced with a wide range, twenty to forty-five minutes. The plan also gained a blunt note about what happens if the old number survives into the window: whoever is watching stands there at the twenty-minute mark with no clear sign anything is wrong, wondering whether the box died.

The window ran and the plan was still incomplete

The reboot happened before dawn. An observer watched from outside the box the whole time: a snapshot beforehand, the site polled every minute, everything logged. The actual downtime came in under the corrected range, roughly two minutes. Then, about twenty minutes after the box came back and everything looked finished, it went down again on its own, for another two minutes.

Nothing in either version of the plan mentioned a second reboot. It turned out to be entirely expected. A separate system component finishes installing the staged patches after the first restart completes, and queues its own follow-up restart to finish the job. That is documented behavior once you know to look for it. Nobody had looked, because nobody had run this specific box through this specific procedure before. The plan had a shape for “the reboot,” singular, and reality had two.

What the rewrite changed

The maintenance document got rewritten the same day, in four places.

• It now says explicitly to expect a second, unannounced restart, and how to tell it apart from a real failure by reading the log entry that names what triggered it.
• The timing range widened again, with the one real measurement attached as a data point next to the estimate.
• A resource ceiling that had shifted on its own across the reboot, with nobody touching its configuration, got documented so the next comparison doesn’t read it as damage.
• A handful of warning-level log entries that fire on every single boot of that box, before and after, got listed as known noise, so the next person watching doesn’t spend time chasing something that was never new.

Why the first number stuck around

Nobody who wrote “about three minutes” believed it was solid. It was a first pass, meant to be checked. But a number sitting in an operational document carries no visible confidence level. It reads exactly the same whether it came from a real measurement or from a best guess on the first draft, and the next reader, or the same person a few days later under time pressure, can’t tell the difference by looking.

It took a direct re-check against the actual box to catch that the number was wrong, since another read of the doc would not have. And it took an actual run of the real procedure to find the second reboot that even the corrected plan had missed, since another round of reasoning about it would not have either.

For any maintenance plan, it’s worth sorting the numbers into the ones measured on the actual system and the ones that are first-draft guesses. Here the three-minute figure was a guess. A direct re-check replaced it with a twenty to forty-five minute range, and the run itself exposed the second reboot. The plan is only done once it has been updated with what the run showed.

dxdev LESSON · paste into your AI coding agent
LESSON: An Estimate Written Down Once Becomes the Plan, Whether or Not Anyone Checks It Again
SOURCE: dxdev.com/blog/the-runbook-said-three-minutes
WHAT HAPPENED: A production box hadn't been rebooted in weeks and was carrying a backlog of staged operating-system patches. The first draft of the maintenance plan said the reboot would cost about three minutes of downtime, stated as a fact, the way a routine detail gets stated. Re-reading the box's actual patch backlog before the window opened, that number got corrected to a wide range, twenty to forty-five minutes, with an explicit warning: anyone relying on the original number would be standing there partway through, wondering if the box was coming back. The window ran. Actual downtime was about two minutes, well inside even the low end of the corrected range, except the box rebooted a second time on its own partway through, for reasons the corrected plan hadn't mentioned either, and the total process wasn't done until both cycles finished. The runbook went through a second round of fixes the same day: the reboot count, the wider timing range with the one real measurement attached, a resource ceiling that had silently shifted during the reboot, and a set of routine warning-level log entries that looked alarming and were not.
THE RULE: a number written into an operational document reads as a fact to the next person who acts on it, regardless of whether it was measured or guessed. Guessed numbers should be labeled as guesses at the moment they're written, and any number that matters for a live decision should be re-verified against the actual system before that decision is made, not carried forward because it was already in the doc.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any operational runbook, playbook, or incident doc with a specific number (a duration, a count, a threshold) that isn't marked as measured, estimated, or sourced.
  2. Any procedure that assumes a single occurrence of an event (one reboot, one retry, one failover) where the underlying system is capable of triggering it more than once.
  3. Any doc last verified against a real run of the procedure it describes, versus one that has only ever been reasoned about.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.