---
title: "Replaying 385 recorded builds against a monitor that paged on one HTTP read"
canonical: https://dxdev.com/blog/close-tool-assumed-release-cadence/
datePublished: 2026-08-01
---
A Discord page at 8:12 PM said our web box and the Cloudflare edge were down. Prod was serving 800 to 1700 requests a minute straight through it.

The box's own TCP probe said it was up, and so did its collector. Only one failed HTTP read from the monitor said otherwise, and the page fired on that single read. The session to untangle it ran 6h 24m.

## Replaying 385 builds

We fixed the monitor in three layers, then replayed all 385 recorded builds through the new logic. The old logic paged four times across that history. The new logic paged zero. A real outage still paged, which mattered as much as the zero.

Then we went to close the ticket that tracked the fix, and the close tooling ran into its own bad assumption.

## The weekly train that wasn't there

The close tooling in our ops repo assumed every repo for our product ships on the weekly release train. Some repos deploy continuously. Closing that ticket exposed the gap, and we changed the tooling so continuously-deployed repos are modeled properly.

## A ticket with no branch

Another ticket had the same shape. Its fix was a server-side SSH change, so there was no branch to trigger the normal close path. The work was fixed and verified but never closed. We re-verified that the mail server clock and its hourly cron were still holding, then closed the ticket as NO CODE and moved it onto the email tools epic, where the rest of that investigation lives.

## A bot that couldn't say what it was running

We ended the monitor session by making the Assistant bot's droplet a real checkout, because nothing there knew what revision was running. It turned out not to have drifted, and the bot had never restarted.

## Hotfixes in the wrong order

We also diagnosed the CI back-merge guard alerts as an ordering slip on a set of hotfixes. Master got the code minutes before develop. We confirmed everything had settled green, then shipped and verified a pre-push guard across all 30 clones so develop-first violations get blocked at push time instead of paging.

## One picture of how work flows

Each of these tools was built around a single picture of how work moves. The close tooling assumed a weekly train. The close path assumed a branch behind every fix. The bot's server had no way to report what it was running. The monitor had the same flaw: it trusted one HTTP read over a TCP probe and a collector that both disagreed with it.

For each monitor and each close or release tool you depend on, it is worth writing down what picture of the work it assumes: a single HTTP read, a weekly release train, a branch behind every fix, a server that can report its own revision. Replaying recorded history, as we did with the 385 builds, shows whether a change pages when it should and stays silent when it shouldn't.