--- title: "Why off-edge went quiet while stalled kept paging" canonical: https://dxdev.com/blog/alerts-that-page-after-the-problem-heals-itself/ datePublished: 2026-09-14 --- ## Fifty-five DMs before breakfast By 7 AM there were [55 Discord DMs in the channel](/everyday-ai/a-watchdog-that-checked-only-one-signal/), one bot message per line. All of them came from the Cloudflare cert watcher, the script that scans every domain on the platform and flags anything wrong with a certificate. We first read it as something that had caught fire overnight. It hadn't. Every one of the 55 was the same three customers who had never pointed their DNS at us. The morning scan found them again, as the scan before it had, and the one before that. The finding kind was "off-edge". Cloudflare has no record of ever serving that hostname, because the domain's DNS still points somewhere else entirely. That is real information, but nobody at the platform has any reason to be woken up for it. The customer has to fix their own DNS. There is no relaunch to run and no fix commit to ship, so nothing on our side is actionable. A cron job restated the same fact every scan cycle to whoever happened to be holding a phone. ## The filter keyed to one string Months earlier, the fast fix had been to add `off-edge` to a small set called `INFO_KINDS` and skip paging on it. That worked for `off-edge`. It did nothing for `stalled`, a second finding kind the same watcher already emitted. `stalled` fired for a domain that looked like it had never pointed DNS at us at all, not even a stale record. It was the same customer-side problem with the same nothing-for-us-to-fix shape, under a different string. Nobody had connected the two, so `stalled` kept paging on its own schedule while `off-edge` sat quiet. We had muted the finding we'd looked at, not the category it belonged to. The cost was weeks of the exact noise the earlier fix was supposed to have ended. It came from a sibling of the thing we'd already fixed, and the filter only knew the first one's name. ## A ledger of already-told-you The fix that held was a place to record that something had already been said, so nothing downstream had to re-derive it. It is a small module with three verbs: - `healed(key, what, venture)` is for a job that noticed a problem and fixed it itself. - `held(key, what, venture, reason)` is for a job that noticed a problem and chose not to page yet. - `resolved(key, venture)` closes one out. Every call appends one line to a ledger file. None of the three sends a DM on its own. Paging becomes a decision made once, upstream, by whatever decides whether a given finding is ours to act on. `off-edge` and `stalled` both route through `held()` now, keyed by hostname with the reason `"customer DNS"`. A morning digest reads the ledger back and shows them as a line item instead of a page. Three other finding kinds got a second gate on top: expiring certs, stuck renewals, and a DNS TXT-drift check. They only page if the exact same signature also showed up on a scan from an earlier calendar day. A cert mid-renewal that clears within the hour never crosses that gate. A second manual scan run an hour later doesn't count as "survived a day" either, because the gate compares dates rather than timestamps. A finding that is still stuck at the next morning's scan does cross it. ## Two DMs for one dead job The same day surfaced a worse version of the problem one layer down, in the [cron wrapper every scheduled job calls](/blog/eleven-tasks-reported-success-three-had-failed/). That wrapper already paged a failure the moment a job exited non-zero, keyed `cron-fail:`, and it wrote an append-only log with one JSON line per run: name, return code, end timestamp. Separately, a health sweep had been built earlier as a safety net for tasks that fail silently without the wrapper ever running. It polled the same Task Scheduler state and paged again under its own key, `task-health::`. The two key shapes went into the same alert gate, so the dedupe logic never recognized them as one event. One dead cron job produced two DMs. The fix was to have the sweep read the wrapper's own history log before building its findings. It now drops anything from its list that already has a failed record there inside the last 24 hours. The wrapper itself stopped paging on the first failure for most jobs. It now reads the previous run's end record before writing this run's, and it pages only if that earlier run also failed. A job that dies once and comes back clean on the next cycle gets logged as healed instead of paged, because by the time a DM about it would land, it would already not be true. A handful of jobs opted back into paging on the first miss, among them a daily backup and a weekly report. For those, the "second occurrence" is a week away, and waiting for a second failure is the wrong trade.