The Alarm That Paged 55 Times and Nearly Missed the Real Emergency Inside the Noise

The morning the alarm cried wolf fifty-five times

We built a watchdog to catch a real, known problem before it became a crisis again. On its first real morning of actual use, it paged 55 times, all at once, all saying the same thing: stuck, broken, needs attention.

Forty-four of the 55 weren’t broken at all. Each one was a site still waiting on a routine migration step, sitting in a state that looked identical to a real failure if you only checked the one thing the watchdog was checking.

Why one signal wasn’t enough

The watchdog looked at a single piece of recorded state to decide whether something was healthy. That single signal is genuinely useful, most of the time. It just cannot tell the difference between “this hasn’t happened yet because it’s not supposed to yet” and “this is supposed to have happened and didn’t.” Both look exactly the same from that one angle.

A pile of 44 false alarms would have been an annoying but survivable morning on its own. What made it a real problem was the other eleven, sitting inside that same batch of 55, wearing the identical “stuck” label: sites whose secure address returned a connection failure while their plain address still answered, meaning they were genuinely down for anyone trying to reach them securely. One signal couldn’t tell those eleven real outages apart from the 44 non-events, because from where it was looking, all 55 looked exactly the same.

What changed

The fix added a second, independent signal: instead of trusting the recorded state alone, the watchdog checked where the thing actually resolves to right now. That second check split the batch correctly. The 44 that were genuinely fine got logged, counted, and muted, with that muted count printed plainly so a quiet morning could never be confused with a clean one. The eleven that were actually broken got paged.

One more piece mattered just as much: before paging anyone, the watchdog now re-checks its finding against fresh, current data one more time. A finding from even a few minutes ago can already be stale. Paging on a stale finding wastes a person’s attention on something that may have already resolved itself.

What a person still has to decide

None of this makes the watchdog the one deciding what’s an emergency. It decides what’s worth putting in front of a person. A person still reads the page, decides how urgent it really is, and decides what to do about it. What the second signal buys is confidence that when the page arrives, it’s pointing at something real, not diluted inside forty-four false ones that would have trained anyone to stop reading it closely.

The rule worth keeping

If your monitoring only checks one thing, it will eventually page you for something that’s fine and stay quiet, hidden in plain sight, about something that isn’t. Before you trust any alert to page a person, ask what it would take to fool that alert with something completely healthy. If the answer is “nothing, one field is enough,” add a second, independent check before the page goes out.

dxdev LESSON · paste into your AI coding agent
LESSON: A Watchdog Judging By One Signal Pages On Noise And Hides The Real Failure Inside It
SOURCE: dxdev.com/blog/2026-08-06_a-watchdog-that-checked-only-one-signal
WHAT HAPPENED: A monitoring check built to catch a real class of failure judged everything by a single field. On its first real morning it paged 55 times, all wearing the identical label. Forty-four of those 55 turned out to be completely fine, sites still mid-migration with nothing actually broken. The other eleven, sitting inside that same batch of 55, were genuinely down, and the one signal being checked could not tell those eleven apart from the 44 false alarms. Adding a second signal, checking where something actually resolves rather than just its recorded state, split the batch correctly: the 44 fine ones got logged and muted with a visible count, the eleven broken ones got paged.
THE RULE: A check that judges by one signal will page on things that are fine and hide things that are actually broken inside the same list. Add a second, independent signal before deciding what's real, and always re-verify against current data right before paging anyone.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any alert that classifies something as broken using a single field, with no independent second check before it decides.
  2. Any deliberately muted alert whose muted count isn't visible anywhere, so a quiet run looks the same as a clean one.
  3. Any page sent from a stale or cached finding without a fresh recheck immediately before it goes out.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.