The Alarm That Paged 55 Times on Its First Morning

We built an alarm to catch a real, known problem before it turned into a crisis again. On its first real morning of use, it paged 55 times, all at once, all saying the same thing: stuck, broken, needs attention.

Forty-four of those 55 were fine. Each was a site still waiting on a routine migration step, sitting in a state that looked identical to a real failure if you only checked the one thing the alarm was checking.

One signal could not tell them apart

The alarm looked at a single piece of recorded state to decide whether something was healthy. Most of the time that is useful. But it cannot separate “this hasn’t happened yet because it isn’t supposed to yet” from “this was supposed to happen and didn’t.” From that one angle, both look exactly the same.

Forty-four false alarms would have been an annoying morning and nothing worse. What made it serious was the other eleven, sitting in the same batch of 55 under the identical “stuck” label. Those were sites whose secure address returned a connection failure while their plain address still answered. For anyone trying to reach them securely, they were down. The alarm saw 55 identical problems, so it had no way to pick out the eleven real outages from the 44 non-events.

What we changed

We added a second, independent check. Instead of trusting the recorded state alone, the alarm now looks at where the site actually resolves to right now. That split the batch correctly. The 44 healthy ones got logged, counted and muted, and the muted count is printed plainly so a quiet morning can never be mistaken for a clean one. The eleven broken ones got paged.

One more piece mattered just as much. Before paging anyone, the alarm re-checks its finding against fresh data. A finding from even a few minutes ago can already be stale, and paging on a stale finding spends a person’s attention on something that may have fixed itself.

Who decides what is an emergency

The alarm does not decide what counts as an emergency. It decides what is worth putting in front of a person. A person still reads the page, judges how urgent it is and decides what to do. What the second check buys is confidence that when a page arrives, it points at something real, and is not buried among 44 false ones that would have taught everyone to stop reading closely.

Before you trust any alarm to page a person, work out what it would take to fool it with something completely healthy. If the answer is “nothing, one field is enough,” add a second, independent check before the page goes out.

dxdev LESSON · paste into your AI coding agent
LESSON: A Watchdog Judging By One Signal Pages On Noise And Hides The Real Failure Inside It
SOURCE: dxdev.com/everyday-ai/a-watchdog-that-checked-only-one-signal
WHAT HAPPENED: A monitoring check built to catch a real class of failure judged everything by a single field. On its first real morning it paged 55 times, all wearing the identical label. Forty-four of those 55 turned out to be completely fine, sites still mid-migration with nothing actually broken. The other eleven, sitting inside that same batch of 55, were genuinely down, and the one signal being checked could not tell those eleven apart from the 44 false alarms. Adding a second signal, checking where something actually resolves rather than just its recorded state, split the batch correctly: the 44 fine ones got logged and muted with a visible count, the eleven broken ones got paged.
THE RULE: A check that judges by one signal will page on things that are fine and hide things that are actually broken inside the same list. Add a second, independent signal before deciding what's real, and always re-verify against current data right before paging anyone.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any alert that classifies something as broken using a single field, with no independent second check before it decides.
  2. Any deliberately muted alert whose muted count isn't visible anywhere, so a quiet run looks the same as a clean one.
  3. Any page sent from a stale or cached finding without a fresh recheck immediately before it goes out.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.