Our Monitor Was Right About the Condition and Wrong About Everything Else

One customer’s domain had triggered our urgent outage page on six of the previous nine days. The monitor behind it watches every domain we serve for certificate and connection trouble, and it pages the most urgent kind as a live outage. Someone finally asked why that particular domain kept getting the serious warning instead of the routine kind.

The answer I had ready

The alert’s logic is simple. It fires when a hostname is not being served through our edge and the server it does point to cannot complete a basic secure connection. This domain met both conditions exactly, so we explained the classification as correct. It read like a good answer. It restated the rule and confirmed the rule had fired the way it was built to fire.

The question it never touched

The question was never “did the alert’s condition match.” It was “is this outage ours to act on.” It was not. The domain pointed at a server the customer runs themselves, somewhere else entirely, and that server being down had nothing to do with anything we operate.

The alert had been built for a narrower, real problem. A migration cleanup had retired some of our own server configuration while customer records still pointed at it, so a handful of sites lost secure access even though the older, less secure version kept working. That is urgent, because it is ours. A customer’s own third-party host being unreachable is a different category of problem. Treating it as the same one meant six false urgent pages in nine days over something no fix on our end could ever touch.

Two passes to get there

The first time, we defended the classification because the trigger condition was, in the narrowest sense, true. That cost us another round of the same page. The second time, we were told this exact case had already been settled once before, and that we were still applying a different rule than the one already agreed on. Only then did we look at what the alert was supposed to mean rather than what it was mechanically doing.

What shipped

The fix narrows the urgent category to fire only when the server that fails is one we actually operate. Everything else, including a customer’s own dead third-party host, still shows up in every scan, so nothing goes unreported. None of it pages as an emergency any more.

The change also added a way to mute a specific domain by hand with a reason attached. A known, expected case like a pending migration can be acknowledged once instead of re-litigated on every page.

The alert’s code was not wrong in isolation. Restating a trigger condition as an answer, instead of checking the assumption underneath it, is how a correct-sounding defense keeps a wrong alert alive.

Check your own pages

Look at each urgent alert in your monitoring and see whether it is limited to systems you operate, or whether it also pages for hosts outside your control. Make sure an alert’s severity reflects who can act on the problem, and that a matched technical condition alone doesn’t decide it. And give yourself a way to acknowledge a known, expected case once, with a reason, so the same page does not keep recurring.

dxdev LESSON · paste into your AI coding agent
LESSON: An Alert That Restates Its Own Logic Isn't the Same as Checking Whether the Target Is Actually Yours
SOURCE: dxdev.com/blog/an-outage-alert-that-kept-blaming-us-for-someone-elses-server
WHAT HAPPENED: A monitor built to catch a specific, real class of outage on infrastructure we operate kept firing its most urgent alert level against a customer's own third-party server, one that was simply down for reasons that had nothing to do with us, six times over nine days. Asked why that particular target got the urgent treatment, the first answer defended the alert's own trigger condition as correctly met, technically true, and missed the actual question, which was whether the target belonged to us at all. It didn't. The fix scoped the urgent category to only fire when the thing that failed was infrastructure we actually operate; everything else still gets reported, just never paged as an emergency.
THE RULE: When an alert's classification is challenged, check whether its target is actually within your control before defending the trigger logic that fired it; a condition can be technically satisfied and still be answering the wrong question.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any alert or monitor whose most urgent tier doesn't first check whether the failing target is something the alerting system's owner actually operates.
  2. Any incident classification that, when questioned, gets defended by restating the trigger condition rather than checking the underlying assumption.
  3. Any alert that has fired repeatedly against the same target with nobody asking whether the target itself, not the alert, is the real anomaly.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.