A monitoring system paged the whole team because one single check came back a little slower than its threshold allowed. The very next check was completely normal. Nothing had actually gone down. The alert had turned one slightly slow moment into a full interruption.
What fixed it was seeing that a possible problem and a confirmed one are different kinds of evidence. We had been answering both the same way.
The alert was correct, and it was still noise
The system had a line for “slow,” and one reading crossed it, so it reported a problem and paged. From the system’s point of view, nothing was wrong. It did exactly what it had been told to do.
For the people who got interrupted, it was pure noise. Nothing was broken. There was one measurement outside a line, followed immediately by a normal one.
A slow response can happen for harmless reasons, like a brief delay or a momentary bottleneck somewhere upstream, without the thing being watched becoming unavailable. A system that pages on every borderline reading teaches people to stop taking its warnings seriously. Once that happens, it has stopped doing the job it was built for.
Two kinds of bad news, two different reactions
We made the response uneven on purpose. A hard failure, something that doesn’t respond at all, still triggers an immediate alert. If something has stopped working, that is the situation that needs the fastest possible response. Waiting for a second check just to be extra sure would add delay to the one kind of failure that can least afford it.
A soft warning sign, like one slow but still-working response, is handled differently. It starts a streak instead of an immediate page. Only when it happens twice in a row does it become a real alert, and one normal reading in between resets the streak to nothing. Under the new rule, the single slow moment that caused the original interruption would have been a measurement that came and went.
The simplest response was to make the “slow” threshold more forgiving and leave everything else alone. That helps a little. But one unlucky reading above any threshold, however generous, can still trigger the same pointless interruption. It moves the line without fixing the problem.
Applying the same two-strikes rule to every kind of bad signal, hard failures included, would have been consistent but wrong. Something that is actually down should not have to wait through a second check because a separate kind of alert had been annoying people lately.
A blanket rule that mutes or delays all alerts for a while after one goes off would have made things quieter too. The cost is that it could hide a real, separate problem that showed up shortly after an unrelated one had already resolved itself.
What to check in your own setup
Look at whether your monitoring pages a person on a single borderline reading. A hard failure should still alert immediately. A soft warning, such as one slow response, should need two in a row before it interrupts anyone. And a normal reading in between should reset the count.
AI Skills
Use this lesson with the AI assistant you already use
A single slow response, barely over a threshold, paged the whole team as if the service were actually down. The very next check came back completely normal. Treating a warning sign and a confirmed failure the same way is what trains people to stop trusting alerts at all.
Paste the prompt, share only the context needed to answer it, and treat the result as a draft for your review. Do not include confidential information or let an AI assistant make changes without your approval.
Optional: for a visual report and saved memory, run /dxdev first.
Don’t have it? Get it at dxdev.com/skills/dxdev. The prompt works without it.
dxdev LESSON · paste into your AI coding agent
LESSON: A possible problem and a confirmed one deserve different amounts of confirmation before anyone gets interrupted
SOURCE: dxdev.com/everyday-ai/not-every-warning-deserves-the-same-reaction
WHAT HAPPENED: A monitoring system paged the team because one single check came back slightly slower than its threshold, and the very next check was completely normal, with no actual outage having occurred. Simply raising the threshold would not have solved the underlying issue, since any single slow reading above a new, higher line could still trigger the same page. The real fix made the alerting rule explicitly asymmetric: a hard failure, something that does not respond at all, still pages immediately, because that is the failure mode that most needs a fast response. A soft warning sign, like one slow-but-successful response, only pages after it happens twice in a row, and a single normal reading in between resets it back to zero.
THE RULE: A hard failure and a soft warning sign are different kinds of evidence, and they need different amounts of confirmation before they interrupt a person. Treating every unhealthy signal the same way, with one blanket rule, either delays real problems or trains people to ignore the alert altogether.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
1. Any alerting or notification rule that treats a single borderline reading the same as a confirmed, sustained failure.
2. Any fix to a noisy alert that only adjusted a threshold number, without also asking how much confirming evidence that kind of alert should require before it interrupts someone.
3. Any blanket rule that suppresses or delays all alerts equally, risking a real, separate problem getting hidden behind an unrelated, already-resolved one.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.
