At 9:58 AM on July 22, our status board was still raising false-down pages whenever one monitoring signal disagreed with another.
The service was not actually down. The fault was in how the board turned evidence into an alarm. A collector reported a problem, the live web probe did not confirm it, and the board treated the collector’s report as enough to announce an outage.
By 1:09 PM, three hours and 11 minutes after we started, production was running a different rule. The board now alarms only when the collector and the live probe agree that the service is down. We also gave it a queryable event log, so the next disagreement has a history to look at.
The alert had one sensor too many
The collector and the live probe were not redundant copies of the same check. They answered related questions from different places in the stack.
The collector supplied one view of service health. The live web probe tested what a user could reach on the web. On the bad pages those views split. One said down and the other did not. The board collapsed that split into a single down state.
That was the whole diagnostic path. The false-down symptom lined up with one fact: our alert decision did not preserve disagreement. It converted one negative signal into a page before the second signal had any say.
A monitoring system can be right about an individual observation and still wrong to escalate from it. A collector can surface a transient failure, a stale result, or a problem limited to its own path, while a live probe stays healthy through all of it. When the board reports those two observations as one unqualified outage, it hides the only fact that mattered, which is that the sensors disagreed.
Make the decision boundary explicit
The production change was small, but it changed what the alarm means. The rule is now:
alarm = collector_down AND live_probe_downThe collector still matters and so does the live probe. Neither can page by itself.
That leaves three states. If both report down, the board alarms. If both report healthy, it stays quiet. If one reports down and the other reports healthy, it records the event without claiming there is an outage.
Consensus was a narrow change. It does not pretend either signal is perfect. It asks for corroboration before an observation becomes an alarm.
The event log is part of the fix
Suppressing a false alarm is not enough if the evidence disappears with it. The mismatch is still operational data. It tells us the collector and the live probe did not see the same thing, and it gives us a starting point if the disagreement becomes a pattern.
The board now keeps those events in a queryable log for troubleshooting history. The next investigation will not start from a vague report that alerting was noisy. We can ask what each signal said at the time, find the disagreement, and decide whether the problem belongs in the collector, the probe, or the decision rule.
We deployed the change and verified it in production. The constant false-alarm path went to zero. We did not ignore the bad signal; we stopped letting one sensor speak for the entire system.
There was no new dashboard and no more complicated severity model. The change was to stop calling a disagreement an outage. If one sensor in your setup can page on its own, check how your alerting decides that something is down, because a split between signals will show up as a false outage.