At 4:59 PM, a Cloudflare edge alert marked a site degraded because one probe came back 53 ms over the slow threshold. The site was up the whole time, and the next cycle was clean. Our alert logic had turned a one-cycle latency blip into a page.

We fixed it in 17 minutes by changing the status board so it treats a possible degradation and a confirmed outage as different events.

A slow response is evidence that something may be wrong. A failed availability check is evidence that something is wrong right now. Those states deserve different confirmation rules.

The page was accurate and still wrong

The board had a slow threshold and a degraded state. One probe exceeded the threshold, so the board reported degradation and paged. From the code’s point of view, it did exactly what it was told to do.

From an operator’s point of view, the alert was noise. There was no customer-visible outage. There was one measurement outside a line, followed by a normal measurement.

Latency is noisy in a way a hard availability failure is not. A probe can hit a slow edge, contend for a connection, or absorb a short network stall, and the response can be late without the application becoming unavailable. A board that pages on every isolated latency violation trains people to discount its degraded state, and then it has lost the job we built it to do.

The diagnostic path was short because the symptom was specific. We checked whether the site had actually gone down. It had not. We checked the alert event, and one probe was 53 ms over the old slow threshold. We checked the following cycle, and it recovered. That made the incident a one-cycle flap, not a sustained performance problem.

The first fix was only half a fix

The original threshold was also too tight for what we wanted “slow” to mean, so we raised it to 2500 ms. That reduces sensitivity, but it does not solve the alerting problem. A single response over 2500 ms can still be a transient, and with only the threshold changed it would still page on an isolated slow sample. The line had moved, but nothing yet separated a momentary measurement from a sustained condition. What needed to change was what a degraded sample is allowed to do.

Two bad cycles for degraded, one for down

The new policy is asymmetric on purpose:

slow_threshold_ms: 2500
page_on:
  down: first_failed_cycle
  degraded: second_consecutive_bad_cycle

A failed availability check still pages immediately. That is fail-closed behavior. If the board cannot reach a service, we want the first failed cycle to create an incident. Waiting for a second check would buy a little protection from probe noise, and it would add delay to the failure mode that needs the fastest response.

A slow response now starts a degraded streak instead of creating a page. A second consecutive slow cycle confirms it, and a clean cycle resets the streak. The single 53 ms excursion behind the 4:59 PM page would now show up as a measurement and disappear without escalating to the whole team.

In plain control flow:

if availability_check_failed:
    page_down_immediately
else if response_time_ms > 2500:
    increment_consecutive_degraded_cycles
    if consecutive_degraded_cycles >= 2:
        page_degraded
else:
    reset_consecutive_degraded_cycles

The ordering is part of the design. The down path is checked first and has no debounce. The degraded path is explicitly stateful. Folding both conditions into one generic “unhealthy” state would have erased the difference that matters.

Why not the other obvious fixes

Applying the same two-cycle rule to everything would have made the alert logic look consistent, and it would have made it worse in practice. A service that is actually down should not wait through another monitoring interval because one transient latency event was annoying.

A global alert cooldown would also cut noise, but it suppresses information indiscriminately. A real outage right after a benign flap could be hidden behind the same timer. Cooldowns control the delivery channel. Consecutive-cycle confirmation controls the evidence required for a specific state, and that is what we wanted.

The asymmetric rule is a little more complicated than a single threshold, but it is easy to read. Anyone looking at it can answer the question that matters: how much evidence does this state require before it wakes someone up?

Checking both sides live

After changing the threshold and the debounce logic, we verified the new behavior live. The status board still rendering was not the test. The policy had to keep both sides of the tradeoff: one bad latency cycle should not page, two consecutive bad cycles should, and a down condition should still page on the first failure.

For each alert, we now look at what kind of evidence each failure state produces, how often that evidence flaps, and what delay is acceptable before someone acts. Here, a down condition pages on the first failed cycle, and a degraded condition pages only on the second consecutive bad cycle.