A 92% drop in error-minutes stalled at 670 failed requests a day

The blank error pages had been going on for weeks before anyone tied a number to them: 39 error-minutes a day, averaged over a 7-day window, out of 41 million requests. That ratio is why it hid so long. 40,000 bad requests out of 41 million rounds to nothing on a dashboard. But those bad requests landed in clusters, and if you were browsing during one, the site was just down.

A customer’s “site is down” email forced the hunt. We ran four passes in parallel: log analysis on the origin, live traffic sampling, a Cloudflare analytics pull, and a counter watch deployed straight onto the box. The origin came back clean every time. Zero 5xx from the app, p95 latency actually dropped during the biggest burst, no app-pool recycles, no crashes. Whatever was breaking wasn’t the server.

The wrong turn: a rate limit, fast

Once the shape looked like floods overwhelming a shared front door, the obvious fix was a Cloudflare rate-limiting rule: block or challenge any IP over some threshold. It’s a Pro-plan feature, it would have stopped every flood we’d observed, and it’s a five-minute change. We nearly shipped it on that logic alone.

Then we remembered the WAF had already done this kind of damage. A prior rule had silently 403’d server-to-server webhook POSTs. That traffic had an empty user agent and never carried a browser signature, so it all looked like bots. It was a payment processor’s asynchronous callback webhooks. The rule sat live for a while before anyone noticed payment confirmations were failing, because a blocked webhook doesn’t throw an error a human sees. It just never arrives.

So instead of shipping the rate limit, we ran it against real traffic first. Two studies, read-only, no config touched. One baselined what legitimate per-IP burst rates look like. The platform’s audience is schools and organizations, so 200 parents on one shared wifi network hitting refresh during an event is a real, legitimate burst, not an attack. The other checked known server-to-server callers (payment callbacks, calendar feeds, importers) for the low-UA-diversity, fixed-IP, bursty pattern that a naive threshold would flag as abuse. Had we shipped the rule on the first instinct, it would have broken exactly the traffic the WAF had already broken once.

What the flood actually was

The counter watch pinned it. During a 6,948-request burst, 74% came from a single IP hammering an internal error route in a redirect loop, at up to 353 req/sec. That sat on top of a WordPress REST scan sweeping dozens of vanity domains at exactly 72 hits each. Cloudflare passed all of it straight to origin. About 6% of the surge got reset (520) or timed out (522) at the connection layer before it ever reached IIS. The W3C logs never showed it, because a dropped TCP handshake never gets logged on the app side. That’s why the origin looked healthy the whole time: the failures were happening upstream of it.

The fix that shipped was staged rather than a single rule. First came WAF skips for an SMS provider’s and a payment processor’s callback paths. Then .php path blocking, a flat block and not a challenge, because challenging 7.5 million requests a week would add load at the exact bottleneck we were trying to relieve. Then a per-IP rate limit on that error route.

A daily error-minutes metric went in alongside it. It counts distinct minutes per day with at least one edge 5xx on a main-site host, not raw request counts, because request counts are what hid this for weeks. Baseline is 39.4/day, the alert threshold is 15, and it pages only on state transitions, so a quiet week stays silent and a standing problem doesn’t re-page every morning.

Three sources, one shared ceiling

The edge rules cut error-minutes by 92% almost immediately. Then they stalled, and 670 failed requests a day kept showing up. Three independent things were all filling the same finite burst capacity on the web box, which has no headroom to absorb concurrent spikes regardless of source.

Two of the three showed up in the leftover 670. 76% of it was two identifiable scanners. One was flagged by user agent (l9scan/2.0, all from a single ASN, 35,971 requests since the rules shipped). The other was unmatched probe paths like /getcmd and /aaa9, which generated zero firewall events because nothing in the ruleset described them.

The third source wasn’t a scanner. There was a hand-run SQL sweep against Query Store. Separately, a calendar-feed retry storm hammered .ics endpoints when an upstream skip rule missed ruleset: current scoping and let a legitimate feed get 403’d into retrying itself into a flood.

Customers hit by the scanner bursts saw their error rate drop once those rules landed. Customers caught in the calendar retry storm, or riding behind the SQL sweep, saw nothing improve, because their burst was never the scanner’s burst. Same symptom, same dashboard metric, three unrelated causes sharing one ceiling. Six follow-up tickets came out of that pass, one per source, because a fix that helps most customers doesn’t help the ones behind a different queue.

Measure outages in error-minutes per day, not raw request counts. Run any firewall or rate-limit rule read-only against real traffic before shipping it, because an earlier rule had already blocked a payment processor’s callbacks without anyone seeing an error. And when a fix helps some customers and not others, look for more than one cause sharing the same capacity limit.

dxdev LESSON · paste into your AI coding agent
LESSON: Separate Symptom From Cause Before Auditing Shared Bottleneck Capacity
SOURCE: dxdev.com/blog/platform-slowdown-compound-burst-causes
WHAT HAPPENED: A tiny error ratio (40,000 bad requests out of 41 million) hid clustered outages until a customer's downtime email forced a four-pronged investigation, which found the origin server was completely healthy the whole time. The near-fix was a Cloudflare rate-limiting rule, almost shipped in five minutes, until the team recalled a prior WAF rule that had silently blocked payment webhook callbacks because they looked like bot traffic. Testing first against real traffic revealed the actual causes: a single IP looping on an error page, a WordPress scanner sweep, and connection-layer drops invisible to app logs. After staged fixes cut error-minutes by 92%, a leftover 670 failed requests a day turned out to come from two more scanners and a misconfigured calendar-feed retry storm, three separate causes all sharing the same burst ceiling on the same box.
THE RULE: When multiple independent traffic sources share one finite capacity ceiling, the same dashboard symptom can have several unrelated causes, so a fix that helps most of the traffic can leave other sources completely unaddressed. Before changing a blocking or rate-limiting rule, test it against real legitimate traffic patterns first, especially server-to-server and low-signal callers that look like abuse but aren't.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Before deploying any new rate-limit, block, or WAF rule, has it been tested read-only against a sample of known legitimate server-to-server callers (webhooks, calendar feeds, importers) for patterns that could be mistaken for abuse?
      2. Does the error-rate or health metric track distinct affected time windows (e.g. minutes with at least one failure) rather than raw request-count ratios, which can hide clustered outages inside a large denominator?
      3. When an error metric partially improves after a fix, is the remaining error volume broken down by distinct cause rather than assumed to be more of the same problem?
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.