---
title: "CPU Sat at 100% While Our Log Watcher Reported Nothing Wrong"
canonical: https://dxdev.com/blog/five-firewall-rules-blind-spot/
datePublished: 2026-05-26
---
When CPU pinned at 100% and stayed there for thirty minutes, we ran the log query we always run. It showed mixed load across customer vanity domains, nothing that screamed coordinated attack. That query only reads one IIS site, and that site was not where the attack was landing.

## What the traffic looked like

The operator was a [German proxy reseller](/blog/operator-not-ip-fingerprint-ripe-maintainer/). Five /19 CIDR ranges across non-adjacent address space, roughly 100 to 200 unique IPs per range, each contributing 50 to 60 hits against the same URL structure. The pattern was participant list enumeration against a single dynamic endpoint. Source IPs rotated to dodge per-IP rate limiting, no request carried a Referer header, and the User-Agent was a spoofed Chrome 134/135 with no Sec-CH-UA field.

Three signals [identify a distributed scraper](/blog/three-signal-triangle-bot-swarm-detection/): asset mix (are requests hitting static files at a realistic ratio, or drilling one endpoint?), UA homogeneity (does the user-agent distribution look like a real browser population?), and Referer presence (real organic navigation almost always carries a Referer after the first hit). This operator failed all three. Requests were purely dynamic-page with zero static files, Chrome 134/135 showed up across every IP, and Referer was absent on nearly every hit.

Five firewall rules at the TCP layer, blocking at the /19 level rather than per-IP, dropped CPU to 13% and open connections from 490 to 146. Once we were reading the right log file, the fix was straightforward. Getting to the right log file was the hard part.

## The bundle said everything was fine

A triage skill we use for slowness incidents surfaces a pre-captured incident bundle when we open an investigation. The bundle pulls from the log watcher script, which tails IIS logs to surface slow requests and top-offending IP ranges. We had relied on it for months.

The script only tails one IIS log directory. There are three IIS sites on the box: one serves customer vanity domains, one handles direct traffic to the production app's main domain, and one serves the media subdomain. The script watched one of them, and we had never audited its coverage.

When the investigation started, the bundle's slow-request output showed mixed URL fan-out across customers on the vanity-domain site. That looked like distributed volume load, not a targeted swarm. The agent read it the same way, and the read was accurate for that site.

## One question from a phone

Mid-SSH-session, from a phone, we sent: "are you checking both versions of the logs."

The investigation pivoted immediately. We pulled the direct-traffic site logs, and they held 49,112 slow requests. The whole load was concentrated there. The proxy reseller IPs were hitting the production app's main domain directly, and the vanity-domain log watcher had been faithfully telling us nothing was wrong.

Five firewall rules later, the attack was gone. The monitoring gap was still there.

## Closing the gap

We hardened the triage skill with a CRITICAL warning that names all three sites and their log directories. It sits at the top of the multi-site sweep step, so the next investigation cannot treat the vanity-domain bundle as comprehensive.

We added two persistent memory entries. The first captures the multi-site sweep requirement, with a description of each site and a note about the log watcher script's single-site scope. The second captures the proxy reseller's fingerprint: known ranges, URL pattern, UA signature, and the firewall naming convention. Memory entries serve the agents and code changes serve the system. Both matter, because entries can drift when the context resets.

We also logged two follow-ups. The first is the structural fix: rewrite the log watcher script to sweep all three sites and aggregate before producing a verdict. The second moves those IP ranges into the [application's IP block table](/blog/decorative-blocklist-table-never-consulted-at-enforcement/) so the application-layer filter can see them. Both exist in writing now so the work survives a context reset.

## Why the drop to 13% was not the end

When CPU dropped to 13%, the site was fast again, and we could have filed a one-line note and moved on. What the drop exposed was a different question: why had this attack stayed invisible until someone on a phone asked about the second log? The answer was a script pointed at one of three sites, trusted for months without anyone checking what it covered. The slow requests had been in the direct-traffic logs the whole time. The watcher just was not reading that file.

So ask what each monitoring tool actually reads, and check that it covers every site you run. An incident is finished when the gap that hid it is fixed and written down.