The CPU on the IIS box was sitting around 80%, the site was slow, and the loudest thing in the logs was one named crawler. AwarioBot, a brand-monitoring service, was walking listing pages in every sort permutation it could construct, one query-string variation after another. It had racked up roughly 1,600 hits from a Hetzner IP. We killed it with one line:

"awariobot":1

That went into the UA blocklist dictionary in the request filter. We added no firewall rule, no IP blacklist and no whois lookup. With one key in a dict, every later request carrying that user-agent got a 302 straight to the error page before it touched the expensive page-render path. CPU came back down, and the visible symptom was gone.

The same incident also had us in a firewall-CIDR fight against a completely different attacker on the same box, a distributed scraping swarm that no user-agent rule could have touched. Bot mitigation is a tiered decision. Pick the wrong tier for the shape of the attacker in front of you and you either miss the attack entirely or block real paying customers. The UA blocklist was exactly right for AwarioBot and would have been useless against the swarm.

Why the UA block fit AwarioBot

Three properties make AwarioBot a textbook UA-blocklist target:

  1. It is identified. It announces itself in the user-agent string. It is a real, named, commercial crawler with a fixed UA.
  2. The UA is the stable identifier, and the block is source-agnostic. A UA rule fires on the header, so it doesn’t matter whether the crawler hits you from one IP or rotates through a hosting provider’s whole block. AwarioBot here was concentrated on a single Hetzner IP, but we never had to care, because the rule keys on the one thing the crawler keeps constant: its name.
  3. A false positive costs almost nothing. Nobody legitimate is browsing a registration site with awariobot in their user-agent. We were never going to 302 a paying customer making a legitimate request.

When all three hold, the user-agent string is the cheapest stable identifier you have. The same pattern already protects the box against a handful of other named crawlers, and AwarioBot just joined the list. The request filter checks the incoming UA against the dict and bounces matches before any real work happens. That early rejection is the performance argument for blocking at the UA layer instead of the IP-filter layer: the request is turned away before the page does any database work to build the response.

The IP that should already have been blocked

The AwarioBot IP was sitting inside a range already flagged as bad in the IP reputation data. In principle the IP filter one layer down should have caught it, and we would never have needed a UA rule.

It got through because of a decay bug. The IP filter had a secondary tier, added in an earlier pass, that auto-cleared its per-IP blacklist entries after 7 days. An IP it had correctly blocked was silently un-blocked a week later, and the crawler sailed back in. The second half of this fix was removing that 7-day auto-decay so a deliberately blocked IP stays blocked. The UA rule stopped the bleeding, and removing the decay closed the door the IP had walked through.

There was a deeper hole too. That filter only ever checks the per-IP blacklist flag and never consults the broader IP range data at all, so range membership was intelligence the request path didn’t enforce.

A blocklist that auto-expires its own entries is only a suggestion with a timer. Decay belongs on organic noise. Confirmed abusers should stay flagged until a human un-flags them.

The other attacker that night

The other half of the incident was a distributed scraping swarm, and its signature looked nothing like AwarioBot’s. There were 55-plus unique IPs, each hitting a single page only one or two times, almost all of them inside the same hosting ASN (Alibaba Cloud). Rank that traffic by hits-per-IP, the way the classic “block the loudest IP” playbook says to, and you find nothing. No single IP is loud. The volume is spread thin across dozens of sources, so per-IP analysis sees noise.

A UA block would have done nothing here. Swarms either rotate UA strings or send a generic real-browser UA, so there is no awariobot:1 line to write. The header holds no stable identifier to key on.

The right tier was the firewall, blocking by CIDR. One Windows Firewall rule against the hosting provider’s published range took the box from roughly 73% CPU and ~1000 connections down to 11% CPU and ~128 connections in seconds. A TCP-dropped packet costs the IIS process zero CPU, because the connection never reaches the application. The UA block still processes the request far enough to issue the 302, which is cheap but not free. Under a real flood you want the packet dropped at the firewall, not bounced at the app.

The firewall tier has its own trap, which is why it is the heavier tool. Within hours of blocking that range, the swarm rotated to a neighboring block that wasn’t in our data at all. We assumed it was more of the same provider’s space. It wasn’t. The public allocation registry (RDAP) showed the new block belonged to a completely different cloud provider. The rotation had crossed providers, and our model of “block the attacker’s ASN” was wrong before we finished typing the rule. Our blocklist was built from the traffic we happened to see, and the attacker was not confined to it. Blocking by range means consulting the actual public allocations, not our own stale subset.

You also have to watch where you swing. Some blocks that look like clean hosting-provider space sit right next to carrier mobile ranges, and a lazy wide block takes out real customers on their phones. That false-positive cost doesn’t exist when you block awariobot by UA. The firewall tier is powerful and dangerous, and you reach for it only when the attacker’s shape forces you to.

Name the identifier first

Before you write any rule, answer one question: what stays constant about this attacker? If it’s the user-agent, block the UA. If it’s the netblock, block the CIDR. If it’s a single IP, block the IP.

The failure runs in both directions. Reach for the firewall against AwarioBot and you start blocking Hetzner CIDRs and risking real users on shared hosting, when one dict key would have ended it. Reach for the UA blocklist against the swarm and you spend an afternoon writing rules that match nothing, because the attacker has no stable string to key on.

The same night this incident closed, we wrote down the triage check we now run first. Name the attacker’s stable identifier (a UA string, a single IP, or a CIDR), then confirm that the filter meant to key on it is actually enforcing. The 7-day decay bug showed that an IP filter can look like it works and still stop doing its job. If the identifier and the enforcement don’t line up, fix that before writing a new rule.