The fastest fix was blocking the bot by name

CPU on the production app’s box climbed to 80 percent and held for about twenty minutes before I dug into the session log. The slow-request log showed roster sort pages: the same URL pattern cycling through sort permutations, request after request from the same address block. User agent: AwarioBot.

AwarioBot is a brand-monitoring crawler. Not malicious, just indifferent to the cost it imposes on a shared ASP-classic box. It was requesting the same logical page repeatedly with different sort parameters, generating a full database round-trip for each one.

My first reflex was to pull the IP range and block it at the firewall. I asked Claude to surface the originating block. Then I checked the blocklist.

The range was already in there.

The seam that let it through

The production app’s bot-blocking has two layers. The first is a firewall-level IP blacklist, maintained in a flat file and pushed to the box periodically. The second is a per-request filter file that runs on every request and handles dirty user agents, IP ranges flagged for review, and the connection-error redirect logic. These two layers are supposed to be additive. They’re maintained separately and had never been audited against each other.

The second layer had an auto-clear rule. IP ranges showing clean activity over a rolling window got flagged as reviewed and removed from the active block queue. AwarioBot’s range had hit roster pages roughly six months before this incident, generated no 500s, no obvious abuse pattern. The rolling-window pass marked it reviewed and cleared it from the per-request queue. The firewall entry stayed. The per-request check moved on.

Claude’s initial read was correct about AwarioBot being the load driver. It suggested tightening the IP-based block at the firewall level. That would have worked until the auto-clear ran again, and then failed the same way. The seam doesn’t appear in any log line. That’s why the IP fix looked correct.

The fix that was actually correct

The answer was simpler, and uglier. I added awariobot to the dirty user-agent array in the per-request filter and redirected matching requests to the existing connection error path, the same one other blocked bots hit. Thirty seconds of work.

On paper this is crude. A user-agent string is trivially spoofable. But AwarioBot is not a determined crawler. It is a commercial product with a published user-agent convention and no reason to disguise itself to scrape roster sort pages. The UA match is the right level of force here.

The bigger reason it was correct: it works independently of the IP layer. The IP blacklist and the auto-review queue can be in any state they want. The UA check fires first in the per-request filter, before any IP logic runs, and the redirect exits the request immediately. It works without requiring either broken layer to be fixed first.

CPU came back down to 35 percent within two minutes.

What this revealed about the layers

The IP blacklist and the per-request filter were designed to reinforce each other. They do, in the general case. But each one has its own maintenance logic, its own update cadence, its own definition of “this range is now safe.” Neither layer knows what the other is doing.

A seam like this stays invisible until something crosses it. Mine showed up when a crawler blocked six months earlier reappeared from the same range after the auto-review queue had quietly made the block meaningless. There is no log line that says “this range is now covered by one layer only.” The seam only becomes visible when something breaks through it.

I added the UA guard and moved on. The right time to audit the gap between those two layers is a quiet afternoon with the incident log as the spec, not while CPU is pegged. The dirty-UA array in the per-request filter is not elegant. It is also the only thing that fired before either broken layer could matter, which is the reason the box is back to 35 percent.

dxdev LESSON · paste into your AI coding agent
LESSON: Block by identifier when the IP layer has its own maintenance clock
SOURCE: dxdev.com/blog/2026-04-23_block-the-bot-by-name
WHAT HAPPENED: A production app had two bot-blocking layers: a firewall IP blacklist and a per-request filter. The per-request filter had an auto-clear rule that removed IP ranges showing clean activity over a rolling window. A crawler that had been blocked at the IP level was quietly cleared from the per-request queue after a clean-activity window. When it returned, the IP blacklist entry was still present but the per-request check no longer fired, and the crawler drove CPU to 80 percent. The AI correctly identified the load driver but suggested tightening the IP block, which would have failed again at the next auto-clear cycle. The seam between two independently maintained layers was invisible in any single log line.
THE RULE: When two defense layers have independent maintenance clocks, add an identifier-based check that fires before both and exits the request without depending on either layer's state.
CHECK MY CODE, then report PASS / FAIL with file:line for each:
  1. Any place where a block decision is made: does it rely solely on IP matching or another value that a separate maintenance process can silently clear or expire?
  2. Any layered filtering setup: is there an audit trail or cross-reference that surfaces when one layer has cleared a range the other still considers blocked?
  3. Any auto-review or auto-clear rule applied to a security list: does clearing from one list also clear from all layers, or can layers fall out of sync silently?
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.