The method outlasts the patch

The ticket closed on the afternoon of April 23rd. The bot swarm was gone, CPU was back to baseline, and I had five new Windows Firewall rules to show for it. The next morning I reopened the session notes and spent two hours not looking for what I had missed, but looking at what I had done right by instinct and needed to do right on purpose next time.

That is the post. Not the attack. Not the five rules. The two hours after.

the investigation frame I kept getting wrong

The first thing I pulled up was the agent’s first read from the morning before. Claude had said: not a swarm, volume-driven load. It was working from the IIS slow-request log for the customer vanity-domain site. The actual swarm was hammering the direct-access site, and the detector could not see it. I covered that blind spot in the last post.

But there was a second wrong read, and this one was mine.

When I finally had the right log data in front of me, my instinct was to sort by IP volume. Top-ten IPs, who sent the most requests. That is the obvious cut. It is also the wrong one for a distributed bot swarm. A residential-proxy botnet or a cloud provider’s scrapers spread across dozens of sibling IPs in a single /16. Sorting by IP volume shows you the tip and hides the body.

The cut that actually worked was two signals together: which URL patterns were hit, and what percentage of requests on those patterns came from bad IP ranges. A legitimate user roster-loads /teams/?u=<teamname> once, maybe twice. The swarm hit it 4,000 times in 90 minutes from 19 unique IPs across a single /19 block registered to a known hosting provider. Sorted by IP volume, those 19 IPs individually look mild. Sorted by bad-range percentage against that URL, they are the only story in the data.

I had found the answer by running the right query accidentally. I wrote it down so I would not need to rediscover it.

writing the access path once

The other problem was memory. Not AI memory. Mine, and the agent’s in each new session.

The production app runs on an aging IIS box. To investigate a bot swarm properly you need four things: SSH access to the box, an ODBC connection to the SQL Server instance, the ability to tail IIS logs across three sites, and a raw curl channel to test responses without hitting any caching layer. None of those are hard to set up. All of them require remembering where the credentials live, which clone has the right SSH config, which DSN to use.

In the incident session I had spent 20 minutes reconstructing access that I had set up in a previous session and not documented. Claude was helpful but it was working from whatever context I had loaded into that specific window. Start a new window and start over.

So on April 24th I wrote a single reference: the SSH alias, the ODBC DSN name (which points at production, so read-only queries only unless you want to write live customer data), the IIS log directories for each site, and the curl invocation pattern for testing direct IIS response. One file, checked into the ops repo so the next session can read it instead of reconstructing it from context.

That is not glamorous. It is also not optional if you want AI-assisted ops to compound rather than reset.

the enforcement gap nobody told me about

The uncomfortable finding from the April 23rd cleanup was about the IP block table.

The production app has a table that stores blocked IP ranges with associated metadata: reason, date added, review status, whether it is active. There is an admin UI for it. It looks, from the inside, like the authoritative block list. If an IP is in that table with Active = 1, it is blocked.

It is not blocked.

The table feeds a SQL lookup that runs on every request and increments a counter. The counter is useful for audit purposes. The table does not affect whether the request gets served. What blocks a request is a Windows Firewall rule. The IP block table and the firewall ruleset have never been synchronized, as far as I could tell from the session notes.

I had assumed they were the same thing. They are two entirely separate systems that happen to live on the same box and share some IP data.

I filed a follow-up ticket to reconcile them. More importantly, I wrote the distinction into the operational reference so neither I nor any future agent session would mistake the UI for the enforcement layer again. In ops, something that looks enforced and is not enforced is worse than something that is visibly unenforced. You stop checking the thing you believe is handling it.

The parallel question is how many other places in the platform work this way. I do not know the answer. I have one confirmed example. The fact that I found this one by accident, after years of operating the system, suggests there are more.

folding it into a skill

The artifact from April 24th was a runbook skill. A markdown file that a Claude Code session loads when I invoke it. It opens with the access path. It says explicitly: rank by URL pattern and bad-range percentage before looking at per-IP volume. It names the IP block table and notes it is not the enforcement layer. It links to the firewall rule naming convention.

The next time I run it, the session starts with that frame instead of building one from scratch.

This is where the automation question gets interesting. How much of the triage flow can be scripted? I could write a query that runs on a schedule and pages me when bad-range percentage on roster URLs crosses a threshold. The reason I have not done it yet is that I do not know the right threshold. I have validated the heuristic once. If I automate a pattern I have only seen once, I teach the system a rule I may not have right. The manual version is slower but it keeps me calibrated. Automation should follow understanding, not precede it.

the fix is not done until the method is portable

The five firewall rules will hold until a different botnet shows up on a range I have not seen before. That is fine. Holding patterns are real strategies. What I was not fine with was leaving the investigation method as tribal knowledge scattered across one session’s context window.

If an incident taught you something real about how your system actually works under load, the fix is not finished when the symptoms stop. It is finished when you have written down the investigation frame so a future version of yourself, or a future agent session, can start in the right place instead of reconstructing the map from memory.

The knowledge is the durable asset. The firewall rule is a tally mark.

dxdev LESSON · paste into your AI coding agent
LESSON: Bot swarm: sort by URL-pattern bad-range percentage, not per-IP volume
SOURCE: dxdev.com/blog/2026-04-24_bot-attack-playbook
WHAT HAPPENED: A bot swarm was missed because logs from the attacked endpoint were never checked. When the right logs finally surfaced, sorting by per-IP request volume still showed each attacker as mild. The swarm was only visible by grouping on URL pattern first, then measuring what fraction of hits on that pattern came from known-bad IP ranges. A second finding: an admin UI for blocking IPs looked authoritative but was not wired to the actual enforcement layer. Assuming the UI was the block was wrong.
THE RULE: Rank bot candidates by bad-range percentage on a target URL pattern before looking at per-IP volume; and verify that every block UI is actually wired to an enforcement layer.
CHECK MY CODE, then report PASS / FAIL with file:line for each:
  1. Any IP-throttling or block logic that counts requests per IP without also grouping by URL pattern or route class.
  2. Any IP block table, admin UI, or in-app block list that is not confirmed to feed the actual request-rejection path (firewall rule, middleware, WAF deny rule, etc.).
  3. Any bot-detection log query or alert that reads from only one log source when the app serves traffic through more than one hostname or site binding.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.