--- title: "Two hours of notes after the ticket closed on April 23rd" canonical: https://dxdev.com/blog/bot-attack-playbook/ datePublished: 2026-04-25 --- The ticket closed on the afternoon of April 23rd. The bot swarm was gone, CPU was back to baseline, and we had five new Windows Firewall rules to show for it. The next morning we reopened the session notes and spent two hours going through what we had done right by instinct and needed to do right on purpose next time. This post is about those two hours. ## Two wrong reads, one of them ours The first thing we pulled up was the agent's first read from the morning before. Claude had said: not a swarm, volume-driven load. It was working from the IIS slow-request log for the customer vanity-domain site. The swarm was hammering the direct-access site, and the detector could not see it. We covered that blind spot in the last post. There was a second wrong read, and this one was ours. When we finally had the right log data in front of us, our instinct was to sort by IP volume. Top-ten IPs, who sent the most requests. It is the obvious cut, and it is the wrong one for a distributed bot swarm. A residential-proxy botnet or a cloud provider's scrapers spread across dozens of sibling IPs in a single /16. Sorting by IP volume shows you the tip and hides the body. The cut that worked used two signals together: which URL patterns were hit, and what percentage of requests on those patterns came from bad IP ranges. A legitimate user loads `/accounts/?u=` once, maybe twice. The swarm hit it 4,000 times in 90 minutes from 19 unique IPs across a single /19 block registered to a known hosting provider. Sorted by IP volume, those 19 IPs individually look mild. Sorted by bad-range percentage against that URL, they are the only story in the data. We found the answer by running the right query accidentally. We wrote it down so we would never have to rediscover it. ## Writing the access path once The other problem was memory, ours and the agent's in each new session. The production app runs on an aging IIS box. To investigate a bot swarm properly you need four things: SSH access to the box, an ODBC connection to the SQL Server instance, the ability to tail IIS logs across three sites, and a raw curl channel to test responses without hitting any caching layer. None of those are hard to set up. All of them require remembering where the credentials live, which clone has the right SSH config, and which DSN to use. In the incident session we spent 20 minutes reconstructing access that we had set up in a previous session and never documented. Claude was helpful, but it was working from whatever context we had loaded into that specific window. Open a new window and you start over. So on April 24th we wrote a single reference: the SSH alias, the ODBC DSN name (which points at production, so read-only queries only unless you want to write live customer data), the IIS log directories for each site, and the curl invocation pattern for testing direct IIS response. One file, checked into the ops repo, so the next session can read it and skip the reconstruction. That is not glamorous. It is also not optional if you want AI-assisted ops to compound and not reset. ## The block table that blocks nothing The uncomfortable finding from the April 23rd cleanup was about the IP block table. The production app has a table that stores blocked IP ranges with metadata: reason, date added, review status, whether it is active. There is an admin UI for it. From the inside it looks like the authoritative block list. If an IP is in that table with `Active = 1`, it is blocked. It is not blocked. The table feeds a SQL lookup that runs on every request and increments a counter. The counter is useful for audit purposes. The table does not affect whether the request gets served. What blocks a request is a Windows Firewall rule. As far as we could tell from the session notes, the IP block table and the firewall ruleset have never been synchronized. We had assumed they were the same thing. They are two entirely separate systems that happen to live on the same box and share some IP data. We filed a follow-up ticket to reconcile them. More importantly, we wrote the distinction into the operational reference so that neither we nor any future agent session would mistake the UI for the enforcement layer again. In ops, something that looks enforced and is not enforced is worse than something visibly unenforced. You stop checking the thing you believe is handling it. How many other places in the platform work this way? We do not know. We have one confirmed example, and the fact that we found it by accident suggests there are more. ## Folding it into a skill The artifact from April 24th was a runbook skill: a markdown file that a Claude Code session loads when we invoke it. It opens with the access path. It says explicitly to rank by URL pattern and bad-range percentage before looking at per-IP volume. It names the IP block table and notes that it is not the enforcement layer. It links to the firewall rule naming convention. The next time we run it, the session starts with that frame already in place. This is where the automation question gets interesting. How much of the triage flow can be scripted? We could write a query that runs on a schedule and pages us when bad-range percentage on those URLs crosses a threshold. We have not done it yet because we do not know the right threshold. We have validated the heuristic once, and if we automate a pattern we have only seen once, we teach the system a rule we may not have right. The manual version is slower, but it keeps us calibrated. Automation should follow understanding, not precede it. ## The fix is finished when the method travels The five firewall rules will hold until a different botnet shows up on a range we have not seen before. That is fine. [Holding patterns are real strategies](/blog/whack-a-mole-firewall-block-legitimate-triage/). What we were not fine with was leaving the investigation method as tribal knowledge scattered across one session's context window. If an incident taught you something real about how your system behaves under load, the fix is finished when you have written down the investigation frame, so a future version of yourself or a future agent session can start in the right place and does not have to rebuild the map from memory. The knowledge is the durable asset. The firewall rule is a tally mark. ## What to check after an incident Ask whether the investigation frame and the access path are written down. Then check whether any admin screen that looks like a control actually enforces anything. Here the IP block table only counted requests, and Windows Firewall rules did the blocking.