Why Our Own Firewall Could Not Stop the Bot Swarm at 2am

The phone rang in the middle of the night, and we already knew what we would see when we opened a laptop. The site would be moving like it was stuck in mud. People who paid to use it would be waiting for pages that would not load. Somewhere, a large group of automated visitors would be hammering at the door.

The calls came nearly every day for a stretch that spring. They did not arrive as polite emails we could read over coffee. They came at whatever hour the traffic showed up. Sometimes we were at a keyboard at 2am, trying to fix a live problem while the thing we needed to fix was already struggling to stay awake.

The visitors were bots, which just means computer programs visiting a site instead of people. Some bots are useful. Search engines use them to find pages. The ones showing up here were not useful. They arrived in swarms, changed their behavior quickly, and made it hard for normal customers to get through.

We had protections already. We had not left the front door wide open and then acted surprised when a crowd came in. Over years, we had added layers meant to catch bad traffic, and they had handled the earlier problems. This new wave slipped through them anyway.

Getting faster at the scramble

For too long, we tried to get faster at the scramble. We kept examples of rules we could reuse. We got better at spotting a pattern. We tightened the protections already on the site. Each time the phone rang, we logged into the production server, the computer doing the work for customers, and wrote another rule there.

That did not solve the problem. By the time we had a rule ready, the swarm had often changed its addresses or its calling card. We would block what we had just seen, then watch the crowd arrive wearing a different coat a few hours later. We lost sleep. Customers lost time and patience because their pages could not load. And every late-night fix rested on the same bad idea: we were trying to reinforce a door while the crowd was already pushing it off its hinges.

The kind of rule we kept writing belongs to something called a firewall. It is simply a gatekeeper that decides who gets through. The trouble was where our gatekeeper lived. It sat inside the same machine the crowd was overwhelming.

That meant every unwanted visitor still got close enough to take something before the gatekeeper could say no. Each one used a connection and a little of the computer’s attention. One visitor was never the problem. A crowd of them was. Even a perfect rule could not give back the pieces that had already been spent.

Moving the gate

The change was moving the gate.

We put a service called Cloudflare in front of the site. Think of it as an outside checkpoint, far away from the shop itself. When traffic arrives, the checkpoint can decide whether to let it continue before it reaches the computer doing the customer work. If an unwanted crowd shows up, the crowded machine no longer has to meet every person at the door first.

That new location changed the feel of an incident. Before, stopping a swarm meant logging into the wounded machine. After, it meant making a change from a healthy laptop. The new instruction could reach the outside checkpoints in less than a minute, without spending any of that minute asking the overloaded site to do more work.

We can now describe the pattern we are seeing to an AI agent. It turns that description into a proposed instruction, sends it to the outside checkpoint, and confirms that the block took effect. That is useful speed, especially in the middle of the night. But the important decision is still ours. We have to be sure we are looking at a harmful pattern and not an ordinary customer or a verified search crawler. Fast help only helps when the person using it knows what should not be shut out.

Not finished

This is not done. Thousands of older customer addresses still need to move behind that outside checkpoint. Until they do, not every site has the same protection. When the phone rings, we still check a list to see which side of the move that site is on.

What changed is where the gate stands. Stopping a swarm used to mean logging into the production server the crowd was overwhelming to write one more rule. Now a block reaches the Cloudflare checkpoints in less than a minute from a healthy laptop, and it is usually a few minutes of work we can forget by morning.

Our firewall used to run on the same machine the crowd was overwhelming, so every unwanted visitor used some of that machine’s connection and attention before it could be blocked. If you run a site, find out where your first line of defense runs, and which of your sites are already behind an outside checkpoint and which are not.

dxdev LESSON · paste into your AI coding agent
LESSON: When an attack overwhelms a machine, move the defense to a checkpoint in front of it instead of writing faster rules on the machine under load
SOURCE: dxdev.com/everyday-ai/night-i-stopped-fighting-at-the-door
WHAT HAPPENED: For a stretch one spring, swarms of bots slowed a live customer site nearly every day, with phone calls at all hours. Layers of protection built over years missed the new wave. Each time, we logged into the production server and wrote another rule there, but by the time a rule was ready the swarm had changed its addresses or its calling card. The firewall lived on the same machine the crowd was overwhelming, so every unwanted visitor still used a connection and some of the computer's attention before being refused. The change was to put Cloudflare in front of the site, so a new block reaches the outside checkpoints in less than a minute from a healthy laptop without asking the overloaded site to do more work. An AI agent can turn a described pattern into a proposed rule, send it, and confirm it took effect, while a person decides whether the pattern is harmful and not an ordinary customer or a verified search crawler. Thousands of older customer addresses were not yet moved behind the checkpoint.
THE RULE: A defense should not live on the machine it is protecting while that machine is under load. Put the first checkpoint outside it, and keep a person deciding what counts as harmful before an automated block is sent.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any incident response step that requires logging into the overloaded production machine to write or apply a block.
  2. Any bot or abuse protection that runs only on the origin server, so unwanted requests still consume its connections and CPU before being refused.
  3. Any automated block rule generated from a described pattern with no human review of whether it could block real visitors or verified search crawlers.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.