I opened the IIS logs to do something boring: size a rate-limit rule. I wanted to know what a sane requests-per-second threshold looked like for the schedule-builder endpoint before I told Cloudflare to start throttling it. Two hours of logs later I had my answer, and a much worse problem. The number two source of traffic to that endpoint was the production server itself, hammering its own public hostname 19,178 times.

The site was DDoSing itself. The CDN I had just put in front of everything is what made it visible.

What the logs actually said

This is on a youth-sports SaaS that has been running on Windows, IIS, and classic ASP for years. The week before, I had flipped the nameservers to Cloudflare and orange-clouded the customer-facing hostnames. So now every request to the origin either came from a real visitor through Cloudflare, or from somewhere on the inside.

I grepped two hours of the IIS log for the production app’s site, filtered to the schedule-builder endpoint, and counted hits per client IP. 43,070 requests in two hours, to one endpoint. The top two:

  • An external IP at 23,892 hits. That’s a real scraper, an outside party pounding the schedule builder.
  • The production box’s own public IP, at 19,178 hits. The server was calling itself.

A scraper I understand. You block it, you rate-limit it, you move on. But the server generating nearly half the traffic to its own endpoint is not an attack. It’s a design that became pathological the moment the topology changed.

Why a server calls itself 19,000 times

The feature is calendar subscriptions. A parent wants their kid’s game schedule in Apple Calendar or Google Calendar, so they subscribe to an iCal feed for the team. Their device then polls that feed forever to stay in sync.

Here is the part that bites. The server-side iCal generator does not build the schedule by calling internal code directly. It fetches the schedule over HTTP, from the schedule-builder endpoint, using the public hostname. Before Cloudflare, that was a loopback that never left the building. The app’s public hostname resolved to the box itself, so the request went out the front door and right back in. Ugly, but cheap and invisible.

After the nameserver flip, the app’s public hostname resolves to Cloudflare. So every one of those self-calls now does a full round trip: origin to Cloudflare’s edge, back to the origin. The server is making outbound HTTPS requests to a CDN to reach a page that lives on the same machine, on the same disk, served by the same IIS.

Multiply that by every subscribed device, polling on its own schedule, and you get 19,178 self-inflicted edge round trips in two hours. The CDN dutifully proxies them, counts them as traffic, and applies the same rate limits and the same billing it would to a stranger. Internal traffic became external traffic the instant it started traversing the edge, and the iCal generator never got the memo.

The second leak: nobody ever stops asking

Digging into who was subscribed turned up the worse waste. A large chunk of this polling is for sites that are expired or unpaid. The season is over. The customer didn’t renew. But the calendar subscription a parent set up two years ago is still on their phone, still polling, every few hours, forever.

Calendar clients do not give up easily, and that is by design. They are built to survive a flaky connection. A subscribed feed is supposed to be durable. So when our endpoint kept answering, the clients kept asking. We were serving schedule data, over the edge, for teams that no longer existed, to devices that would never stop.

The reflex fix is to return an empty calendar for those sites. That is exactly the wrong move.

410 Gone is the kill switch an empty 200 will never be

If you return an empty 200 OK, you have told the client “everything is fine, there is just nothing here right now.” A well-behaved calendar client reads that as “the feed is healthy, keep polling, the games will show up later.” You have not stopped anything. You have confirmed the feed is alive and invited the device to come back tomorrow, and the day after, indefinitely.

410 Gone means something specific: this resource existed and is deliberately, permanently gone. It is not 404 (“never heard of it, maybe you have the wrong URL”) and it is not 503 (“temporarily down, retry later”). 410 is the one status that tells a polling client to stop asking and, ideally, to remove the subscription. It is the difference between “nothing today” and “do not come back.”

For a subscription or polling endpoint, 410 is the underused tool. Most of us reach for 404 or an empty success body, and both of those keep the client in the loop. If you actually want devices to quit, you have to say so in the status line, because the body is not where a calendar client makes that decision.

The two-layer fix

I filed this as two tickets, both under our maintenance epic:

  1. Return 410 Gone from the iCal endpoint for sites past their paid period. The key detail is that this is keyed on actual paid status, not some “inactive for 12 months” heuristic. We know whether a site is paid. That is the real signal, and it is the one a well-behaved client will respect by dropping the subscription.

  2. Kill the HTTPS self-loop. The iCal generator should build the schedule by calling the schedule builder internally instead of fetching its own public URL through the CDN. No DNS, no TLS handshake, no edge round trip, no Cloudflare request count, no rate-limit collision with itself.

Finding these in an old codebase is one grep. Look for any server-side HTTP call whose URL is your own public hostname:

grep -rERn 'http(client|webrequest)|fetch|xmlhttp|curl|server\.createobject\("msxml' . \
  | grep -iE 'https?://[^ "'']*yourdomain\.com'

Every hit is a candidate. Most are innocent calls to a third party. The dangerous one is the call that points back at the box it is running on.

These are independent and both worth doing. Even after you stop serving expired feeds, every live subscription is still a self-call. And even after you stop the self-call, you do not want to keep generating calendars for dead seasons. One fix shrinks the volume of self-traffic; the other tells the long tail of dead subscriptions to give up. Do one without the other and you have only half-solved it.