Column 6 of the IIS log on the production box was c-ip, and it held a Cloudflare edge IP on every single row. The #Fields: directive at the top of the file said so. Reading it was the only reliable way we found the bug. The log-analysis script had been reading that column for weeks. Every “top offender IP” report since the nameservers moved was a ranked list of Cloudflare datacenters, and the scrapers we wanted to rate-limit sat invisible behind them.

Once your nameservers flip to Cloudflare, every IP your origin sees is a Cloudflare edge IP. Your log analyzer, your bot detection, and the “here’s your IP address” page your support team reads to a confused customer all keep working. They just report a Cloudflare datacenter instead of the actual human. Nothing throws an error.

A wrong IP is still a valid-looking IP. There is no exception, no 500, no red line in a log. The records kept flowing, and everything downstream of REMOTE_ADDR had started lying.

This is the postmortem of finding it, on a Windows/IIS live SaaS platform that has accumulated years of code and operational assumptions, after orange-clouding the whole site.

The flip

We moved the nameservers to the two Cloudflare-assigned *.ns.cloudflare.com hosts and orange-clouded everything: www, apex, and the media subdomain. From that second on, the origin web server (IIS) no longer talks to browsers. It talks to Cloudflare’s edge. The browser’s real IP arrives in an HTTP header, CF-Connecting-IP, but only if something on your side actually reads it.

We knew that going in, sort of. An IPv6 hotfix earlier in the same release cycle had already taught the application layer to read the right header. Several application files got an IP resolution helper with a clear precedence:

CF_PSEUDO_IPV4  →  CF_CONNECTING_IP  →  REMOTE_ADDR

Cloudflare hands you a synthetic IPv4 (CF-Pseudo-IPv4) for IPv6 visitors so legacy code that assumes a v4 string does not choke. If that is present, use it. Otherwise use the real connecting IP. Only fall back to REMOTE_ADDR (which is now always a Cloudflare edge) when nothing better exists. The database layer that logged client IPs got the same treatment. As far as the application was concerned, the IP was correct again.

It wasn’t. The application was not the only thing reading client IPs.

What the hotfix never touched

The app got fixed. The ops scripts did not, because nobody told them the world had changed. Two of them mattered:

  • A traffic-watch script that tails IIS traffic for live monitoring
  • A log-analysis script that produces the “top offender IP” reports we use to size rate limits and spot scrapers

These do not read HTTP headers through the application. They read the raw IIS log files on disk, and an IIS log file has a fixed column order. If your code reaches for a hardcoded column index, it does not care what header arrives. It grabs whatever is sitting in that position.

So we went to the production box and read the top of a real log file. IIS W3C logs carry a #Fields: directive that names every column in order, and it laid out the problem:

  • Column 6 (counting from zero, the way the script indexed) is c-ip, the connecting IP. After the flip, that is the Cloudflare edge IP every time.
  • Column 13 is CF-Connecting-IP, the real human, captured as a custom logged header.

The script had been reading column 6 since the flip. The scrapers, the swarm, the abusive ASN you actually wanted to rate-limit were all collapsed behind a handful of CF edge IPs. The report still ran. It still produced a clean, sorted, confident table of garbage.

Nothing alerted on this because column 6 was never empty and never malformed. It held a real, routable, valid IPv4 address the entire time. It was just the wrong one.

The fix, and the part that bites you twice

The application precedence wanted three things in order: pseudo-IPv4, then connecting-IP, then remote-addr. The logs only had two of those columns. So the first move was to add CF-Pseudo-IPv4 as a new logged field. On Windows that is an IIS Enhanced Logging change, applied per site with appcmd / the WebAdministration module, across the three customer-facing sites: the main site, the media subdomain, and the client-SSL site.

That landed CF-Pseudo-IPv4 as column 14. Then the analyzer got pointed at column 13 (the connecting IP), with column 14 preferred when present. An IPv4 visitor shows - in the pseudo column and falls back correctly to the real connecting IP. That - is its own little tell that the field is wired right, since the pseudo column only fills in for IPv6 visitors.

Here is the part that will catch you. A new IIS log field does not take effect retroactively, and it does not take effect the instant you save the config. The new column only appears when the worker process recycles or the log rotates at midnight UTC. So force the application pools to recycle, then watch for a second #Fields: directive to appear mid-file with 15 columns instead of 14. IIS writes a fresh header line every time logging config changes under a running file, so you literally see the schema change show up partway down the log. That is the verification. Saving the setting proves nothing. The recycle and the new header landing do.

Why this stayed silent

Every failure in this story was silent for the same reason. The wrong IP is structurally identical to the right one. It is four octets. It routes. It geolocates. It sorts. The log parser, the rate-limit sizing, the bot report and the IP-lookup page all accept it without complaint, because there is nothing to complain about. Read the wrong column and you get no exception, just a plausible answer. A plausible answer survives spot checks.

A type error would have caught itself. A null would have caught itself. A valid-but-wrong value is the one class of bug that sails straight through.

The same week, a second surface bit us. Once Cloudflare sits in front of an app that calls itself over HTTPS, internal traffic starts traversing the public edge too. Our iCal calendar generator fetched a schedule endpoint over public HTTPS, which now meant origin to Cloudflare and back to origin. Reading the IIS logs to size a rate limit, we saw the production server’s own IP among the heaviest “clients,” hitting one endpoint over 19,000 times in two hours. Same root cause, different surface.

So check that your analyzer reads CF-Connecting-IP and CF-Pseudo-IPv4 ahead of c-ip. And scan for your own origin’s address showing up as a top “client.” That means the app is calling itself back through the public edge.