https://www.customer.example.com returned 200 through Cloudflare, so I marked the cutover verified and moved on. Then I deleted the interim IIS binding I was sure was redundant, and the same URL dropped to 404. The 200 had been the interim answering the whole time. The path I claimed was live had never served a single request.
The embarrassing part is that nothing was broken. The page loaded, the check was green. The mistake was believing the green meant what I wanted it to mean instead of what it actually proved.
The setup
The production app runs on Cloudflare for SaaS. Customer domains are custom hostnames inside our Cloudflare zone, and each one points at a custom origin: custom_origin_server = cf-origin.<zone>, which resolves to a shared catch-all binding on a dedicated IP at the new shared-cert origin. That shared-cert path was the new scalable target. The job was proving it works end to end, with one cert and zero per-domain bindings.
customer.example.com was the smoke-test domain. The cutover step is simple: set the custom hostname’s custom_origin_server to cf-origin.<zone> via the Cloudflare API, then confirm the page serves. I did that, hit https://www.customer.example.com, got a clean 200, and called it done.
The problem is there were two ways for that URL to return 200, and all I had proven was that a 200 came back. I had not proven which origin served it.
The drop
There was an older interim binding still in play: a per-domain SNI entry for the customer domain, sitting on the main IIS site at the old origin IP. It was left over from an earlier arrangement and I thought it was dead weight now that the cutover was in. So I removed it, expecting nothing to change because the new shared path was supposedly already serving.
The URL went to 404 immediately.
That 404 is the entire lesson in one status code. If removing the interim binding had no effect, the new path was carrying the traffic and the interim was genuinely redundant. The instant it broke, it told me the opposite: the interim had been answering every request I used to declare the cutover verified. The new custom-origin path had served nothing. My “verified 200” was the interim’s 200 wearing the new path’s name, and I had signed off on it.
Diagnosing it without guessing
The honest move here is to stop theorizing and isolate the two things that could be lying to me: Cloudflare’s behavior, and the origin’s behavior. They are different layers and they fail differently, so I probed each one on its own.
To mimic exactly what Cloudflare does when it connects to the origin, I hit the origin directly with the TLS SNI set to cf-origin and the HTTP Host header set to www.customer.example.com. That separates “is Cloudflare routing wrong” from “is the origin routing wrong.” The results split cleanly:
- Through Cloudflare: 404, with a
DYNAMICresponse (Cloudflare’s marker for content proxied from origin, not served from its own cache or error page). - Direct to origin with that SNI/Host pair: 302.
A 404 from the edge and a 302 from the origin for the same request is a real signal. It pointed at Host-routing: maybe the origin was sending different responses depending on which Host header arrived, and Cloudflare was landing on the wrong vhost.
So I tested that hypothesis instead of assuming it. I sent every Host variant I could think of at the shared-cert catch-all. Every single one came back 302. If the origin returned 302 for all Hosts, then Host-routing was not the cause, because there was no Host value that produced the 404 the edge was showing. The hypothesis was dead.
That left the boring answer, which was also the correct one: propagation lag. Cloudflare was still connecting to the old origin IP while the custom_origin_server change worked its way through. The 404 was the transitional state, not a routing bug and not a config error.
Roughly two to three minutes later, Cloudflare picked up the new custom origin, pulled from cf-origin, and served a real 46KB page. I checked it six times. Six for six, stable. That was the cutover actually working, and it was the first time the new path had served anything at all.
Why the first check lied
Walk the timeline and the false 200 is obvious in hindsight:
- The interim binding at the old origin IP was up and answering
www.customer.example.com. - I set
custom_origin_servertocf-origin.<zone>, but that change had not propagated yet, so Cloudflare was still connecting to the old origin IP. - I hit the URL. Cloudflare, still on the old path, got a 200 from the interim and passed it through.
- I read 200 and wrote “verified.”
- I removed the interim binding. The old origin IP still had IIS listening, but with no binding for the domain it answered 404, the new
cf-originpath had not yet taken over, and that 404 is what Cloudflare proxied through.
The verification step passed because of the thing I was about to delete. Every property I was checking for was being supplied by the interim binding, and that binding was on the demolition list. A green check that depends on a soon-to-be-removed component verifies the old state, not the new one. The only honest verification of new infrastructure runs after every prior path is gone.
Related
- Cloudflare Error 1014: The Cert Issued. The Page Didn’t Load.: a live-test-driven rewrite triggered by a similar false assumption about a Cloudflare path
- Don’t Trust the Green Deploy: Grep the Live File for Your Ticket Marker: the same verify-don’t-trust-the-green discipline applied to deploys
- The Migration Flip That Took Down a Live Customer (and the Blast Radius I Could Measure in 12 Seconds): measuring blast radius when a cutover affects real traffic
- The Button Clicked and Nothing Happened: IIS Ate My JSON Error and Handed Back HTML: another case where a 200 masked the real failure
- The Cloudflare-for-SaaS + IIS Gotcha Nobody Documents: SNI vs Host Header: SNI/Host mismatch that a 200 response can also mask during a cutover