https://www.customer.example.com returned 200 through Cloudflare, so we marked the cutover verified and moved on. Then we deleted the interim IIS binding we were sure was redundant, and the same URL dropped to 404. The 200 had been the interim answering the whole time. The new path had never served a single request.

Nothing was broken. The page loaded and the check was green. We believed the green meant what we wanted it to mean, and it proved much less.

The setup

The production app runs on Cloudflare for SaaS. Customer domains are custom hostnames inside our Cloudflare zone, and each one points at a custom origin: custom_origin_server = cf-origin.<zone>. That name resolves to a shared catch-all binding on a dedicated IP at the new shared-cert origin. This shared-cert path is the new scalable target, and the job was proving it works end to end with one cert and zero per-domain bindings.

customer.example.com was the smoke-test domain. The cutover step is short: set the custom hostname’s custom_origin_server to cf-origin.<zone> via the Cloudflare API, then confirm the page serves. We did that, hit https://www.customer.example.com, got a clean 200, and called it done.

There were two ways for that URL to return 200. All we had shown was that a 200 came back. We had not shown which origin served it.

The drop

An older interim binding was still in play: a per-domain SNI entry for the customer domain, on the main IIS site at the old origin IP. It was left over from an earlier arrangement, and we figured it was dead weight now that the cutover was in. So we removed it, expecting nothing to change, since the new shared path was supposedly already serving.

The URL went to 404 immediately.

If removing the interim had no effect, the new path was carrying the traffic. The instant it broke, it told us the opposite. The interim had answered every request we used to declare the cutover verified, and our “verified 200” was the interim’s 200 under the new path’s name.

Diagnosing it

We split the problem into the two layers that could be misleading us, Cloudflare’s behavior and the origin’s behavior, and probed each on its own.

To mimic what Cloudflare does when it connects to the origin, we hit the origin directly with the TLS SNI set to cf-origin and the HTTP Host header set to www.customer.example.com. The results split cleanly:

  • Through Cloudflare: 404, with a DYNAMIC response (Cloudflare’s marker for content proxied from origin, not served from its own cache or error page).
  • Direct to origin with that SNI/Host pair: 302.

A 404 at the edge and a 302 at the origin for the same request looked like Host-routing. Our theory was that the origin answered differently depending on which Host header arrived, and that Cloudflare was landing on the wrong vhost.

We acted on that theory and sent every Host variant we could think of at the shared-cert catch-all. Every single one came back 302. No Host value produced the 404 the edge was showing, so the Host-routing theory was dead, and the time we spent chasing it bought us nothing.

That left the plain answer: propagation lag. Cloudflare was still connecting to the old origin IP while the custom_origin_server change worked its way through, and the 404 was that transitional state.

Roughly two to three minutes later, Cloudflare picked up the new custom origin, pulled from cf-origin, and served a real 46KB page. We checked it six times and got six for six, stable. That was the cutover actually working, and the first time the new path had served anything at all.

Why the first check lied

The timeline makes the false 200 obvious in hindsight:

  1. The interim binding at the old origin IP was up and answering www.customer.example.com.
  2. We set custom_origin_server to cf-origin.<zone>, but the change had not propagated yet, so Cloudflare was still connecting to the old origin IP.
  3. We hit the URL. Cloudflare, still on the old path, got a 200 from the interim and passed it through.
  4. We read 200 and wrote “verified.”
  5. We removed the interim binding. The old origin IP still had IIS listening, but with no binding for the domain it answered 404. The new cf-origin path had not taken over yet, so that 404 is what Cloudflare proxied through.

The verification passed because of the thing we were about to delete. Every property we checked for was being supplied by the interim binding, and that binding was on the demolition list. A green check that depends on a component you are about to remove verifies the old state, not the new one. So the final verification of new infrastructure has to run after every prior path is gone, and it has to tell you which path served the request.