A customer emailed in saying his site was dead on his phone. Mobile Safari, “DNS Resolution Error,” nothing loading. The day before, we’d flipped a batch of domains onto the new Cloudflare edge as part of a migration we’d been grinding on for weeks. The timing told us what we’d broken. It did not tell us how many other customers were sitting in the same state without emailing us.
That second question is the whole post. The outage was real and it was our fault. A scan we’d built during the migration, before we needed it, turned a multi-day support slog into a 30-minute fix. It sized the damage in about 12 seconds: 8 domains.
The error that lies about itself
The symptom said DNS, so the instinct was to go check the DNS records.
The DNS was fine. The domain’s CNAME pointed at the Cloudflare edge exactly as it should. What happened was this: Cloudflare received the connection, went to route it through SSL for SaaS, looked for a custom hostname object matching that domain, and found none. That is Cloudflare error 1001, custom hostname not found. Mobile Safari dressed it up as a DNS failure, but DNS had done its job. The traffic arrived at Cloudflare, and Cloudflare had no record telling it where that hostname should go.
If you run Cloudflare SSL for SaaS for multi-tenant custom domains, error 1001 is the one you will misdiagnose first, because the browser blames DNS and the DNS is correct. The fix is never a DNS change. The fix is to register the custom hostname in your Cloudflare zone. The CNAME pointing at your zone is necessary but not sufficient. You also have to create the custom hostname object so the edge knows how to terminate and route the request.
His domain was one of a set of legacy www redirect domains pointed at the production app. These are old customer domains that bounce www to the main app. When we flipped the edge, the www CNAMEs for these redirect domains had not been registered as Cloudflare custom hostnames. DNS now pointed at Cloudflare, Cloudflare had nothing to match, and the result was 1001.
So we knew the failure class: one customer, one domain, one root cause. The dangerous question was the next one.
How many more are sitting like this
You can handle this moment two ways.
The first is to fix the customer who emailed, close the ticket, and wait. Every other domain in the same state is a support ticket you haven’t received yet. Each arrives on its own schedule and gets investigated from scratch, because you’ve already filed the first one as handled. You respond to the same incident ten times.
The second needs a tool that already exists when the first ticket lands. Ours did, barely, because we’d built a lifecycle scan as part of the migration project itself. The scan walks every domain in the migration and classifies its state. We pointed it at the exact failure the customer hit: domains whose www CNAME resolves into the Cloudflare zone but has no matching custom hostname registered.
It came back with 8.
Eight domains of the same class, all legacy redirect domains never registered as custom hostnames. All fixable the same way, by registering the missing hostname in the Cloudflare zone. One class, one remediation, eight applications, restored as a batch with no per-domain re-investigation.
That was smaller than any number we’d have guessed before the migration. Asked cold how bad it would be if the edge flip went wrong, our answer had been “somewhere between one domain and all of them,” which is the answer that makes you afraid to flip at all. The real answer was 8, bounded and known, because the scan could enumerate the class instead of us imagining it.
The ninth hit that wasn’t
The scan flagged a ninth domain. On first read it looked like the same failure. It wasn’t. That domain sat on its own separate Cloudflare zone, outside the SSL for SaaS multi-tenant zone, so the custom-hostname logic didn’t apply to it at all. A false positive.
We’re pointing this out on purpose. A blast-radius scan with zero false positives is usually also missing real cases. The useful version runs slightly over-inclusive and makes you eyeball each hit, rather than being tuned so tight that it drops the edge cases you most need to see. One false positive in nine is a scan we trust. Nine clean hits with no noise would have made us wonder what it wasn’t showing us.
The first version of the scan was too slow to use
Being fast is the reason the scan was usable mid-incident, and the first version was not fast. It timed out.
That version ran a Let’s Encrypt certificate issuance probe against every domain to check TLS health, including the 37 domains whose apex A record hadn’t been flipped to the new relay yet. An LE probe against a domain that isn’t pointing at the relay always fails, and it fails slowly. Multiply a slow failure across a few dozen domains and you get a scan that takes long enough that nobody reaches for it during a live outage.
The fix was a precondition guard: only attempt the relay TLS probe if the apex A record already resolves to the relay IP. With the guard in, the scan covered 39 migrated domains in about 12 seconds and skipped the 37 that weren’t ready. The guard is three lines: resolve the apex, bail if it isn’t the relay IP, only then probe.
Probing external systems feels free until you do it at scale against things that aren’t ready to answer. The happy-path unit test never catches that, because in the test the precondition is always met.
The scan also had to run off-box. Run from the Windows application server, the relay TLS probe fails with an SSPI error. Windows integrated auth tries to negotiate on the outbound connection to a Caddy relay running on Linux, and the relay wants none of it. We run it from an off-box vantage every time.
What to check on your own setup
If you run Cloudflare SSL for SaaS, check whether every custom domain is registered as a custom hostname, because a “DNS error” in the browser can be error 1001 with correct DNS. And make sure your scan skips targets that aren’t ready. Failing probes against 37 domains made our earlier scan too slow to use while a customer waited.
Related
- The False 200: When Removing a Binding Reveals Your Verification Was Lying: plausible-looking success states that hide a real failure
- Cloudflare Error 1014: The Cert Issued. The Page Didn’t Load.: the CF error class that rewrites architecture assumptions mid-migration
- An Off-Origin Caddy Relay for Bare-Domain SSL (Because Cloudflare Pro Won’t Proxy Your Apex): the relay architecture this migration was building toward
- Route a Change by Its Blast Radius, Not Its Size or Your Schedule: the general principle behind per-domain rollout as a safety mechanism
- The Cloudflare-for-SaaS + IIS Gotcha Nobody Documents: SNI vs Host Header: another CF SaaS failure class encountered during the same migration