My whole 1,200-domain provider split died instantly: keep a second DoH provider on hand

I was halfway into a Cloudflare-for-SaaS migration audit when every DNS lookup I depended on failed at once. Not some of them, not the flaky ones, all 1,264 of them, and they failed instantly. That speed was the whole tell. The fix was a one-line endpoint swap, but the lesson behind it is worth more than the line of code.

The setup

The sports SaaS runs about 1,264 customer domains. Before I could plan a migration wave I needed to know, per domain, which registrar each one actually points at, because the registrar decides whether I can flatten an apex onto Cloudflare or whether I’m stuck doing a relay. The cheap way to answer that for 1,264 domains is a DNS-over-HTTPS lookup of each domain’s NS records, then bucket by the nameserver suffix. domaincontrol.com means GoDaddy. The split that came out of this run was GoDaddy 1,120 of 1,264, roughly 88 percent, with the rest small: 10 on Network Solutions and 73 that didn’t resolve an NS at all. That GoDaddy number is load-bearing for the whole migration plan, which is exactly why it was alarming when the lookups all died.

I was hitting Cloudflare’s resolver:

GET https://cloudflare-dns.com/dns-query?name=example.com&type=NS
Accept: application/dns-json

Standard DoH. Cloudflare’s JSON resolver. Nothing exotic.

The symptom that told me it wasn’t DNS

Every single lookup failed, and they failed with no delay. That is the part to pay attention to.

A real DNS resolution failure has a shape. The name doesn’t resolve, or the upstream is slow, or you hit a timeout, and you wait for that. Slow is the signature of a resolution problem. When 1,264 calls all come back failed in the time it takes to fire 1,264 requests, DNS never entered the picture. Nothing got far enough to resolve anything. That instant, uniform failure is the fingerprint of a connection or TLS error sitting in front of the resolver, not a resolver problem.

So I stopped treating it as “DNS is broken” and started treating it as “I never reached the resolver.” Different bug, different fix.

The actual cause

The TLS certificate for cloudflare-dns.com would not verify on that box. The exact error was Python’s:

SSLCertVerificationError(5, '[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1028)')

Cert verification failed, the HTTPS connection never completed, and so every request bailed out before a single DNS query went over the wire. From the calling code’s point of view it looked like a total DNS outage. It was a verification failure on this particular machine, not a DNS problem.

I’m not going to pretend I fully root-caused why that box couldn’t verify Cloudflare’s cert that day. “Unable to get local issuer certificate” points at a trust-store gap, a missing intermediate or root the box couldn’t chain to, and chasing it down would have stalled the migration audit I actually needed to run. The point is I didn’t have to. The failure was reproducible, it was specific to one endpoint, and there was a drop-in alternative one host over: the same probe against dns.google/resolve returned the answer fine.

The fix: a second DoH provider that speaks the same JSON

Google runs a DoH endpoint with the same JSON response shape as Cloudflare’s:

GET https://dns.google/resolve?name=example.com&type=NS
Accept: application/json

The request is dns.google/resolve instead of cloudflare-dns.com/dns-query, the Accept header is application/json instead of application/dns-json, and the answer comes back in the same structure. Same Answer array, same record fields. I swapped the endpoint, the lookups went through, and the GoDaddy split came back clean. domaincontrol.com for the GoDaddy bucket, the rest sorted by suffix, 1,120 of 1,264 on GoDaddy.

It was genuinely a drop-in. No reshaping the parsing, no second code path for a different response format. The two providers are close enough that you can fall through from one to the other without your consuming code knowing which one answered.

That mattered more than just unblocking the registrar split. I leaned on DoH all day for that migration. It was how I verified cutovers without waiting on the box’s local resolver cache, which held the old record for the full one-hour TTL. After flipping a www CNAME to the Cloudflare-for-SaaS target hostname, the local resolver kept handing back the old IIS-direct answer, so curl on that box hit the old origin and looked like the cutover had failed. Querying dns.google directly showed the record already pointing at Cloudflare. The local cache lies to you for up to an hour; DoH gets you the current answer. So the resolver call wasn’t a side errand I could skip for a day. It was load-bearing for the whole migration, and a single failing cert had taken it down.

Why this is the part worth remembering

There are two separate lessons stacked here, and the smaller one is easy to miss.

The smaller one is the diagnostic read. Instant, uniform failure across a whole batch is not a resolution failure. When everything dies at once with no wait, the problem is in front of the thing you think is failing. It’s the connection, the TLS handshake, the auth, the cert. Slow is DNS. Instant-and-total is the layer below DNS. If I’d read those instant failures as “DNS is down” I’d have gone hunting in the wrong place. The latency profile of the failure pointed straight at TLS, and that read saved the whole detour.

The bigger one is the design lesson. A batch job that depends on a single external resolver inherits that resolver’s bad day. A cert that won’t verify on one box, a missing intermediate in the local trust store, a rate limit, a regional outage, any of those takes down the entire run, and you find out at the worst possible moment, halfway through 1,264 lookups you needed for a migration plan. None of that has anything to do with whether your logic is correct.

The mitigation costs almost nothing. dns.google/resolve and cloudflare-dns.com/dns-query are drop-in compatible. Keep both on hand, and if the primary fails on connection or cert verification, fall through to the other. One provider having a cert problem on one machine should never tank a whole batch job, and it won’t if your code can reach for a second resolver that speaks the same JSON.

So: keep a second DoH provider available, read instant batch-wide failures as a connection or TLS problem rather than a DNS one, and don’t let a single resolver’s bad day be a single point of failure for work that depends on it.