Every DNS lookup we depended on failed at once, halfway into a Cloudflare-for-SaaS migration audit, and they failed instantly. That speed was the whole tell. The fix was a one-line endpoint swap, but the reasoning behind it is worth more than the line of code.

What the audit needed

The SaaS platform runs well over a thousand customer domains. Before we could plan a migration wave we needed to know, per domain, which registrar each one actually points at. The registrar decides whether we can flatten an apex onto Cloudflare or whether we’re stuck doing a relay. The cheap way to answer that for that many domains is a DNS-over-HTTPS lookup of each domain’s NS records, then bucket by nameserver suffix. domaincontrol.com means GoDaddy.

The split that came out of this run was GoDaddy at roughly 88 percent, with the rest small: 10 on Network Solutions and 73 that didn’t resolve an NS at all. That GoDaddy number is load-bearing for the whole migration plan, which is why it was alarming when the lookups all died.

We were hitting Cloudflare’s resolver:

GET https://cloudflare-dns.com/dns-query?name=example.com&type=NS
Accept: application/dns-json

Standard DoH, Cloudflare’s JSON resolver, nothing exotic.

Reading the failure

Every single lookup failed, and none of them waited first.

A real DNS resolution failure has a shape. The name doesn’t resolve, or the upstream is slow, or you hit a timeout, and you wait for it. When every call comes back failed in the time it takes to fire them all, nothing got far enough to resolve anything. That instant, uniform failure is the fingerprint of a connection or TLS error sitting in front of the resolver.

So we stopped treating it as “DNS is broken” and started treating it as “we never reached the resolver.” Different bug, different fix.

The certificate

The TLS certificate for cloudflare-dns.com would not verify on that box. The exact error was Python’s:

SSLCertVerificationError(5, '[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1028)')

The HTTPS connection never completed, so every request bailed out before a single DNS query went over the wire. From the calling code it looked like a total DNS outage. It was a verification failure on this particular machine.

We did not fully root-cause why that box couldn’t verify Cloudflare’s cert that day. “Unable to get local issuer certificate” points at a trust-store gap, a missing intermediate or root the box couldn’t chain to, and chasing it down would have stalled the audit we actually needed to run. We didn’t have to. The failure was reproducible, it was specific to one endpoint, and there was a drop-in alternative one host over: the same probe against dns.google/resolve returned the answer fine.

Swapping to a second provider

Google runs a DoH endpoint with the same JSON response shape:

GET https://dns.google/resolve?name=example.com&type=NS
Accept: application/json

The request goes to dns.google/resolve instead of cloudflare-dns.com/dns-query, and the Accept header is application/json instead of application/dns-json. The answer comes back in the same structure, with the same Answer array and the same record fields. We swapped the endpoint, the lookups went through, and the GoDaddy split came back clean: domaincontrol.com for the GoDaddy bucket, the rest sorted by suffix.

There was no reshaping of the parsing and no second code path for a different response format. The two providers are close enough that you can fall through from one to the other without the consuming code knowing which one answered.

That mattered beyond the registrar split. We leaned on DoH all day for that migration, because it was how we verified cutovers without waiting on the box’s local resolver cache, which held the old record for the full one-hour TTL.

After flipping a www CNAME to the Cloudflare-for-SaaS target hostname, the local resolver kept handing back the old IIS-direct answer, so curl on that box hit the old origin and looked like the cutover had failed. Querying dns.google directly showed the record already pointing at Cloudflare. The local cache can hand you a stale answer for up to an hour, while DoH gets you the current one. So the resolver call was load-bearing for the whole migration, and a single failing cert had taken it down.

Two lessons stacked together

The smaller one is the diagnostic read. When everything dies at once with no wait, the problem is in front of the thing you think is failing: the connection, the TLS handshake, the auth, the cert. Slow is DNS. Instant and total is the layer below DNS. Reading those failures as “DNS is down” would have sent us hunting in the wrong place, and the latency profile pointed straight at TLS.

The bigger one is the design lesson. A batch job that depends on a single external resolver inherits that resolver’s bad day. A cert that won’t verify on one box, a missing intermediate in the local trust store, a rate limit, a regional outage, any of those takes down the entire run. You find out halfway through a thousand-plus lookups you needed for a migration plan, and none of it has anything to do with whether your logic is correct.

The mitigation costs almost nothing. dns.google/resolve and cloudflare-dns.com/dns-query are drop-in compatible. Keep both on hand, and if the primary fails on connection or cert verification, fall through to the other. One provider having a cert problem on one machine should never tank a whole batch job, and it won’t if your code can reach for a second resolver that speaks the same JSON.