---
title: "Why Your Verification Tool Is Confidently Wrong"
canonical: https://dxdev.com/blog/2026-06-12_verification-tool-layered-systems/
datePublished: 2026-06-12
---
The alert wasn't an alert. It was a customer telling us their site was down, at 3:57 PM on June 12th, four days after we'd already migrated their domain to Cloudflare and marked it done.

The domain belonged to a small-town youth hockey league. Our migration verify tool had signed off on it. Green check, moved on. But `www` was still pointing at the apex, and the apex bounced through a relay to a dead address. Anyone hitting the bare domain got the site. Anyone hitting `www` got nothing.

## Why the tool missed it

The verify tool's DNS check pinned `www` to the Cloudflare edge before testing it. That's the part that mattered: it wasn't checking what the internet actually resolved for that hostname, it was checking what Cloudflare's edge returned when asked directly, with Cloudflare doing the DNS resolution. If you migrate `www` correctly, those two answers agree and the check is fine. If you don't, the pinned check still hits Cloudflare's edge, Cloudflare still answers for the hostname because it's a proxied custom hostname in the zone, and the tool reports success. The actual public DNS record for `www` was never queried. The tool was testing the edge's opinion of the domain, not the domain.

That's a layering bug, not a logic bug. The check was correct for the question it asked. It just asked the wrong question. "Does Cloudflare correctly serve this hostname" and "does the public internet correctly resolve this hostname to Cloudflare" are different tests, and only the second one is what a customer experiences.

## The fix, and why the obvious fix wasn't enough

The immediate fix: four domains had `www` still aimed at the apex instead of the Cloudflare-facing CNAME target. We repointed all four inside the hour.

The harder question was scope. Fixing four domains doesn't tell you if there's a fifth. So the next step was a full sweep: every one of our 1,297 Cloudflare custom hostnames, checked against live public DNS resolution rather than the edge-pinned path the tool had been using. That sweep ran through a verifier subagent so the check itself could be re-run and cross-confirmed rather than trusted on a single pass. Zero other domains came back broken. Four was the whole blast radius, not the first four found.

I considered just re-running the existing verify tool against all 1,297 and calling it done. We rejected that, because the existing tool was the thing that had produced false confidence in the first place. Running the same test again just re-confirms the same wrong answer faster. The sweep had to use a resolution path the original tool didn't: actual DNS lookups against public resolvers, not Cloudflare's own answer for its own zone.

Two related bugs surfaced in the same window and they compound the same failure mode from opposite directions. The first was the CFStatus badge computing "partial" for domains where `www` sits on Cloudflare but there's no apex custom hostname, even when the apex was correctly steady-state on our relay. That's the tool being wrong in the pessimistic direction, flagging roughly 1,025 correctly migrated domains as "Awaiting DNS" when they weren't. The hockey league's domain was the tool wrong in the optimistic direction, reporting success on something broken. Same root cause: the status logic was inferring migration state from what Cloudflare's config said should be true, not from what DNS resolution actually returned.

## What changed in the tool

One fix patched the verify step to check live `www` and apex DNS resolution directly, using pinned curls against the actual public answer rather than the edge-scoped one. A wrong `www` can't pass verification now, because the check no longer asks Cloudflare to grade its own homework. The other fixed the inverse: when the lifecycle scan proves the apex is provably on the relay (`apex_provenance_ok`), the badge upgrades to "migrated" instead of sitting on a stale "partial" heuristic.

The generalizable failure here: a verification tool that queries the same system it's verifying, through that system's own resolution path, will confirm that system's beliefs about itself. It won't catch the gap between "Cloudflare thinks this hostname is proxied correctly" and "a customer's browser gets an IP address that answers." If your test and your target share an execution path, a failure in that shared path is invisible to both. The fix isn't a smarter check, it's a check that leaves the layer it's validating and asks an independent one.