The admin console had a clean button. Click it, and each domain got queued for Cloudflare custom-hostname registration. A worker drained the queue and did the real registration over the API, no token sitting in the browser. Nice separation of concerns. I queued ten domains, watched the queue accept all ten, and waited.
After several minutes, zero of the ten were registered. So I went looking, and the worker’s create path turned out to be literally stamped “NOT YET IMPLEMENTED” in the tree. The button had been writing queue files into a void. The 26 domains already live on Cloudflare must have been registered directly against the API instead, because nothing else could have created them, and nobody had ever wired the drain that was supposed to do it automatically.
This is a story about a queue with no proven drain, and the one habit that caught it before I wasted an afternoon.
The setup that looked airtight
I was migrating a batch of the sports SaaS managed domains onto Cloudflare. The www hostname for each one becomes a Cloudflare SaaS custom hostname: CNAME the www record at the Cloudflare edge, Cloudflare HTTP-validates it, issues a DV cert in about five minutes, done. I’d already done 26 of these. The remaining 10 were “not started” in the tracking, so I reached for the console that was built to do exactly this.
The documented design was genuinely good. The console writes one queue file per domain into a queue directory. A separate PowerShell worker drains that queue and calls the Cloudflare API to register each custom hostname. Keeping the API token out of the browser and on the worker box is the right call. The abstraction was clean enough that I trusted it on sight, which is exactly the problem.
Here’s the thing about a clean abstraction: it tells you what is supposed to happen. It does not tell you whether it happened. The console’s job ended when it wrote the queue file. From the browser’s point of view, ten successful writes look identical whether the worker behind them is a finished daemon or a stub that never runs.
Verify the effect, not the trigger
My instinct, after the first couple of minutes, was not to re-click the button or stare at the queue directory wondering if the worker was slow. The queue is the trigger. Whether a file landed in a directory tells me nothing about whether a hostname exists at Cloudflare. That’s the effect, and the effect lives in exactly one authoritative place: the Cloudflare API.
So I polled it directly, read-only, for all ten hostnames. Not the console’s view of the queue. The actual zone, asking Cloudflare: do these custom hostnames exist? Zero. After minutes of “draining,” nothing had been created on the other end.
That gap between “I triggered it” and “it happened” is where most automation rots quietly. A button click, a queued job, an enqueued message, a 202 Accepted: every one of those is a promise, not a receipt. The only honest confirmation is to ask the downstream system whether the thing you wanted now exists. If you can query it, query it. Polling the Cloudflare API for the ten hostnames took less effort than the speculation I’d have burned otherwise, and it turned a vague “is this slow?” into a hard “this never ran.”
The discovery: the worker was never built
Once the API said zero, the next move was to read the worker, not theorize about it. The worker’s create path was stamped “NOT YET IMPLEMENTED.” There was no race, no token problem, no slow drain. The drain didn’t exist. The queue was a TODO with extra steps and a UI on top.
Which raised the obvious question: how were the other 26 domains live? They couldn’t have come through this worker. The only path left is that they were registered directly against the API, and the console’s tidy queue-and-worker story had grown up around a process that was quietly carrying the whole load by hand. The abstraction described an intent. The intent had never shipped.
This is a specific flavor of stub I keep running into: the one that returns a plausible non-result and gets papered over by a UI. It’s a cousin of a billing path I’d hit the same week, where a downgrade-confirmation dialog was just a stub that routed to “email support.” A transition nobody built, stubbed to “email support,” sitting on the happy path looking like working code. Stubs that return something are far more dangerous than stubs that throw, because the something flows downstream and nobody notices the hole until an unexpected input falls through it.
The fix: lift the proven config off a known-good instance
I had two options. Wait for someone to implement the worker’s create path, which is open-ended work on a path I now knew was fiction. Or do what the 26 working domains had clearly done already: register the hostnames directly, using a config I knew worked.
I took the second one, and the key was not to reinvent the registration config from the docs. The docs are what produced the unimplemented worker. Instead I pulled the actual settings off an existing live hostname, one of the 26 that was already serving traffic correctly, and copied them verbatim:
ssl: { method: http, type: dv, settings: { min_tls_version: "1.2" } }custom_origin_server: <your-cf-origin-hostname>That config wasn’t a guess. It was lifted from a domain Cloudflare was already validating and serving. When the abstraction you were handed is vapor, the most reliable spec is a running instance of the thing you’re trying to reproduce. A known-good instance is documentation that can’t drift, because it’s currently in production proving itself.
I registered all ten against the API with that config. Each came back pending, which is the correct state: Cloudflare now knows about the hostname and is waiting for DNS validation. Then I flipped the www CNAMEs so Cloudflare could HTTP-validate and issue the certs, the same roughly five-minutes-each cadence the 26 prior domains had taken. I left every apex untouched on the origin, because the www path and the bare-apex path are entirely different problems and I wasn’t going to conflate them.
End to end, then verify again
Same discipline at the finish as at the start. Within minutes, 8 of the 10 were active. Smoke tests on those returned 200 with server: cloudflare in the headers, which is the real receipt: not “I registered it” but “an independent request to the hostname comes back served by Cloudflare.” The remaining two were mid-validation on the normal timeline, not stuck.
The point of re-checking at the end is the same as checking at the start. The registration call returning success is still a trigger. The hostname actually serving a 200 through Cloudflare is the effect. You confirm both ends against the downstream system, or you don’t actually know.
The takeaway
When automation should have run, verify the effect, not the trigger. Did the thing get created? Ask the system that owns the thing, directly, read-only, and believe its answer over the button you pressed or the queue you filled. “I clicked it” and “it happened” are different claims, and only one of them is checkable in the place that matters.
Two corollaries that fell out of this:
A queue with no proven drain is a TODO with extra steps. A clean producer-side abstraction tells you nothing about whether the consumer exists. Before you trust a worker, confirm it does work, ideally by reading its actual implementation rather than its README.
And when the documented path turns out to be vapor, replicate the config off a known-good instance instead of rebuilding from the docs that produced the vapor. A running production instance is the one spec that can’t lie to you. The 26 domains that worked were a better source of truth than the worker that was supposed to make them.
Related
- The Dry-Run That Lied: When —dry-run and —apply Run Different Code Paths: another case where the “it ran” confirmation didn’t match the actual effect
- Don’t Trust the Green Deploy: Grep the Live File for Your Ticket Marker: the same “verify the effect, not the trigger” discipline applied to deployments
- A var EDIT_DATABASE = false Sat in a Branch for 2.5 Years. Then We Shipped It.: a stub that stayed in production code long past its intended removal
- The Whole Feature Was Built and Verified in Sandbox, Then Prod Said E00044: CIM Not Enabled: a feature that appeared complete until the real environment surfaced the missing piece
- How to Triage an 880-Ticket Backlog Without Deleting Real Work: treating triage as a verification pass, not just a sorting exercise