Why TXT Validation Left 956 Customer Hostnames Waiting on Someone Else's DNS

The emails arrived as “[Action required] Validate to renew SSL certificates”, and they were right that something needed doing. What surprised us was how many hostnames were behind them.

956 of 1,265

Every customer domain we host rides a CDN as a custom hostname with a 90-day certificate. Renewal requires the domain to be validated again, and there are two ways to do it. With the HTTP method, the CDN answers the challenge at its own edge, so nobody has to add anything anywhere. With the TXT method, the customer’s own DNS needs a record at every renewal.

Normal onboarding had always used HTTP. The June bulk migration had registered 956 hostnames on TXT, and nobody had chosen that for them. All 956 certificates came due in September, which is why the notices started in August. Each one needed a record in the customer’s own DNS, and a renewal with that record missing fails silently.

We moved all 956 to HTTP. The estate now reads 1,265 of 1,265 on HTTP, and no customer was contacted. The work log closes that stretch, watchdog included, at 10:49 PM, 4h 30m after it started.

Building the watchdog

Fixing the batch doesn’t stop the next email, so we built a certificate watchdog with three commands:

• check settles one “action required” email. It reads the CDN’s record and the certificate the edge is actually serving, then says whether the email needs anything. Most of those emails are stale by the time they arrive.
• scan flags a certificate inside 14 days of expiry that hasn’t renewed, a hostname drifting back to TXT, a renewal stuck in flight, and an onboarding that never got a certificate. With --notify it DMs us on Discord with an SMS fallback, deduped by a signature so a standing finding doesn’t page every day.
• inventory dumps the whole estate to JSON and CSV.

It runs daily at 07:20 as a scheduled task. It costs nothing in model spend, since it is only the CDN’s API plus the notification path.

One false page, then 55

Two days later it paged us that one hostname was stuck. Its renewal had already completed. check said healthy, and scan had never asked. So scan --notify now re-verifies every finding against a fresh fetch just before paging, and drops anything that has healed.

The bigger miss came four days after launch. One morning scan paged us 55 hostnames as “stuck”. None were stuck. All 55 were domains still waiting to be cut over to the CDN, with DNS still pointing at the old server. Their CDN certificates sat unused and could never finish validating, because HTTP validation needs the traffic to arrive at the edge. That is what pending looks like mid-migration, and nothing a visitor saw was wrong.

Inside that batch was the thing we would actually have wanted a page for. Eleven domains had lost their bindings and certificates on the old server in the migration cleanup, while their DNS still pointed at it. Loading their https address returned a connection reset, while the plain http address still served. Certificate state on the CDN couldn’t tell those eleven from the rest of the batch. Where the hostname resolves could.

So scan and check now resolve the name first and sort each finding:

• off-edge: not on the CDN yet, and HTTPS is fine wherever DNS points. Reported every scan, never paged. The muted count is printed and logged, so a quiet run is never mistaken for a clean one.
• origin-https-down: not on the CDN, and the host can’t complete a TLS handshake. This outranks everything, because https is dead for that customer right now.

The eleven had their bindings and certificates restored. The recheck before paging runs the same classification. Without that, a reclassified finding could never match its earlier self, and every one would look healed.

We began the day thinking the risk was 956 certificates. The bigger risk turned out to be that our alarm couldn’t tell an unused certificate from a dead site.

What to check in your own setup

Find out which renewal method your certificates use. If any of them depend on a record in a customer’s DNS, a missing record fails silently at renewal. Then check that your alerting separates a certificate nobody is using yet from a site that is actually down, and that it reports how many alerts it chose to mute. Otherwise an outage gets buried among false pages, which is what happened to our eleven.

dxdev LESSON · paste into your AI coding agent
LESSON: A Watchdog That Judges by One Signal Pages on Noise and Hides the Real Failure Inside It
SOURCE: dxdev.com/blog/956-certificates-needed-a-customer-to-renew
WHAT HAPPENED: A bulk migration in June registered 956 customer hostnames on the validation method that needs a record in the customer's own DNS at every renewal, while every hostname added through normal onboarding used the method the CDN answers itself. The 90-day certificates all came due in September and the CDN began emailing "action required" notices. I moved all 956 to the self-answered method (the estate went to 1,265 of 1,265 on it, no customer contacted) and built a daily watchdog with three commands: check, scan and inventory. Four days later its first real morning produced 55 "stuck" pages. All 55 were hostnames not yet cut over to the CDN, whose certificates were unused, and underneath them were eleven domains whose old server had lost its bindings and certificates in a cleanup while DNS still pointed there, so their https address returned a connection reset. Certificate state alone could not tell those apart. Where the hostname resolves could.
THE RULE: A check that judges by one signal will page on things that are fine and hide things that are broken inside the same list. Add the second signal that separates them, mute the expected state but print the muted count, and recheck against fresh data right before you page anyone.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any alert that classifies health from a single field (status, expiry, exit code) without checking where or how the thing is actually reached.
  2. Any deliberately muted alert class whose muted count is not printed or logged, so a quiet run cannot be told from a clean one.
  3. Any page sent from a cached scan without re-verifying the finding against fresh data first.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.