---
title: "Why 1,282 Custom Hostnames Did Not All Get Retried"
canonical: https://dxdev.com/blog/infrastructure-self-healing-no-ops-team/
datePublished: 2026-06-26
---
At 4:10 PM, a migrated site had the right DNS records and no HTTPS.

The customer had completed the [Cloudflare domain migration](/blog/thousands-of-domains-to-a-layer-above/), but the custom-hostname certificate had stopped moving. From the outside it looked like ordinary DNS propagation. The domain pointed where it should. The staff interface did not surface the actual certificate condition. The site just could not serve HTTPS.

That distinction matters. "Wait a little longer" is fine advice when DNS is still converging. It is bad advice when the certificate workflow is wedged. The customer sees the same broken site either way, but the repair path is completely different.

We fixed the affected certificate first. Then we turned the incident into a system that can spot the same state, repair it, and show staff what is happening.

## DNS and certificate were one vague state

Our first input was blunt: a customer had done a Cloudflare migration and it had not gone through yet. A migration has several moving parts, so the obvious first suspect was a bad or incomplete cutover.

The audit changed that. We listed all **1,282** custom hostnames and classified the failing domain instead of treating every moved hostname as a retry candidate. The domain was pointed correctly. Its certificate was the part that was stuck.

A pointed domain can still be waiting on certificate issuance, and the absence of a surfaced certificate condition does not mean there is nothing to do. Staff needed certificate state, and what they had was a generic migration state. DNS and certificate had been collapsed into one idea of a migration being "in progress." Once we could see the certificate state directly, the repair was straightforward.

## The bulk retry we got as far as considering

The tempting fix was to re-trigger every migrated hostname. We got as far as treating that as the fast path during the audit. It would have cleared the problem from the queue. It would also have been reckless: reissuing across 1,282 custom hostnames would have caught domains in the middle of a legitimate cutover and forced them into a fresh issuance cycle they did not need, on the strength of one customer's report. The urge to clear the whole list came from wanting the incident closed. Nothing showed the whole list had the same problem. The classification step talked us out of it before we ran anything.

The useful condition was never "hostname has moved." It was **stuck but pointed**. That DNS gate separates domains still finishing a cutover from the narrow class with a certificate failure. So the worker lists custom hostnames, identifies the stuck-but-pointed records, and re-triggers only those. That is the difference between an automated repair and an automated source of noise.

A customer would still wait for something a scheduled worker can recognize in minutes, and the staff interface would keep hiding the state by omission. So we shipped both paths.

## The recovery loop runs every three minutes

The production certificate worker now runs every **3 minutes**. Its recovery step, `Invoke-HealStuckCerts`, does three things in sequence:

1. Lists the custom hostnames.
2. Filters for the stuck-but-pointed certificate condition.
3. Re-triggers certificate issuance for that narrow set.

The loop has a cooldown ledger. Without it, a certificate that stays stuck across several scans would be re-triggered every three minutes, which is an accidental retry storm with a nicer name.

The ledger is where we went wrong right after deployment. It was named `cf-heal-state.json` and written into the worker's queue directory. The [queue scanner globbed](/blog/automation-that-ate-its-own-state-file/) `*.json`, so it picked up its own ledger and processed it as a bogus command. The file the worker wrote to remember not to retry too soon became work for the worker.

We caught it in the live production log. The fix was small: cooldown ledgers use `.txt` in that directory. A queue directory has a schema even when the filesystem is the schema. If the worker defines work as "every JSON file here," any JSON file placed there becomes work, so state files need a suffix the consumer will never mistake for input. After the rename we verified the recovery loop end to end.

## What staff see now

Automation does not replace observability. A worker can fix most cases in three minutes and still leave staff blind during the exception it cannot classify.

One release changed the domain dialog in two ways. It now shows the true SSL state, including a stuck certificate, and it gives staff a **Re-trigger** action. The scheduled loop is the primary mechanism. The button is the manual override for when staff are already looking at the domain and need to act now.

The two belong together. A raw provider status helps only if the reader knows what to do next, and a button without the state behind it is a ritual. So the diagnosis and the recovery action sit side by side.

The whole flow is now short enough to reason about:

```text
migration completes
    -> custom hostname is pointed
    -> certificate state is checked
    -> stuck-but-pointed records enter the recovery path
    -> worker re-triggers issuance every 3 minutes, with cooldown
    -> staff can see the same state and re-trigger once when needed
```

We also documented the certificate flow and the surrounding infrastructure, so it does not live only as a memory of one customer incident.

The original site was fixed in the same pass that found it. When a customer reports correct DNS and no HTTPS, check whether your staff can see certificate state separately from DNS state. Let any automated repair act only on records matching the specific stuck condition, and give it a cooldown. Then check that its own state files cannot be mistaken for work by the process that consumes the queue.