Every Customer Domain Was Down for Eleven Hours and the Monitor Stayed Green

At 9:01pm one night a single web server binding disappeared. Every customer custom domain on our platform returned a bare 404 until 8:11 the next morning. That is about eleven hours.

We had monitoring. It checked customer domains on a five minute loop, and a failure on it was wired to an external uptime service that phones people. It never fired once.

This is not a story about one bad command. The command is the least interesting part. The interesting part is how many separate things all had to be true for a single removal to sit there overnight, and how each one was a layer we had assumed was covered.

What broke

All of our customer custom domains land on one IIS site through a catch-all HTTPS binding. One binding, on one IP and port, with no host name, so that any customer domain the CDN sends us is accepted. That is a sensible way to run a multi-tenant platform. It also means one object carries every customer.

An AI coding agent made the change. We had dispatched it to set up a new internal site on the same server. It took a configuration backup first, which is the only reason the fix later took one command. Then it created the new site. Its first script errored on a binding that already existed. Its second script removed the new site’s HTTPS binding so it could recreate it with a different setting, and the removal named the new site and the protocol, and nothing else. No address, no port, no host.

The server config history shows the catch-all binding gone in that same minute, and the new site’s own binding intact. When the agent re-ran that script, the removal failed with “cannot find binding”, which fits the first run having removed something other than what the agent meant.

We have to be straight about the evidence here. The exact cmdlet is inferred from timing and from the transcript. We deliberately did not reproduce its matching behavior on production to prove it. What we can show is the before and after of the config, not the mechanism.

The agent then reported the new site verified. It had loaded the new site and nothing else.

Here is one customer request, hop by hop, and where the monitor looked versus where it broke.

Layer 1: detection coverage

This is where we were most sure we were covered, so it is where we were most wrong.

The five minute monitor had four test domains, put there on purpose as stand-ins for customers. When we asked afterwards why they did not trigger, the answer was plain. Each probe requested the apex form of the domain, with redirects switched off, and expected a 301 to the www form.

The apex is answered by a separate small relay server. The relay was healthy all night, so the monitor correctly reported what it checks. But customers do not live on the apex. They live on www, which goes through the CDN into the web server that broke. The monitor stopped at the redirect and never made the second hop.

The check was even labelled as the relay check, with the four domains listed under it as sub-results. At a glance it looked like a full domain check. It was not.

Where it goes wrong: a probe proves only the exact request it makes. A redirect proves the redirector answered. A 200 proves something answered, not which something.

A second detection signal we were not using: the support inbox. While the monitor stayed green, our customers noticed. Dozens of emails from concerned users arrived overnight and sat in the inbox. We only saw them after a staff member reached out in the morning. The inbox was an alarm nobody was listening to. Your customers noticed hours before your tools did, and their emails were sitting in an inbox.

A sudden jump in support tickets, emails or chats is a detection signal in its own right. It does not care which host form is probed or what a health page reads, so it is a backstop for the day the synthetic monitor is blind. It should raise an alert, including outside business hours, to someone who will act on it. We have not built this yet.

What goes here: probe every host form a customer actually uses, through the same path a customer takes, and require content that only the right backend can produce. We added a www check for each test domain, with redirects not followed. It requires a 200 from a tiny marker page that returns which IIS site answered and a one-time token we send, plus the CDN’s own request header, so a cached copy, a default page or a wrong site all fail. The next addition is the inbox alert above.

Layer 2: the health signal

We assumed that once the monitor was fixed we were done. Then a code review turned up something worse.

An external uptime service polls one health page every five minutes and phones people when it stops saying “healthy”. That page reported healthy whenever the latest finished monitor run existed. A run that finished with a failure is still a finished run. So the page could not report a failure from any monitor, ever. We checked this against production data and found failed runs from the week before that still read healthy.

So the chain was broken in two places. The monitor never looked at customer www, and even when it did fail, the page the phone service reads would have said everything was fine.

Where it goes wrong: a health page proves only what it reads. Stale results are the quiet version of the same bug: if the monitor itself stops running, the last good result sits there looking healthy.

What goes here: the health page now requires the latest run to have passed, reports an error if the newest finished run is more than twenty minutes old (our real gaps never exceeded eleven), and treats a run that was closed out after timing out as a failure rather than the pass the engine had been storing.

Layer 3: reaching a human overnight

The uptime service stayed green all night, so nothing escalated. Nobody was paged, because the page it reads said healthy. The first person to reach us was a staff member, in the morning. Only then did we see the dozens of emails from concerned users that had come in overnight.

When we fixed the layers above, we wanted proof the last link worked, not an assumption. So we ran a deliberate, announced test: point the new check at a domain that does not exist, let the monitor fail, and watch the whole chain. The check failed, the health page flipped, the uptime service marked us down on its next poll, the emails went out on the second failed check and a phone rang.

It also found a bug no unit test would have. We wanted a WhatsApp message on the second failure, because the service cannot send one. We built a small watcher on one of our own machines for it. Its test outage message arrived without the “TEST RUN” label we had put on it, and claiming the phone tree had been called, which the watcher cannot know. The cause was a model that reviews every outbound alert and had rewritten the text. Health check alerts now go out word for word.

Where it goes wrong: an alert proves nothing until a human is shown to receive it. Software that edits alert text can lose the part that matters. And any step that lives on one machine fails when that machine does. The watcher runs on one machine, so it is a convenience, not a guarantee. The phone calls do not depend on it.

Layer 4: blast radius

One binding, every customer. That is the structural weakness that made this an all-customer outage instead of a one-site problem. We are not going to change that architecture this week, because the shared binding is what lets us run on one certificate and no per-customer config.

What we can do is make the object loud. It is now listed with a do-not-touch mark in a single page that enumerates every site and binding on the box. While reading the server we also found a smaller related follow-up on the same box. Nothing is broken today, and it is queued.

Where it goes wrong: any object that serves more than one tenant is a single point of failure, and you usually find the list by getting burned by the first entry.

Layer 5: preventing the change

No document said never to remove that binding. It was described in several places, but only described, and none of them was the runbook for changing the web server. The agent had read a registry note describing the catch-all binding sharing that IP with another site, and then made a change that removed it.

A written rule that an agent has to read and choose to obey is a weak layer. The sanctioned path now is a wrapper script: it takes a backup, snapshots every site and binding, runs the change, diffs everything against the snapshot, loads the customer domains, and restores automatically if anything outside the target changed or two of four domains fail. A binding can only be removed by giving the full address, port and host, and the shorthand form is refused outright. We ran it with a dry run and its self-tests on the real server before trusting it, and an outside model reviewed it. The review found real bugs, including paths where it could have left a bad change live. Those are fixed.

A guard is now live in our own tooling: a command from our agents that changes the production web server is refused unless it goes through the wrapper, and a command it cannot read is refused too. It covers the routes our own agents use, so treat it as a strong default rather than a promise that no other route exists.

Layer 6 and 7: verifying and rolling back

The agent verified the thing it changed and nothing else. The fix for that is in the wrapper: verify a sample of unrelated customer domains against the before state, every time. The backup the agent took before starting is what made recovery a single step. The binding was restored exactly as it had been, with no restart or reboot, within minutes of the report.

Here are the six stages in the diagram side by side, with what failed in each and what we put in place.

What we would tell you to copy

Do not copy the fix list. Copy the order of questions.

1. Does your monitor request the exact host forms your customers use, apex and www, every tenant pattern, and require something only the right backend returns?
2. Do you watch the rate of inbound support tickets, emails and chats, with an alert on an abnormal spike outside business hours, and who receives it?
3. Can your health page ever say healthy while the last run failed or is stale?
4. Has anyone proved, with a real test failure, that a human is reached at night?
5. What is the one object that serves every tenant, and can an unrelated change reach it?
6. Is there a path that makes the safe change the only change, and something outside the agent that enforces it?
7. After a change, do you check the things you did not touch?
8. Can you restore in one step?

The prompt below runs those questions against your own system, layer by layer, starting with the host forms. It is read only.

Eleven hours, one shared binding, and a monitor that only ever asked the apex relay. Every layer above had a reason to be trusted, and the first alarm that actually rang was a staff member in the morning, after dozens of customer emails had sat unread in the inbox all night.

Related

• The False 200: When Removing a Binding Reveals Your Verification Was Lying: a 200 that was the old path answering, in a cutover on the same kind of setup
• The Page Was Broken for Five Days. Our Monitors Did Not Catch It.: a monitor can only catch the failure shape it was built to recognize
• The plain-words version of this story: the same outage without the server details, for people who run a business on a website they do not build, with six questions to ask whoever runs it

dxdev LESSON · paste into your AI coding agent
LESSON: One Failure Should Have to Get Past Every Layer, So Audit Each Layer
SOURCE: dxdev.com/blog/every-customer-domain-down-eleven-hours-monitor-was-green
SAFETY: This audit is READ ONLY. Do not change any config, DNS record, certificate, binding, monitor or alert rule. Do not run anything that writes to production. If a check needs a change to answer, tell me what you would run and stop.
WHAT HAPPENED: A multi-tenant SaaS serves every customer custom domain through one shared catch-all HTTPS binding on a web server behind a CDN. An agent doing unrelated setup work on the same server ran a removal command that named only the site and the protocol. The shared binding disappeared. Every customer domain returned the web server's bare 404 for about eleven hours overnight. A 5 minute monitor with test customer domains existed and stayed green, because it only requested the apex form of each domain, which is answered by a separate relay that was healthy. The www form, the one customers actually use, was never requested. Behind that, the health page the external uptime service polls reported healthy whenever a monitor run had finished, even when that run had failed, so even a correct check could not have paged anyone. Customers noticed first: dozens of emails from concerned users arrived in the support inbox overnight and were not seen until a staff member reached out in the morning.
THE RULE: One failure has to get past every layer, so audit the layers separately. A monitor proves only the exact request it makes. A health page proves only what it reads. An alert proves nothing until a human is shown to receive it.
HOW TO RUN THIS: Work one layer at a time, in this order. For each check report PASS or FAIL with file:line or config path and the evidence you read. Where you cannot see something from the repo or read-only access, say UNKNOWN and name who has to answer it. Do not guess.
LAYER 1, DETECTION COVERAGE (start here):
  1. List every host form a real customer or visitor types or follows: apex, www, any per-tenant subdomain, any custom domain pattern. Then list which exact host forms each monitor, probe and synthetic check requests. Report every form that is served to customers and checked by nothing.
  2. For each monitored domain, say what the check actually proves. A redirect with redirects disabled proves the redirector answered, not the app. Report any check that stops at a redirect or at a shared relay and never reaches the system that serves the page.
  3. Does at least one probe request a tenant domain end to end, through the CDN, to the origin, and require content that only the right backend produces (an identifying marker, not just a 200)? Report any pass condition that a wrong site, a default page or a cached copy could satisfy.
  4. Do the probes come from somewhere other than the origin host? Report any check that runs only from the machine it is checking, and say what that blind spot is.
  5. Is the rate of inbound support tickets, emails or chats monitored, with an alert on an abnormal spike outside business hours? Who receives that alert, and would they see it overnight? Report FAIL if a sudden jump in customer contact would sit unseen in an inbox until someone opens it. This is the backstop for when the synthetic monitor is blind.
LAYER 2, THE HEALTH SIGNAL:
  6. Find the page or endpoint your external uptime service polls. Show what it actually evaluates. Report FAIL if it can say healthy while the last monitor run failed, timed out, or never finished.
  7. Report FAIL if a stale result reads as healthy, meaning the newest monitor run is old and nothing flips the status to an error.
  8. Report any monitor that records a run that died partway as a pass.
LAYER 3, ESCALATION AND REACHING A HUMAN OVERNIGHT:
  9. Write out the alert chain from first failed check to a person's phone: how many consecutive failures before anything fires, which channels, in what order, and who is on it. Mark every step you could not see.
  10. Is there any step where a person must be awake, at a desk, or on a laptop that is switched on? Report any channel that depends on a single machine being up.
  11. Is there any step where software rewrites, summarises or reviews the alert text before it is sent? Report FAIL if an outage alert can be altered, labelled wrongly or made to claim something the sender cannot know.
  12. When did anyone last prove the whole chain end to end with a deliberate, announced test failure, and confirm a human received it? If never, say so.
LAYER 4, BLAST RADIUS:
  13. List every single object that serves more than one tenant: shared bindings, catch-all virtual hosts, wildcard certificates, one load balancer rule, one DNS zone, one CDN setting. For each, say how many customers fail if it is removed or edited.
  14. For each one, can an unrelated change on the same server or account reach it? Report any case where a command can match it without naming it in full.
  15. Report any shared certificate or binding where selecting a certificate for one site could swap the certificate for every tenant.
LAYER 5, PREVENTING THE CHANGE:
  16. Is there one sanctioned path for changing production web server, DNS, CDN or certificate config? Report FAIL if a person or agent can also copy a script to the box and run it directly.
  17. Does a removal command have to name the full target (address, port and host), or can it match by site name and protocol alone? Report any removal that can match more than one thing.
  18. Does the sanctioned path take a backup, snapshot every site and binding before and after, diff the two, and fail on any difference outside the stated target?
  19. Is the rule that says "never touch this shared object" written where the person making the change will actually read it, or only in a registry or a different doc? Report where it lives.
  20. Can an agent or script that wants to change production be blocked by something outside its own judgment, such as a hook, a permission or a lock, or does it rely on it reading and obeying a rule?
LAYER 6, VERIFYING THE CHANGE AFTERWARD:
  21. After a change, does anything load a sample of unrelated tenant domains and compare against the before state? Report FAIL if verification covers only the thing that was just changed.
  22. Are the verification probes strict enough that a redirect, a default page or a cached response fails them?
LAYER 7, ROLLBACK AND CERT AND DNS EXPIRY:
  23. Is there a tested restore path, and how long does it take? Is a backup taken immediately before every change, and does an automatic rollback fire when verification fails?
  24. List every certificate and DNS record that serves customers, when each expires or renews, which process renews it, and whether that process can touch a shared binding. Report any renewal path that has no monitor and no alert on failure.
THEN PRINT: a table per layer (check, PASS/FAIL/UNKNOWN, evidence, fix), a verdict (applies / partially / OUT_OF_SCOPE / no), the single layer where one more failure would reach customers soonest, and the single most important next action. Rank any gap in Layer 1 or Layer 3 first, because those are the layers that decide whether anyone finds out.