--- title: "One Change Overnight Locked Every Customer Out. Nobody's Phone Rang." canonical: https://dxdev.com/everyday-ai/one-night-change-locked-every-customer-out/ datePublished: 2026-10-05 --- One night in October, one change took every one of our customers' own web addresses offline. It started around 9pm and ran until about 8 the next morning. That is eleven hours. We had a monitor. It checked on a five minute loop. It never went off. ## What happened Our customers each have their own web address, like www.yourbusiness.example. Behind the scenes, every one of those addresses comes in through one single door. One door, every customer. An AI coding agent made the change. We had set it to add a new internal tool on the same server. While redoing a small setting on its own new tool, it ran an instruction that was meant to remove one small thing. It was too loose, and it removed that one shared door instead. We should be straight about the evidence. We know the door disappeared in that same minute, because the server keeps a history. We do not know the exact instruction for certain. We chose not to recreate it on the live system to find out. Customers who typed their address got a blank error page. Their sites still existed. Nobody could get in. ## The monitor checked the front gate Here is the part that stings. We had set up four test addresses on purpose, as stand-ins for customers. We had a check on them. But each address has two steps. The first is a small front gate that just says "go to the real site." The second is the door where the real site lives. Our check only knocked on the front gate. The front gate was fine all night, so the check reported green all night. The door customers actually walk through was the one that was locked, and we never tried it. The check was even labelled in a way that made it look complete. That is easy to do and hard to spot. ## Green meant "finished", not "passed" Then it got worse. When we looked deeper, we found the page our outside alert service watches had a second flaw. It said "all good" whenever the last check had finished. Not when it had passed. A check that finished with a failure still counted. So even a perfect check would have said something was wrong, and the page would have told the alert service everything was fine. Two things had to break for nobody to be woken up, and both had. ## Nobody's phone rang Our outside alert service did not call anyone, because it was told everything was fine. Nobody was woken up. ## The inbox was an alarm nobody was listening to Here is the part we think about most. While our monitor said everything was fine, our customers knew it was not. Dozens of emails from concerned users came in overnight and sat in the inbox. The first person to reach us was a staff member, in the morning. Only then did we see them. Your customers noticed hours before your tools did, and their emails were sitting in an inbox. A sudden jump in support emails, tickets or chats is a warning in its own right. It should raise an alert, even outside business hours, to someone who will act on it. It is the backstop for the day your monitor is looking at the wrong thing. We have not built this yet. ## One change, every customer The shape of the problem is plain. One door served everyone, so one slip took out everyone. We are not changing that design this week, because it is also what keeps things simple. What we did was make that door loud. It is now marked "do not touch" on the one page that lists everything on the server, and any change to the server now goes through a checking routine. That routine saves a copy first, takes a list of everything before and after, tries a sample of customer addresses, and puts things back by itself if anything outside the target moved. That saved copy is also why the fix took minutes. The door was put back exactly as it had been, within minutes of the report, with no restart. ## What we fixed, and what is not finished We now check the door customers walk through, not only the front gate. The status page now says "all good" only when the last check passed, and it complains if the checks stop running at all. Then we tested it for real. We warned our staff, made a check fail on purpose, and watched the whole chain. The alert service marked us down, the emails went out and a phone rang. Now live: a guard in our own tools that refuses that kind of change from our agents unless it goes through the checking routine. Not built yet: the alert on a sudden jump in support emails. Here are the six safety nets side by side, with what went wrong in each and what we changed. Table of six safety nets: stopping the mistake, noticing, the all good signal, waking someone, responding and undoing. For each, what went wrong and what we changed. A loose instruction from an AI agent removed the one shared door, and changes now go through a checking routine. The check knocked on the front gate and dozens of customer emails sat unread overnight, and the check now tries the door customers use, while an alert on a jump in support emails is not built yet. All good meant the check had finished, and now means it passed and is recent. Nobody's phone rang, and a deliberate test now proved a phone rings. A staff member reached out in the morning, and one page now lists everything. A saved copy made the fix quick, and the routine now puts things back automatically. ## Six questions to ask whoever runs your website You do not have to know how any of this works. You can ask. 1. **Do you check the address customers actually type?** Not only your main site. Every way a customer might reach you. 2. **Does anyone watch for a sudden jump in customer emails or tickets, and who gets alerted?** Ask whether it would reach a person at 2am, because customers often notice before the tools do. 3. **Does "green" mean it passed, or only that it ran?** Ask what would turn it red. 4. **Who gets woken up, and have you tested it?** Ask when someone last made it fail on purpose and a real person's phone rang. 5. **Is there one thing that, if changed, breaks every customer at once?** Ask what it is and who is allowed to touch it. 6. **How fast can you undo a change, and is a copy saved first?** Ask for the time in minutes. If the answer to any of them is "not sure", you have found the thing to fix first. ## If you use an AI assistant There is a short version of these questions you can paste into the AI assistant you already use, below. It only reviews how you are protected. It does not test anything. One shared door, one check that only ever knocked on the front gate, and dozens of customer emails that nobody was watching. Ask your provider the six questions above before a quiet night asks them for you. ## Related - [The technical version of this story](/blog/every-customer-domain-down-eleven-hours-monitor-was-green/): the same outage with the server details, layer by layer, and a longer audit prompt for developers