One Change Overnight Locked Every Customer Out
What stays with me is how ordinary the failure was. Dozens of customer emails sat in an inbox overnight while our tools said all was well.
One Change Overnight Locked Every Customer Out. Nobody’s Phone Rang.
Every customer’s own web address went offline for about eleven hours, from roughly 9 pm until 8 the next morning, and nobody’s phone rang. An AI coding agent ran a loose instruction meant to remove one small thing, and it removed the single shared door that all customer addresses come in through. Our monitor only checked the front gate, and the status page said ‘all good’ whenever a check finished, even a failed one. Both are fixed, the door was put back in minutes, and a guard now covers agent changes; an alert for sudden jumps in support emails is not built yet. Ask whoever runs your site if the checks test what customers actually use.
If you try the six questions with your own provider, reply and tell me which answer surprised you most.
Not sure where AI would actually help in your business? Our AI workflow readiness assessment is the place to start.
Curious how it is built behind the scenes? Subscribe to Follow the Build, our technical weekly.
Ben Downey