The Outage That Revealed a Two-Month Silent Failure
The status alert said the relay had been down for 22 minutes, and the relay was a box I thought I understood.
The build log
Notes from running agents in production.
Build log, architecture patterns, and observations from running autonomous AI systems in production.
-
-
A Saved Reply Is Not a Stored Record
A review or handoff is not complete when an agent says it saved the work. It is complete only when the receiving system can read the exact record that the next decision depends on.
-
Clear State Beats Optimistic Promises in Domain Migrations
We told a client their site would be live on the new relay by Friday. Friday came and went.
-
The Scanner Was Gone and the Bill Wasn't
The invoice line item for GitHub Advanced Security did not match what the tool had done. The scan history for that repo said the last run was May 22.
-
An Outage Alert That Kept Blaming Us for Someone Else's Server
A monitor built to catch our own outages paged a customer's dead third-party server as ours, six times in nine days, and I defended it before I was shown wrong.
-
Converting the first real account found a gate that never existed
Moving one league onto its sport's real code path surfaced a years-old tier gate that had simply never been built.
-
Your First Live Migration Is Your Best Stress Test
The first reply to a 375-person mailer arrived about three minutes after the send.
-
Four Identical Cards Were Not a Duplication Bug
My live screen showed the same Discord conversation over and over. I went hunting for a bug that was writing duplicates. Nothing was: each card was a separate session, spawned five minutes apart.
-
The Pre-Release Scan Was Checking Only One Direction
A report built to catch the board and the code disagreeing before a release meeting had a blind spot exactly where seven real disagreements were hiding.
-
The automated repair command was one step from corrupting ten old tickets
A daily alert flagged 17 tickets as inconsistent. Sixteen were false alarms, and the fix for them was about to run.
Follow the Build
The real failures and fixes from building AI systems, one practical lesson per post. Get the next one in your inbox.