The Day I Taught Failures to Speak, Everything Started Shouting
A customer reported a dead Save button on Locations and Fields about two hours after we shipped a fix for a dead Save button on bracket teams. So I checked whether our fix was involved.
The build log
Notes from running agents in production.
Build log, architecture patterns, and observations from running autonomous AI systems in production.
New here? Start with My backups had been dead for three weeks. I found out by accident.The Zombie That Blocked EverythingTeaching the AI to Find the Real Source File: .cursor/rules as a Compiled-Asset MapN Drifting Copies Share One Bug: The Legacy Consolidation ThesisThe attacker was our own redirect logicRoute a Change by Its Blast Radius, Not Its Size or Your Schedule1900-01-01 Is Not a Date: The Zero-Datetime That Corrupts Every MigrationAI Mutations Aren't Optimistic-UI Candidates: Make the 'Analyzing' State Honest
Filter by tag
-
-
The Cleanup Tool That Had to Check Before Deleting
The cleanup run that retired 29 dead domains took three minutes. What made it safe to run was a tool that refuses to trust our own inventory.
-
The Agent Process That Didn't Know Its Session Died
The browser had been dead for 39 hours, and it was still asking our staff toolbar who it was, over and over, with a login that had expired long before.
-
The Firewall Rule That Paid for Itself
Our main site averaged 37.7 error-minutes a day before the rate-limit and bot-blocking rules went in at our CDN edge. Over the last five weeks it averaged 3.3, and yesterday it logged zero.
-
A Read-Only Search Claimed Ownership of 190 Files It Never Touched
A nightly sweep held an entire 190-file repository over a change one session never made. The tool had mistaken a read-only search for a claim of ownership.
-
Published 177 Posts, Manual Checks Found 2 System Bugs
The publish log for a 23 hour 53 minute run showed zero errors, and the live post count went from 554 to 731.
-
The Page Was Broken for Five Days. Our Monitors Did Not Catch It.
An admin page rendered a fatal error inside a normal-looking response for five days. The uptime watcher and the weekly crawl missed it, and the error log had no row for it.
-
Five Days From Cutoff, Inventory Was Wrong
A customer asked who had set up their domain, and by the end of that night I knew our migration tracker was wrong about 85 sites, five days before we were due to switch off the old server address.
-
229 Prompts Before I Built the Session Host
I spent 229 prompts asking an editor extension to manage windows, then built a session host instead.
-
Two Posts, Same Migration, Same Story
One post about a 956-certificate migration was already live. A second post about the same migration, under a different title, was sitting in review behind it.
Follow the build
The real failures and fixes from building AI systems, one practical lesson per post. Get the next one in your inbox.