The bug was not PayPal, it was us throwing away the buyer's currency
A CAD customer paying a USD invoice exposed the classic accounting bug: the system stored the merchant currency and lost the buyer's actual money story.
The build log
Notes from running agents in production.
Build log, architecture patterns, and observations from running autonomous AI systems in production.
-
-
I overwrote a live production mailer because I assumed an ID was free
My "idempotent" upsert assumed trial ID 14 was unused. It wasn't. It was a live "Staff Reminders" mailer, and at 15:58:09 EDT my update path overwrote its subject, from-address, and body with "Get ...
-
The cockpit outage was just IIS taking a nap
What looked like a mysterious site hang turned out to be a boring Windows hosting default with exactly the wrong behavior for an AI-facing app. The fix took five minutes. Encoding it took another ten.
-
Stop letting the agent guess which clone to touch
A better AI workflow started with a blunt policy change: orient first, review read-only by default, and stop auto-routing work into the wrong checkout. The cockpit prototype and the routing-skill change came from the same root cause.
-
7 games became 14: making a bulk insert idempotent with an origin key
How I tracked down a doubled game table after an app-level site recovery, and the origin-key guard that makes bulk inserts safe to re-run.
-
The release said LIVE. The code was not.
A ticket hit LIVE in JIRA with Fix Version stamped. The merge to master never ran, and the bug it fixed was still reaching users. On workflows that accept a false final state and call it done.
-
How I Run AI Peer-Agents on Each Other
An architecture-review AI claimed I was avoiding real work. I asked it to audit its own claim against my actual logs. It retracted, with data.
-
One Private Dependency, Five Different Failures
Adding a private git dependency broke my blog deploy in five completely unrelated ways across two CI targets. SSH keys, git config overwrite semantics, GitHub's not-found lie, pnpm hoisting divergence, and a silent deploy you only learn about from email.
-
Your IIS Logs Start Lying the Moment Cloudflare Goes Live
Orange-cloud a legacy IIS app and every IP your origin sees becomes a Cloudflare edge IP, with no errors anywhere. How to find and fix the lying logs.
-
The Button Clicked and Nothing Happened: IIS Ate My JSON Error and Handed Back HTML
My backend returned a 400 with a JSON error body. IIS replaced it with HTML, the client swallowed the parse error, and the button did nothing.
Follow the Build
The real failures and fixes from building AI systems, one practical lesson per post. Get the next one in your inbox.