x_trans_id Is Not Your Invoice Number: An Authorize.Net Direct-Post Bug
An OR fallback treated x_trans_id and x_invoice_num as the same identifier, so every prod receipt failed while tests passed. The fix was deleting half a line.
The build log
Notes from running agents in production.
Build log, architecture patterns, and observations from running autonomous AI systems in production.
-
-
Stripe Needs a Post-Back, Authorize.Net Doesn't: One Page, Two Flow Shapes
Stripe expects your landing page to capture the charge; Authorize.Net DPM settled before the redirect. Copying one handler for the other fails silently in prod.
-
Shipping a Throwaway Relay Page to Debug a Payment Integration in Prod
When a gateway won't POST to localhost, ship a throwaway relay page to prod and read the real payload. Three bugs surfaced that a faked callback never would.
-
Recovering a 327-Unit Group: Plan-Run Before You Touch Prod
A 327-unit recovery was too risky to validate by instinct. A plan mode, diffable before-and-after artifacts, and a disposable test database made the production run inspectable before it wrote live data.
-
Don't leave the recovery gun loaded: scrubbing prod constants at end of day
After a successful data recovery, the script was still armed: real customer username hardcoded, phase set to run. The most dangerous minute for a destructive tool is the one right after it succeeds.
-
N Drifting Copies Share One Bug: The Legacy Consolidation Thesis
Three recovery scripts shared one un-scoped UPDATE bug that could corrupt other tenants. Consolidating them gave the bug one place to live.
-
A code-first schema audit: keeping a 1,477-line recovery file in sync with a 14,536-column DB
When a hand-maintained recovery script has to stay in sync with a 14,536-column database, discipline alone won't cut it. Here's how a 418-line Python differ and a source-of-truth decision solved the problem.
-
We Wrote a 1,477-Line Site-Recovery Engine: Copy First, Remap IDs Second
When a customer's entire site gets deleted in a shared-database multi-tenant system, RESTORE DATABASE is not an option. Here's the copy-first, remap-IDs-second pattern that makes application-level recovery work without cross-tenant corruption.
-
1900-01-01 Is Not a Date: The Zero-Datetime That Corrupts Every Migration
SQL Server's datetime epoch (1900-01-01) looks like a real date, serializes cleanly, and inserts without error. That's exactly why it corrupts migrations. Here's how to catch it and why you need to omit the column, not blank it.
-
From 3.5 million round-trips to hundreds: batching a remap with temp tables
How we collapsed a 3.5-million-round-trip ID remap into a handful of set-based join-UPDATEs using temp tables, and why the index matters as much as the query.
Follow the Build
The real failures and fixes from building AI systems, one practical lesson per post. Get the next one in your inbox.