Theme
Data correctness
Databases that quietly hold wrong data or run slow, and getting back what was lost.
45 posts by DX, in two lanes: plain-language stories and the technical build log.
Everyday lane
Plain-language stories. No jargon. 11 posts.
Start here
A 418-line tool now reads our recovery file as plain text
A recovery plan was only as good as its handwritten inventory, so we built a check that showed what the list missed.
Recovering and repairing bad production data
-
A 418-line tool now reads our recovery file as plain text
A recovery plan was only as good as its handwritten inventory, so we built a check that showed what the list missed.
-
3.5 Million Database Messages Against a Thirty Minute Recovery Window
A recovery job only became practical when we stopped sending one tiny instruction at a time.
-
Four Seconds of Silence After a Routine Update at 3:58
A shared record got addressed by a slot number instead of a name, and a real notice got overwritten before anyone read it. The evidence we checked in the next six minutes is what turned a scary-looking mistake into a non-event.
Keeping data honest, reversible and backed up
-
When the First Check Flagged Records That Were Never Broken
A process can look broken when the problem is a bad assumption about the information underneath it. A small diagnostic can prevent an automated fix from making the situation worse.
-
Why the Undo Restored the Old Value Instead of a Fresh One
The routine meant to catch broken accounts only ever looked at the ones already flagged as broken. One bug un-flagged them first.
-
Why pasting display text into a database command failed
A routine data-maintenance task failed because a date was turned into display text and then treated as dependable data again.
Slow queries and numbers you can trust
-
Why 3,853 Was Never the Number for Active Paid Accounts
Five new reports only became useful after their definitions, bad data, and proof were pinned down.
-
The Trial Site That Would Not Delete and the Cutoff of 100
A trial account could not delete an empty site because a simple rule was using the wrong number.
-
The Schedule Bug That Pointed Back to March 4, 1973
A schedule went out of order because a date number became text, and text follows a different kind of order.
-
Why `"` Appeared After Saving a Label Twice
A small edit-box mistake turned normal quotation marks into text that grew more broken with every save.
-
Two Texts to One Person After a Crash at the Worst Moment
An automated system's text-message callback had a duplicate-send risk hiding in the gap between the message provider accepting a send and our own system recording it as complete.
Technical lane
The build log: the logs, the code, the fix. 34 posts.
Start here
The Console UPDATE Is the Most Dangerous Line in Production
The console UPDATE is the most dangerous tool you can point at production. A four-part repair-script discipline: dry-run, assert, idempotent, JSON record.
Recovering and repairing bad production data
-
The Console UPDATE Is the Most Dangerous Line in Production
The console UPDATE is the most dangerous tool you can point at production. A four-part repair-script discipline: dry-run, assert, idempotent, JSON record.
-
1900-01-01 Is Not a Date: The Zero-Datetime That Corrupts Every Migration
SQL Server's datetime epoch (1900-01-01) looks like a real date, serializes cleanly, and inserts without error. That's exactly why it corrupts migrations. Here's how to catch it and why you need to omit the column, not blank it.
-
Restoring 327 Units From One Backup Without Writing a Row
A 327-unit recovery was too risky to validate by instinct. A plan mode, diffable before-and-after artifacts, and a disposable test database made the production run inspectable before it wrote live data.
-
We Wrote a 1,477-Line Site-Recovery Engine: Copy First, Remap IDs Second
When a customer's entire site gets deleted in a shared-database multi-tenant system, RESTORE DATABASE is not an option. Here's the copy-first, remap-IDs-second pattern that makes application-level recovery work without cross-tenant corruption.
-
A code-first schema audit: keeping a 1,477-line recovery file in sync with a 14,536-column DB
When a hand-maintained recovery script has to stay in sync with a 14,536-column database, discipline alone won't cut it. Here's how a 418-line Python differ and a source-of-truth decision solved the problem.
-
From 3.5 million round-trips to hundreds: batching a remap with temp tables
How we collapsed a 3.5-million-round-trip ID remap into a handful of set-based join-UPDATEs using temp tables, and why the index matters as much as the query.
-
Null oldGameID Columns Pointed Straight at the Duplicate Game Inserts
How we tracked down a doubled game table after an app-level site recovery, and the origin-key guard that makes bulk inserts safe to re-run.
-
Wrong Group on Login Twice in One Week and the Repairs Pointed Opposite Ways
Two accounts, same symptom, same root-cause label, opposite fixes: one row re-pointed, one row deleted. Query the actual data shape before reusing a repair script.
-
We overwrote a live production mailer because we assumed an ID was free
Our "idempotent" upsert assumed trial ID 14 was unused, but it was a live "Staff Reminders" mailer. At 15:58:09 EDT our update path overwrote its subject, from-address, and body with "Get back in the game..." content, in production.
-
SET IDENTITY_INSERT pinned a mailer row to ID 14 on prod
A new exit-feedback page shipped cleanly. The provisioning script had already overwritten a scheduled digest mailer row that had been live for months, with no error.
-
A Titleless Event Entry Vanished From Production 71 Minutes After Deploy
A titleless entry showed up in an event listing, and 71 minutes later it was gone from production. The read-side guard did that, not the delete fix.
-
114 Domains Marked for Shutdown and the Number Nobody Expected
114 domains were on the board as about to go dark. The real number was 59.
-
The Error Log That Lied: When Recent and Lifetime Counts Blur Together
The Error Log table had 492 rows tagged Script timed out over the trailing 120 days, and the ticket that came in described it like a site-wide admin outage.
-
Three Flags and 1,024 Sandbox Rows to Delete Before Going Live
984 of the 1,024 pages we imported from the old staff wiki had not been edited in five years, and the oldest was last touched in 2009. That number shaped the whole feature.
Keeping data honest, reversible and backed up
-
Parked Content Looked Like Corruption to the First Diagnostic
Many apparent software failures are mismatches between the story a user reports, the state a system stores, and the assumptions a repair would make. A focused diagnostic can separate genuine drift from valid but unfamiliar data before anything is changed.
-
Restoring From R2 Required R2 Credentials Stored Inside R2
At 2:53 PM on June 24, we had a freshly verified recovery bundle and the bad kind of confidence. It could restore the important parts of our machine.
-
Expired Domains Vanished From Every Account Page Until June 23
At 6:44 PM on June 23, we stopped treating a removed domain as if it had never existed.
-
domainID 1368 Was Deleted by a Cleanup We Trusted
At 9:59 AM, we deleted a ticket built around 13 supposed production duplicates and restored a test-site domain row, domainID 1368, that the cleanup had removed.
-
A +98/-16 Hotfix Restored removedOn Instead of Wrapping a Transaction
The reconcile only ever retries removed rows. That's the whole bug in one sentence, and it took a half-written domain to find it.
-
The optional Sprint field on one JIRA transition behind 21 lost values
Twenty-two customer tickets closed without a sprint, and only one of them could still be repaired. The other 21 lost that field for good.
-
Five Controls Around One Exceptional Database Change
When a normal workflow cannot make a needed, authorized change, resist the one-off production update. A narrow operator tool should make intent, authorization, validation, review, execution, and recovery explicit.
-
When a Payment Confirmation Event Overwrote Decisions Made by Hand
A payment-confirmation automation overwrote a status that some administrators managed by hand. The remediation separated payment evidence from approval, added explicit tenant controls, and used a narrowly scoped, audited recovery process.
-
Why a Non-Empty Value Crashed a Numeric Conversion
A value can be present, non-empty, and still be the wrong kind of value for the operation that follows. The durable fix is to validate the real contract, separate identifiers from their representations, and fail safely at the boundary.
-
Why Our Data Repair Died on a Date That Looked Obvious
A date value was formatted for display and then handed back to a database parser. The durable lesson is to preserve typed values, use parameters, and perform only authorized, reviewed, validated, and reversible data changes.
-
Two Live GA4 Properties Reporting on the Same Signup Funnel
At 3:21 PM, we found three analytics properties collecting data for the same site: two live GA4 properties and one dead Universal Analytics property.
-
Why Editing an Existing Custom Page Dropped the Reorder Option
A display-order field that appeared to save and then silently dropped its value. The investigation found that ADD and UPDATE had been writing different data shapes for years inside the same module save path.
-
We Staged a dispatch_hours Migration and Never Ran It
At 8:04 AM the brief fired into Discord five times in under two minutes, all five for the same ticket. That was the flap we sat down to fix.
Slow queries and numbers you can trust
-
One Composite Index on org_id and season_id Changed the Page
The page editor was stalling, well past slow: you'd open the leaderboard for a customer's page and wait past the point where you'd normally assume something had crashed.
-
8,000 Logical Reads Behind a Green Index Seek Icon
A SQL Server execution plan showed Index Seek. The same query was doing 8,000 logical reads. Here is what the operator name hides, and how a second covering index collapsed the worst case from 14,718 reads to 3.
-
Why Staff Stopped Opening the Marketing Admin Page After 5 Seconds
The Marketing admin page took about 5 seconds to load, and the fix took 1h25m to trace and ship. Nearly all of that time went into the query, not the page.
-
A New Feature Is the Best Fuzz-Tester for Your Existing Data Model
A new guard exposed a missing lifecycle flag. When a feature needs a real domain fact, adding it to the shared status model can be safer than re-deriving it in one consumer.
-
Reading Our Own Diff Five Months After It Shipped
A new folder needed to appear first without undoing the order people had arranged by hand. The fix moved to the create path, and it left a wrinkle behind.
-
We hunted for one bad query and found 80 blocking ones instead
An internal overview page accumulated more than 80 blocking queries per load. The important fix was to reconsider its page contract and defer nonessential sections, not merely tune one query.
-
Why a typed date turned into 2001 on a schedule wizard
Our schedule wizard dropped day-first dates without any error. Fixing the wizard was the wrong size of fix, because seven admin screens shared the same hole.
Hitting one of these walls in your own codebase or your own machine? Start an assessment or talk to us.