What the First Real Account Found That Sample Data Missed

The first real customer account we ran through the conversion tool turned up two problems that testing had never touched. The tool had already been built and tested. It moves an account from a generic configuration onto its category’s own dedicated code, the same move every account in that category will eventually need. Sample data had never exercised either problem.

The gate that was never built

This category has a stats feature, entry of detailed data. Two comparable categories both restrict it to a paid tier. This one didn’t restrict it at all. Every account on it, whatever the tier, had full access to a feature that was supposed to require an upgrade.

The account we were converting was on the entry tier, so the gap showed immediately. After the fix, that account lost access to a feature it had never been entitled to and never should have had.

Nothing we found recorded this as an intentional exception. It looked like a gap in the category’s original build that nobody had closed, in code the other two categories got right from the start. It surfaced now because this was the first time a real entry-tier account ran the detailed stats feature through this code path with anyone checking whether it was gated.

The second gap: photos left behind

The conversion also has to move photo assets to the new storage location. That step was a separate option, and the main move didn’t enforce it. If you skipped it, the photos stayed behind and nothing complained. There was no failed job and no flag, and nothing a future run or a support ticket could use to show the conversion was incomplete.

It only bites when someone runs the conversion without remembering the extra step. Sample data never makes anyone remember it.

What verification looked like

We checked both fixes against the real account on both sides of the change.

• Tier gate, before: the gate was open and the feature worked without a paid tier.
• Tier gate, after: the gate applied, and a comparable paid-tier account kept full access exactly as before.
• Photos: a live dry-run against the real account confirmed the tool now moves photos by default. That day’s real conversion had already exercised the underlying move code the fix now defaults onto.

The gate fix collided with a mailer

Earlier, we had sent a conversion mailer to a large batch of customers. It told everyone the detailed stats feature came with their account regardless of tier. That was true, but only because the gate had never existed. Once the tier gate was fixed, the promise was false for anyone below the paid tier.

Why the first real run needs its own review

A suite built on synthetic accounts would never have caught either problem. Synthetic accounts don’t carry the accumulated shape of years of real usage. Here that meant a tier nothing had ever had reason to check, and an optional step nothing had ever forgotten to take.

A clean test environment can’t expose assumptions that were only true because nothing real had tested them yet. The first real account does.

What to check

Treat the first run of any migration against a real customer account as its own review. Check which paid-tier entitlements are actually enforced, and which optional steps can be skipped without an error. A missing tier gate made an earlier mailer’s promise false, so check what customers have already been told before you close a gap like this one.

dxdev LESSON · paste into your AI coding agent
LESSON: Running the First Real Account Through a New Path Is Its Own Test Suite
SOURCE: dxdev.com/blog/first-real-run-found-the-gap
WHAT HAPPENED: A tool converted an account from a generic configuration onto that category's dedicated code path. The first time it ran against a real customer account rather than test data, it surfaced two problems test data had never triggered: a paid-tier gate that two comparable categories both enforced had simply never been built for this one, letting every account on it use a paid feature for free since the feature was added years earlier; and moving the account's photo assets to the new storage location was a separate option rather than something the main move enforced, so skipping or forgetting it silently left photos behind with no error to signal the gap. Both were fixed and verified against the live account before and after the change, then shipped as hotfixes. A separate mailer had already told a large batch of customers the paid feature was included at every tier, which the tier-gate fix now contradicted, so that mismatch was handed to a follow-up ticket rather than resolved inline.
THE RULE: A feature can pass every test built against synthetic or sample data and still carry a real gap, because the gap only shows up against the shape of an actual account: real data volume, a real subscription tier, a real history of partial edits. The first time any migration or conversion tool touches a real account, treat that run as a verification pass in its own right, not a routine execution of already-tested code. Check the account against its peers on adjacent, comparable paths (a similar feature in a similar category, a similar tier on a similar account) rather than only checking it against its own prior state, because a gap that has existed since the original build will look completely normal in isolation.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any feature-gating or tier-restriction logic that is duplicated per category (plan, product line) rather than shared, where one category could silently lack a gate the others enforce.
  2. Any migration, import, or conversion script where moving secondary data (assets, related rows) is a separate opt-in step rather than something the main move enforces, so skipping or forgetting it leaves orphaned or incomplete data with no error surfaced.
  3. Any marketing or customer communication (a mailer, a changelog, an announcement) asserting a feature is available at every tier, cross-checked against the actual gating code for that feature.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.