The session ran 13h 27m, and the last thing we caught before closing was that the live site still ran the payment processor’s test keys.
By then we had put a card through the live payment account, the first payment it had ever taken. We had also walked the client billing flow screen by screen, which surfaced four defects the test suite could not see. All four are fixed and deployed. We built a 22-step walkthrough too, and moved it off a chat link and onto the internal tool itself, linked from the billing ticket.
What the real payment did and didn’t prove
Everything had been proven on localhost. The real-money payment went through the live account, but it said nothing about how the deployed site was configured. “A payment succeeds against the live account” and “the site can take a payment” are separate claims, and only the first had been shown. We filed tickets for the follow-ups.
The same night, the same shape
Around the same time we built a multi-tenant CRM from a plan. It lives in an org of its own beside the main one, so the asset is owned separately. It had a full schema, row-level security and an adversarial isolation suite. We ran that suite against the live database project, and it passed 28 assertions. It also fails when one expectation is falsified, so it measures something.
Going live caught four defects a container could not show. One was a real leak: commission_balances ran with definer rights and would have handed a rep the whole org’s commission totals.
What I check before calling billing done
A passing test run, or a successful payment from a development machine, tells you nothing about which keys the deployed site is using. The live site still runs test keys, so its key configuration needs its own check, and the live checks have to run against the live system.
AI Skills
Use this lesson with the AI assistant you already use
We walked a client billing flow with a real card through the live payment account and found four defects that the test suite could not see. Just before closing the ticket, we noticed that the deployed site still ran the payment processor's test keys.
Paste the prompt, share only the context needed to answer it, and treat the result as a draft for your review. Do not include confidential information or let an AI assistant make changes without your approval.
Optional: for a visual report and saved memory, run /dxdev first.
Don’t have it? Get it at dxdev.com/skills/dxdev. The prompt works without it.
dxdev LESSON · paste into your AI coding agent
LESSON: A Passing Test Suite Only Vouches For The Environment It Ran In, So Verify Against The Live Target Too
SOURCE: dxdev.com/blog/proven-on-localhost-production-still-test-keys
WHAT HAPPENED: The billing flow counted as built because it worked on localhost, but the suite could not say what a real card, a real account and real screens would do. Walking the flow with live money surfaced four defects, and all four were fixed and deployed. Before closing the ticket, we realized the real payment proved nothing about the deployed site, which was still configured with test credentials. The same night, an adversarial isolation suite for a multi-tenant CRM passed 28 assertions against the live database project, and it also failed when an expectation was falsified. Going live still caught four more defects, including a view running with definer rights that would have handed a rep the whole org's commission totals.
THE RULE: Evidence only covers the environment where it was produced, so a passing suite or a successful payment from one place says nothing about another. Check the deployed target's own configuration and behavior separately before calling work done.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
1. For each claim of done, name the environment where the evidence was produced and confirm it is the same environment that ships, including its credentials and keys.
2. Before closing a release ticket, run at least one end-to-end pass with real accounts and real data against the deployed target, not only against localhost or a container.
3. Audit deployed configuration, such as payment keys and database view permissions, directly, and file a follow-up ticket for any gap instead of treating a passing suite as proof.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.
