Thirteen Hours Lost to a Payment Receipt and the Wrong Number

Thirteen hours. That was how long it took us to learn that a payment receipt had been looking for an order with the wrong number.

On paper it made no sense. Someone paid with a card. The payment service sent back the result. The receipt page should have found the order and said thank you. Instead it was handed a number that looked official and got nowhere with it.

Pretending did not settle it

We started at a computer at our desk. We tried to copy the message the payment service was supposed to send and feed it into the page by hand. It did not settle the question. The service only sends the real message to a public web address, not to a machine on our desk. Every pretend version was built on what we thought the message looked like, and that was the very thing we needed to check.

So we put a small, temporary relay page on the live site. A payment callback is the message a payment service sends back after a card is processed. The relay page had one job: receive the real message and let us see what was in it.

It was not elegant. It was faster than spending another hour arguing with a made-up version of the facts.

Two numbers, and the wrong one

The real message carried two numbers that are easy to confuse. One was the payment service’s transaction number, much like the number printed at the bottom of a store receipt. The other was the invoice number that had been attached to the order before the card was charged. The receipt page had been using the transaction number to search for the order.

That number was real. It just was not ours.

Our order number was in the invoice field. The transaction number belonged in a separate place, as a record of what the payment service had done. The old rule tried one number and then the other. It had appeared to work in testing and failed when real payments arrived. Once we could see the actual message, the fix was plain: use the invoice number to find the order, and keep the payment service’s number as its own piece of information.

What else the real message showed

The order number from that public form was going straight into a request to the records system. A public payment address is still public. Anyone can send a form to it, not only the payment service, and a stranger could type anything into the order-number box.

We added a basic guard. Before the number can be used, it has to be a real number greater than zero. It is the same check you would make before writing a house number on an envelope: if the address is empty or made of letters, you do not mail it and hope.

Another failure made no noise at all. The receipt sometimes showed a blank merchant name. The page already had the account name, but it was asking for it again from the wrong place. That extra request was wrapped in a safety net that swallowed the error, so nothing visibly broke. There was just an empty line where a name should have been. We removed the unnecessary request and used the information already on the page, with sensible fallbacks if it was missing.

We were also carrying one assumption over from a different payment service. We had treated this system as if it needed a second conversation with the provider before the charge could count. It did not. By the time the message reached the receipt page, the card had already been processed. The page needed to read the result, mark the order paid when the result said success, and keep the transaction number. Nothing more.

Seven small releases

The work went out in seven small releases, from 08:06 in the morning to 20:52 that night. Each had a label, like a bookmark in a long book. One changed which number found the order. Another fixed the name on the receipt. Another removed a broken detail from the page.

We could have gathered everything into one large change. That would have made it harder to tell which piece caused trouble if something went wrong. Small releases gave us a known good place to return to after every step. They also made the temporary relay page less frightening. It was disposable scaffolding, and it left cleanup behind, including backup and scratch copies, but no single release made the site impossible to undo.

In the end, thirteen hours came down to one swap: an invoice number in place of a transaction number in a single lookup. The receipt page now checks the right number first, and the relay page that proved it is gone, its job finished the moment the message showed its true shape.

If your payment receipt cannot find an order, find out which number the lookup uses: your own invoice number, or the payment service’s transaction number. Treat any public payment address as open to anyone, and check the order number before it reaches your records.

dxdev LESSON · paste into your AI coding agent
LESSON: Stop guessing what a webhook sends. Capture the real payload.
SOURCE: dxdev.com/everyday-ai/thirteen-hours-to-learn-which-number-was
WHAT HAPPENED: A payment receipt page couldn't locate the order it was supposed to confirm, because it searched using the payment gateway's own transaction number instead of the merchant invoice number the gateway actually returned in a separate field. Thirteen hours were spent before a temporary relay page, deployed to the live site for the sole purpose of capturing the gateway's real callback payload, revealed the field confusion directly. The fix also added input validation (must be a positive integer) on the order-number field, since a public callback endpoint can be hit by anything, not only the real gateway.
THE RULE: When a third-party callback's exact payload is in question, stop guessing from documentation or memory and capture what it actually sends in the real environment. Any identifier pulled from a public inbound endpoint needs validation before it's used to look up records.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any payment or webhook handler that assumes a specific field name for an order or transaction identifier without a comment citing the exact captured payload it was verified against.
  2. Any handler that takes an identifier from a public inbound request and uses it in a database lookup without validating its type or range first.
  3. Any place two similarly-named IDs (a gateway transaction ID vs a merchant invoice ID) could be swapped, with no test exercising both against production-shaped data.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.