Why a customer's account showed expired weeks after the charge went through

A customer had paid on time, and their account still showed expired. Nothing in the system had thrown an error, because nothing had gone wrong that the system could see.

What looked like a one-off

My first read was ordinary: a billing mix-up, the kind that gets fixed with a note and a credit. We corrected the account, applied the missing payment, and extended the term. Most versions of this story stop there.

This one didn’t, because the numbers didn’t add up cleanly. The charge was real. The record of that charge inside our own system was not. A payment had never shown up to be handled at all, so our handling of it was never the issue.

Three places to look, one that fit

There were three plausible explanations.

• A renewal step could have run and failed, which would leave a payment on record with the wrong expiry date.
• Our own handling of the notification could have accepted it and dropped it partway through, which would leave an error somewhere in the logs.
• The notification itself never arrived.

The first two leave a trace. This had neither. No payment record, no error. That absence pointed outward, past our own code, to a connection point between an outside service and ours that had stopped passing messages through. It had been that way for weeks. Customers were still being charged. Our side simply never heard about it.

Why watching for mistakes wasn’t enough

Everything we had in place watched for something going wrong after a message arrived. None of it asked whether the message had arrived at all. On a dashboard built around errors, a connection that delivers nothing looks exactly like one that is healthy and quiet. Support tickets and account checks are useful once a customer notices. They are no way to catch the problem before that.

The fix was a comparison

The instinct is to add an alert for the specific thing that broke. That patches the one incident and leaves the gap in place. The next unrelated change to the same connection point could cause the identical silence, and we would miss it again.

The fix that held up was a standing comparison between two sides. One side is what the outside service reports actually happened. The other is what our own system recorded for the same period. When the two stop agreeing, something is wrong, whether the cause is a routing change, an authentication problem, or a system on either end failing to send. The comparison doesn’t need to know which cause it is. It only needs to know the two sides no longer match.

What the AI agent did, and what stayed with us

The tracing work was a comparison an AI agent could run quickly and show its work on. It ruled out the two explanations that would have left evidence, then followed the absence to where it actually lived. It also drafted the shape of the ongoing check: what to measure on each side, and when a gap between them should count as a problem worth waking someone up for.

Deciding this deserved a real investigation instead of a one-time correction was our call. So was deciding what the customer was owed, and what counts as a normal enough gap between the two sides that the alert should stay quiet. An AI agent can show you that two numbers no longer match. It shouldn’t be the one deciding what that mismatch is worth to the business, or what happens to the account in front of it.

If your business depends on another service telling you something happened, watch for that service going quiet as well as for it failing loudly. On a dashboard that only counts errors, silence and health look identical.

dxdev LESSON · paste into your AI agent

LESSON: Check for the absence, not just the error

WHAT TO SHARE:
A de-identified description of a process where an outside system is supposed to notify yours on a regular basis (a payment, an order, a status update), and how you currently find out when something goes wrong with it.

ASK YOUR AGENT TO:
1. Name the outside fact and the inside fact that should always match (what the other system says happened, and what your system actually recorded).
2. Say whether your current monitoring would notice if the inside fact simply stopped updating, with no error anywhere.
3. Suggest one comparison, run on a schedule, that would catch a gap between the two before a customer does.
4. List what a person should decide before that alert goes live, including what counts as normal enough to not page anyone.

RETURN:
The outside fact, the inside fact, the comparison, and the one open decision a person still needs to make.

BOUNDARY: This is a review of your monitoring, not a live change. Do not have the agent build or deploy the alert without a person choosing the threshold first.