At 2:07 PM on June 22, we fixed a payment that had been charged but had never reached the account that paid it. The customer had done their part. Our recurring-payment integration had not.
The account had expired because the payment record never appeared in our application. The charge was real. The renewal was not. We corrected the account, applied the missing payment, extended the term through August 15, and added a free month for the trouble. Then we opened an incident instead of filing it as a one-off support mistake.
That decision uncovered a worse problem. Since late May, an edge proxy had been blocking posts from the recurring-payment gateway before they reached our webhook handler. Customers could pay and the gateway could accept the charge, but our system received nothing. There was no application exception because the application never got a request. There was no alert because we had never defined a missing stream of payment events as an alertable failure.
The integration did not fail loudly. It failed by going quiet.
From an expired account out to the edge
The first symptom looked like a normal reconciliation issue: a charge in June, an account still showing expired. We had three plausible places to look.
The first was the account update itself. A valid payment might have arrived and a renewal job might have failed afterward. The second was our payment-recording path. The handler might have accepted the webhook but failed before committing the payment. The third was delivery. The gateway might have charged the customer but never completed the request into our system.
Each of those leaves different evidence. A renewal bug leaves a recorded payment with an unchanged expiry date. A handler failure leaves a request or an application error. This incident had neither. It had a real charge and no matching internal payment event.
The missing payment ruled out the cleanest version of an account-state bug, and the absence of any handler-side record pushed the investigation outward. We traced the gateway’s posts through the public route and found that the edge proxy had been silently blocking them. The request was stopped before it crossed into the application boundary.
We had monitoring around code that ran after a request reached us. We had nothing watching whether a critical outside system was still reaching us at all. A blocked webhook is not necessarily an application error. It can look like nothing.
Why the existing signals told us nothing
We had support tickets, account inspection, and payment reconciliation. Those are useful after someone has found a bad outcome. They are not integration health checks.
The delivery path crossed the gateway, the internet, our edge proxy, the public route, the webhook handler, the payment record, and the account update. The block sat outside the part of the chain that produced normal application logs. Handler-error monitoring could reveal a malformed payload or a failed database write. It could not reveal that the handler had received zero valid callbacks for weeks.
The customer ticket ended up as the detector, which is the wrong job for it. An expired account after a valid payment tells you a customer was already hurt. By then we were repairing trust as well as data.
The incident took 2 hours and 15 minutes to resolve from the initial payment report. The root-cause investigation took another 1 hour and 17 minutes. That time went to proving which boundary had gone silent and how long it had been silent. The code fix was small.
A monitor with two sides
The monitoring follow-up we filed avoids a generic “webhook failed” alarm. A single request failure is noisy. Gateways retry. Networks flap. Some payment schedules naturally produce uneven traffic. An alarm that pages on every non-200 response becomes a notification people learn to ignore.
The pattern is a two-sided health check. One side measures the outside fact that matters, payments settled by the gateway. The other measures the inside fact, recurring-payment events recorded by our system. The monitor compares the two over the same window and investigates a gap.
integration: recurring-payment-webhookobserve: outside: gateway recurring payments accepted inside: payment events recorded by the applicationalert_when: expected_events_are_missing: true outside_and_inside_counts_diverge: truecontext_to_capture: last_successful_delivery public_route_response edge_proxy_decision affected_payment_windowThe configuration is deliberately about evidence. Normal volume and payment cadence set the threshold. A system with continuous recurring activity can alert after a short quiet period. A system with sparse schedules needs a longer window and a reconciliation check. In both cases the rule is the same: alert when the system producing payments and the system recording them stop agreeing.
The second signal matters because it catches more than edge blocking. It would also surface a disabled route, an authentication change, a certificate problem, a handler that accepts requests but does not persist them, or gateway configuration drift. The failure modes differ, but the customer-visible outcome is identical: money moved and the account did not.
The options we passed on
One option was a route-specific fix: restore the blocked delivery, write down the edge-proxy behavior, and move on. That repairs this exact incident but keeps the detection gap. Another edge rule, an upstream configuration change, or a future webhook integration could recreate the same class of failure.
We also passed on a monitor that only watches application errors. It gives false confidence at precisely the boundary where this incident happened. Zero requests can look healthier than a burst of errors if the dashboard only displays failures from code that ran.
A scheduled manual reconciliation was the third option. It would have found mismatches eventually, but the team would still discover a live break through batch review. The missing-event monitor detects the integration losing its pulse while there is still time to restore delivery and make affected accounts whole quickly.
We restored delivery, corrected the affected customers, documented the edge-proxy gotcha, and created the monitoring work so recurring payments cannot disappear again. A system that should be receiving money events and goes silent is in a failure state, and it needs its own evidence and its own alarm. A webhook route is healthy when the events that should arrive still arrive, get recorded, and produce the state change customers paid for.