When a Payment Confirmation Event Overwrote Decisions Made by Hand

A payment-confirmation webhook rewrote registration statuses for one organization, and no request failed. The background process did exactly what it was configured to do. The policy behind the write was incomplete.

That organization tracked approval by hand. A previous release had let the webhook update a registration status automatically, and the automation overwrote their curated decisions. There was no crash and no stack trace. The feature succeeded technically while changing a field whose meaning varied by tenant, and that kind of failure stays invisible until an administrator notices that a carefully managed workflow no longer reflects their decisions.

What the feature did

The release added a payment-confirmation-driven status update. The convenience was understandable: when payment was confirmed, the system could advance a related workflow without an extra click.

But payment confirmation and approval to participate are different facts. Some administrators used approval to reflect eligibility, paperwork, capacity, or other local policy. The automation collapsed those facts into one field and let the payment event overwrite a human decision.

One organization used the field exactly that way. The webhook’s status update replaced its manually curated decisions, and that made the policy gap visible.

The fix had two halves

The instinct is to fix the bug: roll back the write, ship it, done. That is only half the job here, and it is the less important half.

Making the new write opt-out-able per tenant

The automation was useful for some organizations, but useful did not make it safe as an unconditional write. The remediation made the policy explicit and tenant-aware, with clear ownership, discoverability, and auditability for the setting.

The right default for a newly introduced automated write depends on the risk, the established contract, the reversibility of the effect, and whether tenants have been informed. Where a field can encode a human decision, the safer design is often to keep the separate facts apart and let an explicit workflow decide any state transition. If an organization opts into automation, the system should record the policy, the triggering event, and the resulting write, so the action can be explained and reversed.

Getting the overwritten data back

The policy control stopped further overwrites. It did not repair the curated decisions that had already changed. Those records needed their own recovery plan.

The recovery process compared a reviewed candidate set against a point-in-time reference and proposed only records matching the verified incident pattern. Before applying anything, we reviewed the dry-run output, captured the affected current state, got approval, and logged each change. It stayed narrow by design. It was not a generic restore button, and it avoided overwriting legitimate changes made after the incident.

A surgical, auditable recovery beats a wholesale restore when the restore would discard valid work done after the reference point. The restraint that should keep an automation from writing broadly should also govern the repair.

Before the next automated write

Ask which fields it touches and whether any customer uses one of them to record a human decision. Make the automation a per-tenant setting with an audit record, and plan the recovery path before you need it. If data has already been overwritten, restore only the records that match the verified incident, so valid later changes survive.

dxdev LESSON · paste into your AI coding agent
LESSON: Gate automation writes that touch human-curated data
SOURCE: dxdev.com/blog/automation-that-clobbered-user-data-opt-out-plus-restore

WHAT HAPPENED:
A background webhook handler updated a status field for every tenant without knowing that some tenants managed the same field by hand.
The automation completed successfully but overwrote curated decisions without an alert or review step.
The root cause was collapsing separate facts-payment confirmation and human approval-into one write without an explicit policy or control.
Recovery required tenant-aware controls, a clearly scoped candidate set, a dry run, approval, audit logging, and a reversible restore process.

THE RULE: Before automation writes a field, identify every human and system that can also write it. Model distinct business facts separately, make the write policy explicit per tenant or workflow where appropriate, preserve provenance, and ship a tested recovery path with the change.

CHECK MY CODE, then report PASS / FAIL with file:line for each:
  1. Any automated writer that changes a status, approval, or classification field without an explicit business policy, provenance, idempotency handling, and an applicable tenant or workflow control.
  2. Any automated write that shares a field with a user-facing workflow but cannot distinguish the underlying business facts or preserve a human override.
  3. Any recovery script without a reviewed candidate set, dry-run output, backup or snapshot reference, approval record, transaction or rollback plan, and audit log of changed rows.

THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.