One task-bot email reached our inbox three times on October 1. “[TASK-BOT] (monthly) Verify Last Month” came at 4:14 AM, then twice more at 10:06, while every other task-bot email that morning arrived once.

The 4:14 copy was legitimate. Our early-morning Monitor report is scheduled for 4:12 AM and gated to run once a day, so dbo.monitor normally holds one run per day for it. The two 10:06 copies were reruns.

Finding the caller without being able to reproduce it

The IIS log on the production server had it:

14:05:53  /ajax/monitor-report?reportType=earlyMorning&runType=solo  HeadlessChrome/150
14:06:00  /ajax/monitor-report?reportType=earlyMorning&runType=solo  HeadlessChrome/150

14:06 UTC is 10:06 EDT. No cookie and no username were logged. The same user agent had made extra runs on 09-25, 09-26 and 09-29, and dbo.monitor showed multi-run days on 09-17, 09-22, 09-23, 09-24, 09-25, 09-26, 09-28, 09-29 and 10-01.

We couldn’t test any of this by loading the URL, because loading the URL is the bug. Every clone of the app talks to the live database and live mail, so a single test request sends real email to real staff. Everything was verified by reading code, bounded read-only SELECTs, and unit tests.

The mechanism is a pair of links on the staff Monitor page, “RUN AGAIN & view report” and “RUN with raw data”. Both carry runType= and bypass the once-a-day gate. Anything that requests one of those URLs re-runs every action in the report, including the ones that send email.

The first suspect was our own site crawler, since a 09-22 note in its source says it followed these links and caused the earlier duplicates. This time the user agent was HeadlessChrome, and our tooling’s saved browser sessions held a headless tab that named the run URL. That tab was restored and re-requested it. Nobody caught the tab in the act, so that is the best-supported cause rather than a reproduced one. The tooling fix quarantines saved tabs like it.

The fix that was real and didn’t help

We had been wrong about this once already. After the 09-22 duplicates, we read the Monitor code and found a real race. The report’s due-check does a SELECT on dbo.monitor and then INSERTs its claim row as two separate round trips, with no lock and no unique index. Two page loads at the same moment could both start the same report. We verified that and built a claim lock: a transaction-scoped sp_getapplock around the claim, plus a prepared filtered unique index. It passed review and landed on 09-23.

Then dbo.monitor showed extra runs on 09-24, 09-25, 09-26, 09-28 and 09-29, and three copies of an email on 10-01. The lock did exactly what we’d built it to do. It stops overlapping runs, not a second run after the first finishes. The two requests in the log came seven seconds apart. We’d fixed a real race, just not the one producing the duplicates, and those kept going for another eight days.

Why only one recipient tripled

On a rerun, real staff are skipped. The reminder code checks people.staffLastNoticeSent and sees they were already notified today. The hardcoded “Verify Last Month” task goes to a placeholder “Support Staff” recipient that has no peopleID, so that check never applies to it. It re-sent on every rerun, which is how one email became three.

In plain terms, the safety net covered every real person and missed the one recipient that wasn’t a person. We got lucky that the only victim this time was an internal inbox. The same kind of rerun is how 161 duplicate customer emails went out across 09-17 and 09-22.

What shipped

We approved a hotfix with two parts on the app side:

  • A rerun of the staff reminders on the same day must not re-send to a recipient with no people row. The brief said to prefer a check keyed on what was actually sent that day, rather than inventing a fake person record to hang a flag on.
  • A rerun from the Monitor page must not silently repeat email-sending actions, and the report page now requires a staff login. Before that, a request with no cookie was enough to run it, which is what the log showed.

The release note for it reads “Re-running a monitor report no longer re-sends the staff and customer emails that already went out the first time.”

On our own tooling, the browser tooling now refuses any request to the monitor report endpoint and any Monitor link carrying runType=, whatever its value. We asked for tests that prove the guard fires on the exact URL from the log. Passing normal URLs proves nothing about a guard.

The lock answered “is this running right now?” The question that mattered was “did this already happen today?”, and we had answered the wrong one. We kept both hotfix tickets open until the next day’s scheduled 4:12 AM and 2:23 PM email runs were checked.