Moving a 4:12am Monitor Report Ahead of the Afternoon Send

At 2:49 pm a message came in. The admin was “super slow to load.” I suspected our afternoon run, so I asked an AI agent to look. It was read-only and could not restart or kill anything without my go.

By the time it checked, the slowdown was gone. At 3:20 pm the home page, /admin/ and a team page each loaded in 0.09 to 0.17 seconds. The server’s per-minute samples from 2:50 pm to 3:17 pm were calm, with CPU between 10 and 28% and 0 to 4 slow requests a minute.

What “the afternoon run” turned out to be

I had been calling it ours, but it is the platform’s own daily “Afternoon” monitor job. It fires on the first ping after 2:23 pm. The agent found that nothing in our ops repo, no agent session and no deploy lined up with it.

Between 2:24 pm and 2:28 pm the web server’s own IP sent about 125 lifecycle emails. One of them was mailer 27, “Unsubscribe Batch (Bad Email Addresses)”. For each bad address, its code ran a leading-wildcard LIKE '%email%' scan of the mailer lists. Today that meant 39 scans between 14:25:35 and 14:27:50, about 1,127 seconds of SQL CPU. SQL CPU hit 89% at 2:26 pm and 2:27 pm, then went back to its 27% baseline by 2:29 pm.

So the afternoon run causes a real, repeatable CPU spike. That part was confirmed.

Where I was wrong

I had gone in suspecting the afternoon run was the cause. That spike was over 20 minutes before the 2:49 message, and the agent’s report said what it could not confirm: that this spike was what the reporter felt.

The likelier match was two stalls at 2:48 pm and 2:49 pm. About 15 unrelated admin and team requests hung for 15 to 28 seconds and then all finished together. SQL CPU at that moment was only 27 to 31%. Some of those requests looked like a staff session, possibly the reporter’s. A customer’s bulk schedule import was running at the same moment. The agent called blocking a guess, and so do I. We have no blocking snapshot from 2:49 pm to prove it.

Moving the scan out of prime time

I looked at the confirmed part and decided it was a bad pattern by itself. We had put the scan in the afternoon because we wanted to contact people in the afternoon. The scan could run in the morning with the results stored, so the afternoon job only has to send the mailer.

It needed to ship as a hotfix because it ties in with settings stored directly in the database. The ticket was first filed as a STAGING ticket, so I asked for it to be retyped as a HOTFIX. A sub-agent changed the issue type and read it back from the ticket system to confirm. The status was still preparing, the Fix Version was still empty, and nothing else on the ticket moved. The ticket tool’s 8-word title cap also cut the title from the 10 words in the brief.

The change shipped to production. Here is how it works:

• The earlyMorning Monitor report, which runs at 4:12 am, does the searching through a read-only action, email_badEmailScan. It logs {email: [listID, ...]} in that run’s dbo.monitor row.
• The afternoon job loads that morning row and reads only those lists, by primary key. It still does every removal, flag and notification itself.
• If an address is new since 4 am, or no morning row exists for the same report date, the afternoon job falls back to the old per-address scan. Without the morning action configured, behavior is unchanged.

The setting that turns the morning scan on was the next step after the code shipped, and two other tickets track verification of both runs. Starting the next day the afternoon spike should go away.

What is still open

What I could measure was the 39 scans and the 89% CPU spike, and that is what we moved out of prime time. The cause of the 2:49 slowness is still open. If the 2:49 pm stall happens again, a snapshot of what is blocking at that moment would settle it.

dxdev LESSON · paste into your AI coding agent
LESSON: Separate The Confirmed Cause From The Suspected One Before Fixing, And Say Which Is Which
SOURCE: dxdev.com/blog/slowdown-same-time-every-afternoon
WHAT HAPPENED: The reporter said the admin was slow at 2:49 PM, and the author suspected the daily afternoon job. By the time an agent looked, the slowdown was gone. The agent confirmed that the job ran 39 leading-wildcard LIKE scans between 14:25:35 and 14:27:50, pushing SQL CPU to 89%. That spike ended more than 20 minutes before the report, though. The likelier match was two stalls at 14:48 and 14:49, when about 15 admin and team requests hung for 15 to 28 seconds while SQL CPU was only 27 to 31%. A bulk schedule import was running then, but no blocking snapshot existed to prove it was the cause. The author still moved the scan to a 4:12am report and left the afternoon job to use the stored results. The post says plainly that the cause of the 2:49 slowness is still open.
THE RULE: When an investigation finds a real problem that does not match the reported symptom, fix it on its own merits and say so, and do not claim it explains the report. Record what was and was not confirmed, and capture the missing evidence for the next occurrence.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Compare the timestamp of each confirmed problem against the time the symptom was reported, and flag any confirmed cause that ended before the report
      2. Mark each claim in an incident write-up as confirmed, likely or guessed, and confirm no fix is described as resolving a symptom it was not shown to cause
      3. Confirm that a diagnostic snapshot, such as a blocking-query capture, can be taken at the moment of the next slowdown if the cause is still open
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.