The Error Spike at 8pm When Nothing Had Been Deployed

A little after 8pm, the client-side error log lit up. New error types, several different ones, all appearing for the first time that night, clustered in the same forty-minute stretch. That pattern usually means one thing: something just got deployed and it broke.

Nothing had just been deployed.

Splitting the spike

Pulling the error detail split it into two unrelated causes. Most of the volume, maybe eight different error types, traced to a single visitor on an eleven-year-old browser. Its JavaScript engine couldn’t parse one of the site’s script files. Every function that file was supposed to define came back undefined, so every click on the page threw a different reference error. One stuck device retried the same broken action for forty minutes. That isn’t a regression, and there was nothing to chase.

The second cause was smaller in volume but real. The schedule page’s filter crashed for several distinct visitors on current browsers, and it had been doing so for at least four days before that night. It had been happening the whole time, and one of its occurrences landed inside the same window as the noisy browser.

Only one of those two things was a bug worth fixing. Filing it apart from the noise was the first thing that mattered.

Filing it at the wrong priority

The filter crash got a ticket and went into the normal queue, on track for the regular weekly release. It felt proportionate. It wasn’t a new outage, and it had already been going on for days.

For most of them the answer was no. For that one it was yes. A defect already confirmed hitting several real users repeatedly, for four days running, doesn’t wait behind the routine queue just because it wasn’t found as an emergency. So the ticket was rebuilt as an immediate fix. It was merged and deployed, and we confirmed it live on the real page with a cache-busting check before calling it done.

Why it looked like a flood

With the filter crash shipped, the spike still needed explaining. Why did a rare crash and one dead browser look like a flood at 8:50pm specifically?

The site’s error monitor counts rows in a rolling thirty-minute window and pages someone once that count crosses a threshold. Querying the same window it uses showed the count climbing from 24 errors at 8:30pm to a peak of 68 to 69 around 8:50 to 8:55pm. It then fell back to single digits by 9:25pm. That is the shape one stuck browser retrying for forty minutes would produce on top of ordinary background noise.

What the rest of the sweep turned up

Understanding the shape of the spike didn’t make the error log clean. We worked through the rest of it and checked each candidate against the tracker to confirm nothing already covered it. That turned up five more previously unfiled bugs:

• A role-management dialog that crashed for a class of staff accounts.
• A sponsors page erroring on an empty background selector.
• A schedule wizard misrouting to a handler that had been deleted years earlier.
• A chat widget wiring itself up before the feature flag that gates it was ready.
• A leftover call into an embed integration that no longer needed it.

All six, counting the filter crash, shipped as immediate fixes before the night was over.

A crash cluster that arrives at the same time isn’t automatically one incident, so split the signal from the noise first. Once a bug is confirmed as recurring and hitting several users, that fact sets its priority, whichever pass of triage happened to find it.

dxdev LESSON · paste into your AI coding agent
LESSON: A Recurring Production Crash Doesn't Wait for the Regular Release
SOURCE: dxdev.com/blog/two-incidents-in-one-error-spike
WHAT HAPPENED: A spike in the client-side error log looked like one incident. It was two: a single visitor on an eleven-year-old browser whose JavaScript engine couldn't parse a script file at all, throwing a different error on every click for about forty minutes (noise, not a bug), and a genuine crash in a filter that had been hitting real visitors on current browsers for several days running, which happened to also fire during the same window. The real bug got filed as a normal ticket, queued for the ordinary weekly release. When asked whether these should all have been emergency fixes, the honest answer was no for most of them, but yes for that one: a defect already confirmed hitting multiple real users repeatedly should not wait behind a queue built for lower-urgency work.
THE RULE: Ticket priority should be set by how many real users a confirmed bug is already hitting and for how long, not by which triage pass happened to surface it. A bug discovered during a big investigation is not automatically urgent just because the investigation was urgent, and a bug discovered during a quiet routine sweep is not automatically low-priority just because nothing was on fire when it was found.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any incident-triage or bug-filing process that assigns ticket priority based on the intensity of the search that found the bug, rather than on independently confirmed real-world impact (user count, duration, recurrence).
  2. Any error-log or monitoring pipeline that treats every simultaneous cluster of errors as one incident, with no step that checks whether the errors trace to more than one distinct root cause before deciding severity.
  3. Any place a bug confirmed to be recurring against multiple real users over multiple days is routed through a queue meant for routine, non-urgent work, with no override path.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.