The spike that wasn't

Thirty-two error rows landed in one hour.

I read the spike as a fresh break in the platform. Claude began where the alert invited it to begin, with hourly counts and the largest error clusters. I had mistaken a rising error-row count for a rising server failure rate.

That distinction cost time because the log was telling two stories at once. One was about a system that had changed. The other was about software that had been broken for longer than the graph made visible.

The count moved before the system did

The first useful fact was buried in the reporting path. The loudest error message never got close to naming it. Client-side JavaScript reporting had gone live on 11 August, sending browser failures into the same error table that server-side AJAX try/catch blocks already used. The query measured occurrences, a different thing entirely from a changing server fault rate.

The daily client series made the new baseline visible. It showed 76 records on 11 August, then 146 on 16 August, alongside ordinary variation on the days between. Yet the server-side rate had not moved. The error log had acquired a second producer, and its raw total now mixed browser observations with the server failures that used to define the alert.

I had treated the table as a stable instrument because it had the same columns and the same dashboard. Claude reinforced that assumption by ranking the burst before either of us established what a new record meant. The numbers were real. The implied outage was not.

That reversal mattered because the client records were still valuable. They had made a class of failures visible that the older server-side path could never record. The better question was no longer why the graph was suddenly high. It was which browser failures the graph had finally started seeing.

The browser was rejecting the whole file

One cluster answered that question with unusual clarity. Safari 8 could not parse arrow functions or for...of syntax in the admin’s main scripts. The minifier preserved both constructs, so the shipped bundles failed at parse time. A rejected file also removed every global it was supposed to define, which is why follow-on actions produced a run of missing-variable reports.

The burst was not abstract. In the top recent rows, 14 reports named AdvancedSettingsAdd and 13 named Clean, each across two IPs. The earlier syntax failure was upstream of those names. Once the browser declined the main script, every later click reached for globals that were never created.

This was exactly the sort of defect the new reporter should surface: evidence that an old browser population had been failing without a route back to the log, not proof that the deployment had just failed. The source rewrite was small, arrow functions and for...of replaced with ES5-compatible code, but the failure mode was total for the affected users.

The reporter was multiplying what it found

The same investigation uncovered a more uncomfortable problem in the reporting layer. ClientErrorLog.asp described two safeguards: a 10-minute duplicate window and a cap of 30 reports per hour. Both depended on the username attached to the record.

Inside Catch_Bug_Ajax, a local var username shadowed the global for the entire function because of JavaScript hoisting. The insert received a blank username. With no usable username, neither safeguard could key the report correctly. The duplicate window and the hourly cap had never fired, and the counterfeit rows had been accumulating for 90 days before anyone traced them back to a shadowed variable.

The flawed reporter generated 1,393 error rows over 90 days. It also gave the 32-row hour its shape. An organizer’s half-hour session was converted into 32 rows in one hour, a multiplication that made the new client signal look much more like an active incident.

The baseline now has a boundary

I left the broader client-error clusters in their existing follow-up work. This investigation had already changed the question enough. The server rate was stable. The new reporter made old browser failures observable. A missing username disabled the controls that should have compressed repeat reports.

The 32-row hour remains in the log, and so does the 11 August boundary that explains what it is counting.

dxdev LESSON · paste into your AI coding agent
LESSON: A rising count of error rows is not the same signal as a rising rate of actual failures
SOURCE: dxdev.com/blog/2026-08-27_the-spike-that-wasnt
WHAT HAPPENED: A 32-error spike in one hour was initially read as a fresh break in the platform. Investigation found that client-side JavaScript error reporting had gone live weeks earlier, feeding browser-side failures into the same table server-side errors already used, so the raw count now measured how many things got observed rather than how often the server actually failed, and the underlying server-side rate had not moved at all. The browser errors the new reporter surfaced were real and valuable, an old browser rejecting a modern syntax construct in a shipped script, breaking the entire file and cascading into missing-variable errors, but they represented a long-standing, previously invisible failure, not a new one. A separate bug compounded the apparent spike: a local variable in the error-logging code shadowed the global username variable due to normal language hoisting rules, so every inserted error row carried a blank username, silently disabling both intended safeguards, a duplicate-suppression window and an hourly cap, since both keyed off that field. One person's half-hour session was converted into 32 duplicate rows in the log, and the same defect had inflated the table by roughly 1,393 rows over 90 days before anyone traced it back to the shadowed variable.
THE RULE: When a monitoring table's raw count suddenly jumps, check whether a new data source was added to the same table before assuming the underlying system changed; a shared table with multiple producers measures occurrences, not a system's fault rate, and any safeguard meant to suppress duplicates or cap volume is only as effective as the field it keys off, since a silently blank or shadowed key value can disable that safeguard without throwing any error.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Any sudden jump in a monitoring or error count investigated as a system regression before checking whether a new data source or producer was recently added feeding into the same table.
      2. Any deduplication or rate-limiting safeguard that keys off a field, such as a username or session id, without verifying that field is reliably populated, especially in code with variable shadowing or scoping risk.
      3. A raw occurrence count being used as a proxy for an underlying rate or fault frequency, when the count and the rate could diverge if the population being measured changed.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.