The shadowed username variable behind 1,393 error rows in 90 days

Thirty-two error rows landed in one hour, and we read that as a fresh break in the platform. Claude started where the alert pointed, with hourly counts and the largest error clusters. We had mistaken a rising error-row count for a rising server failure rate.

That mistake cost time because the log was telling two stories at once. One was about a system that had changed. The other was about software that had been broken for longer than the graph showed.

The count moved before the system did

The first useful fact was buried in the reporting path, and the loudest error message never came close to naming it. Client-side JavaScript reporting had gone live on 11 August. It sent browser failures into the same error table that the server-side AJAX try/catch blocks already used. Our query counted occurrences, which says nothing about whether the server fault rate was changing.

The daily client series showed the new baseline. It had 76 records on 11 August, then 146 on 16 August, with ordinary variation on the days between. The server-side rate had not moved. The error log had gained a second producer, and its raw total now mixed browser observations with the server failures that used to define the alert.

We had treated the table as a stable instrument because it had the same columns and the same dashboard. Claude reinforced that by ranking the burst before either of us had established what a new record meant. The numbers were real. The implied outage was not.

The client records were still valuable. They made visible a class of failures that the older server-side path could never record. The better question was which browser failures the graph had finally started to see.

The browser was rejecting the whole file

One cluster answered that clearly. Safari 8 could not parse arrow functions or for...of syntax in the admin’s main scripts. The minifier preserved both constructs, so the shipped bundles failed at parse time. A rejected file also removes every global it was supposed to define, which is why follow-on actions produced a run of missing-variable reports.

The top recent rows showed it. 14 reports named AdvancedSettingsAdd and 13 named Clean, each across two IPs. The syntax failure sat upstream of those names. Once the browser declined the main script, every later click reached for globals that were never created.

This is the kind of defect the new reporter should surface: an old browser population had been failing with no route back to the log. The source rewrite was small, with arrow functions and for...of replaced by ES5-compatible code, but the failure was total for the affected users.

The reporter was multiplying what it found

The same investigation turned up a worse problem in the reporting layer. ClientErrorLog described two safeguards: a 10-minute duplicate window and a cap of 30 reports per hour. Both depended on the username attached to the record.

Inside Catch_Bug_Ajax, a local var username shadowed the global for the entire function because of JavaScript hoisting. The insert received a blank username. With no usable username, neither safeguard could key the report correctly. The duplicate window and the hourly cap had never fired. The extra rows had been piling up for 90 days before anyone traced them to a shadowed variable.

The flawed reporter generated 1,393 error rows over 90 days. It also gave the spike its shape. An organizer’s half-hour session had been converted into those 32 rows, a multiplication that made the new client signal look like an active incident.

What the baseline looks like now

We left the broader client-error clusters in their existing follow-up work, because this investigation had already changed the question enough. The server rate was stable. The new reporter made old browser failures observable. A missing username had disabled the controls that should have compressed repeat reports.

That hour is still in the log, and so is the 11 August boundary that explains what it is counting. When an error count jumps, check whether anything new started writing to the same log before concluding the system got worse. Here the server rate had not moved, and client reporting that went live on 11 August had added a second source of rows.

dxdev LESSON · paste into your AI coding agent
LESSON: A rising count of error rows is not the same signal as a rising rate of actual failures
SOURCE: dxdev.com/blog/the-spike-that-wasnt
WHAT HAPPENED: A 32-error spike in one hour was initially read as a fresh break in the platform. Investigation found that client-side JavaScript error reporting had gone live weeks earlier, feeding browser-side failures into the same table server-side errors already used, so the raw count now measured how many things got observed rather than how often the server actually failed, and the underlying server-side rate had not moved at all. The browser errors the new reporter surfaced were real and valuable, an old browser rejecting a modern syntax construct in a shipped script, breaking the entire file and cascading into missing-variable errors, but they represented a long-standing, previously invisible failure, not a new one. A separate bug compounded the apparent spike: a local variable in the error-logging code shadowed the global username variable due to normal language hoisting rules, so every inserted error row carried a blank username, silently disabling both intended safeguards, a duplicate-suppression window and an hourly cap, since both keyed off that field. One person's half-hour session was converted into 32 duplicate rows in the log, and the same defect had inflated the table by roughly 1,393 rows over 90 days before anyone traced it back to the shadowed variable.
THE RULE: When a monitoring table's raw count suddenly jumps, check whether a new data source was added to the same table before assuming the underlying system changed; a shared table with multiple producers measures occurrences, not a system's fault rate, and any safeguard meant to suppress duplicates or cap volume is only as effective as the field it keys off, since a silently blank or shadowed key value can disable that safeguard without throwing any error.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Any sudden jump in a monitoring or error count investigated as a system regression before checking whether a new data source or producer was recently added feeding into the same table.
      2. Any deduplication or rate-limiting safeguard that keys off a field, such as a username or session id, without verifying that field is reliably populated, especially in code with variable shadowing or scoping risk.
      3. A raw occurrence count being used as a proxy for an underlying rate or fault frequency, when the count and the rate could diverge if the population being measured changed.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.