The fix that ran on every request

The original ticket was a two-hour fix. Someone deleted an organization record that still had an active staff session pointing at it, and session init hit a stale reference and entered a loop: redirect to the org selector, check session, find stale org, redirect again. Staff were stuck. The fix looked straightforward.

Four hours later, a second ticket was open.

what session init actually is

Session init in this codebase is not a special function that fires once at login. It runs at the top of every authenticated admin request. Every page load. Every AJAX call. Every status ping. It is the first thing that runs on any page that requires authentication.

The fix I shipped added a SELECT to check whether the session’s current org still exists, and an UPDATE to clear it if not, directly into that function. Which meant I had added those two SQL operations to every authenticated request, including the ones where the session was healthy and no record had ever been deleted.

That was the wrong model. I had treated session init as a recovery pathway. It is a hot path.

what the pileup looked like

The symptoms did not announce themselves. No 500s. No error pages. Pages loaded slowly. That is the dangerous shape: the code was doing exactly what I told it to do, on a path that was never designed to carry that cost.

The admin panel has around a dozen active staff on a busy afternoon. Each page load now carried two extra SQL operations. Staff clicking through brackets, updating schedules, pulling reports: multiply it out. IIS worker threads started queuing. Slow pages prompted retries. Retries queued behind slower pages. The thread pool filled up with work that was, individually, fine.

I added a live SQL activity page during the incident because my other visibility into database state was not fast enough. Running a DMV query by hand in SSMS while also reading IIS logs and session traces produces wrong conclusions. The activity page pulls sys.dm_exec_requests and sys.dm_exec_sql_text on demand and renders the live picture in the admin UI itself. I should have built it before the incident, not during it.

What it showed was not a query taking too long. It was a column of near-identical SELECT and UPDATE calls, all hitting the session table, all slightly overlapping, all from different worker threads. The query ran fast. It ran constantly.

moving the code to where it belonged

The fix for the second ticket was to move the cleanup out of session init entirely, rather than optimizing the SELECT or caching the result, and put it in the one place where a stale org reference could actually cause harm: the code path that fires when session init determines the current session is invalid and cannot proceed.

That redirect to the org selector is the exact place the loop lived. So it is the exact place the cleanup needed to run: strip dead org refs, retry session init once, continue if the retry succeeds, redirect if it fails again. No SELECT on every healthy request. No UPDATE on every authenticated page load.

The IIS thread queue cleared in under two minutes after the fix deployed.

two other things from the same day

A third issue surfaced from the same afternoon. A view shipped with updated source HTML but the deployed bundle still contained CSS and JS compiled against the old structure. The view loaded unstyled. The compiled assets were not wrong in isolation. They just did not match the source they were supposed to style. A rebuild cleared it.

I also tightened two pieces of the agent tooling. The sync skill had no concept of a hotfix branch, so running it mid-hotfix would try to pull develop changes in and break the branch topology. It now detects the branch prefix and skips that step. The investigation skill previously started every investigation by waiting for me to supply the repro URL. It now reads the ticket’s Links field first and uses whatever is already there. Two minutes saved per investigation, one fewer thing to hold in my head.

the actual lesson

Defensive code in authentication and session plumbing gets scrutinized for correctness. Does it handle the bad case? Does it catch the stale reference? Does it prevent the loop? Those are the questions that get asked before it ships.

The question that does not get asked often enough is: what does it cost on the good case, which runs a thousand times more often than the case it was written for?

Before you add defensive session or auth logic anywhere in this kind of codebase, find out how often that path actually runs on the healthy case, not just the failure case. If it sits on a hot path like session init, put the check and the cleanup only on the branch that already knows something is wrong, the invalid-session redirect, not on the branch every request walks through.

dxdev LESSON · paste into your AI coding agent
LESSON: Hot-path antipattern: recovery code that runs on every request
SOURCE: dxdev.com/blog/2026-04-22_recovery-code-hot-path
WHAT HAPPENED: A redirect-loop bug was fixed by adding a stale-reference check directly into the session init function. Session init runs on every authenticated request, not just broken ones. The fix was correct for the bad case but added two SQL operations to every healthy page load. Under normal staff usage the worker thread pool saturated, slow pages triggered retries, and retries stacked behind slower pages. The system never errored. It just slowed until it stalled.
THE RULE: Recovery code belongs in the failure branch, not in the function that runs a thousand times more often than the failure it handles.
CHECK MY CODE, then report PASS / FAIL with file:line for each:
  1. Any function that runs on every request (session init, middleware, auth guard) contains SQL writes or DELETE/UPDATE calls outside of an explicit error or failure branch.
  2. Any recovery or cleanup logic (stale-ref clearing, dead-record pruning, session repair) is placed where it runs unconditionally rather than only when the bad case is detected.
  3. Any retry or re-init call inside a hot path lacks a guard (attempt counter, short-circuit flag, or prior-failure check) that prevents it from firing on healthy requests.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.