Theme
Debugging craft
A method for finding the real cause: measure first, read the whole system, distrust the label.
78 posts by DX, in two lanes: plain-language stories and the technical build log.
Everyday lane
Plain-language stories. No jargon. 11 posts.
Start here
When the Customer Reports a Failure and Our Records Say Nothing Happened
A customer's save failed with an error on screen. Our own records had nothing matching it. The missing record turned out to be the answer, not a dead end.
Symptoms that looked like attacks or bugs
Case files of hard to find bugs
Sessions, restarts and processes that fail silently
When the bug is not where it looks
-
Why a Control Panel Logged Every 302 Redirect Against the User
A control panel kept blocking a real administrator because its safety check counted its own detours as suspicious clicks.
-
A Name From 2016 Broke the Embed Button Overnight
A browser change stopped a copy button from working for some people, and the safest fix was smaller than a full software update.
-
Why a Tiptap focus callback fired before anyone touched the writing box
A writing toolbar opened itself on page load because a software signal meant less than its friendly name suggested.
-
Why Saving an Item Dropped Us at the Parent Account After 500 Milliseconds
A small choice in a page address made a useful view easy to lose until the links learned to carry it forward.
-
Three New Fields and One Comment Beside the Old category Label
A familiar label on a web page pointed to the parent account after a new drill-down feature made the selected event different.
Blaming the wrong thing: network and certificate faults
-
A 106 Millisecond Ping Spike Sent Us Chasing the Wrong Provider
A browser kept choosing a road our computer could not use, so ordinary network tests looked healthy while pages stalled.
-
The 37 Millisecond Refusal That Fixed Nothing
A small network-setting change turned a familiar local shortcut into a ten-second wait on every page load.
More in this theme
Technical lane
The build log: the logs, the code, the fix. 67 posts.
Start here
It Looked Exactly Like a Bot Swarm. It Was SQL Server Parameter Sniffing.
Traffic-correlated CPU spikes can mean a scraper swarm or a cached bad execution plan. Here is how to tell them apart and what to do when the answer is the plan.
Symptoms that looked like attacks or bugs
-
It Looked Exactly Like a Bot Swarm. It Was SQL Server Parameter Sniffing.
Traffic-correlated CPU spikes can mean a scraper swarm or a cached bad execution plan. Here is how to tell them apart and what to do when the answer is the plan.
-
SOS_SCHEDULER_YIELD at 89% CPU and the swarm that never existed
The uptime monitor fired on a 9 PM CPU spike. Everything pointed to hostile traffic. It was one cached plan in a request-log table doing clustered scans on every request, and one OPTION (RECOMPILE) settled it.
-
The Error Spike at 8pm When Nothing Had Been Deployed
One error spike, two causes. Only one of them was real, and we filed it at the wrong priority.
-
The shadowed username variable behind 1,393 error rows in 90 days
Thirty-two error rows landed in one hour.
-
We hunted a duplicate writer for a screen showing four Discord cards
Our live screen showed the same Discord conversation over and over. We went hunting for a bug that was writing duplicates. Nothing was: each card was a separate session, spawned five minutes apart.
-
Three weeks of snapshot drift and one column named referer
`Invalid column name 'referer'` was the yellow error box on a teammate's admin schedule page. The same URL rendered fine on another machine, with the same commit, branch and code path.
-
The IIS Log Timestamps That Made a Working Block Look Broken
We spent rounds of an active production incident convinced a firewall rule wasn't working. The log timestamps looked like they fell after the rule went in, and the bad traffic was still showing up.
-
"Bamboo is broken" was wrong: a deploy that races the filesystem under CPU pressure
A stale note said the CI pipeline was broken. A bot swarm had starved the box, and the deploy's extract-then-verify step lost a filesystem race.
-
Exit 127 on a Command That Had Already Worked
Four near-identical JIRA tickets showed up seconds apart, with no human in the loop. Our first guess was a flaky board or a double-submit somewhere, and neither was the cause.
-
E00007 and I00001 Came From the Same Sandbox Login
A sandbox checkout died with \"merchant login ID or password is invalid.\" The exact same credentials had authenticated cleanly in a standalone Python proof minutes earlier, returning I00001.
-
The search Method That 404ed Against Server and Cost Us Our Login
A routine JIRA ticket-create came back 403 Basic Authentication Failure - AUTHENTICATIONDENIED with credentials we'd been using all session.
-
The Prod Box Was DDoSing Itself: An iCal Calendar Feature Looping Through the Public Edge
Two hours of IIS logs showed the prod box hitting its own schedule endpoint 19,178 times through Cloudflare: an iCal self-loop, plus the case for 410 Gone.
-
880 Tickets, One Iframe, and a Login Page the Watchdog Never Caught
Cross-origin cookie isolation means an embedded JIRA iframe always renders logged-out. The fix is a popup with a fixed window name so every navigation reuses the same window instead of spawning a new one.
Case files of hard to find bugs
-
Three Bugs Merged the Same Day Into develop From Two Hotfix Branches
Two hotfix branches merged the same day. Three tickets looked like one customer complaint and turned into three different bugs: bad payment data, wrong scope resolution, and a deploy that would have re-broken the fix by Tuesday.
-
Why a Blank Layout Slot Was Never a Rendering Bug
A stacked set of customer tickets became clearer once small, access-controlled diagnostics separated data-state problems from code regressions before remediation began.
-
The Hotfix That Shipped at 3:36 PM Without a Repro
At 3:36 PM, we shipped a release because one organization-account event was stuck and the exact trigger would not come back on demand.
-
Why a Scheduling Error Left No Trace on the Server
A save button failed with a message on screen and left nothing behind in the server logs. The missing entry turned out to be the diagnosis.
-
Why a Mobile Login Button Complaint Led Us Into the Layout-Save Module
A mobile-button complaint looked like a style ticket, but the same investigation found a boolean error in the layout-save path that could remove modules without a visible explanation.
-
Local Dev Passed Every Test While Production Served Broken /photos/ Images
Inside a day of debugging an old web form, where a querystring namespace collision hid behind an upload bug and showed what AI-paired work in legacy code looks like.
-
The COUNT(*) behind every hover label on the admin console
An internal admin screen that looked like a layout ticket required a trace of its real database work. The incident showed why query plans, scope, and user value matter more than the surface description of a slow page.
-
Five Commits Hid Behind One Closed Ticket and a 404
The ticket reopened the same day it closed. A broken admin redirect after new event creation turned into five separate failures across image persistence, custom event categories, freeform dates, edit routing, and cart verification, each hiding behind the last.
-
Why the Rebuilt Module Hid Its Competition and Find-a-Team Filters
The wrench dialog said the competition and find-a-team filters should be showing. On the live test site, they weren't.
-
The Display Type Toggle and the Breakpoint That Hid Our Controls
At desktop width, the event navigation disappeared. The controls were still in the page. They were just not where we thought the breakpoint logic would put them.
-
One Line Set auto_renew to 0 and the Event Saved
At 5:55 PM, a staff checkout for a live event refused to save because the cart still carried a recurring monthly plan. The screen said auto-renew off.
-
Blue Outlines Stuck in One Customer's Firefox. The Fix Was in Our Hover Script
Blue outlines stuck on a calendar in one customer's Firefox. The first hypothesis pointed at her machine. The outline came from a class our own JavaScript toggled on hover, and moving it to CSS ended the reports.
-
The credits_balance column that stopped matching the delete dialog
The dialog said 18 credits. The page it opened from said 0.
-
Why 499 search events on July 24 meant there was no code to write
GA4's view_search_results report was logging about 205,000 site-search events a day, and a chunk of them weren't searches at all.
Sessions, restarts and processes that fail silently
-
Weeks Blaming the Phone Connection for Sessions That Kept Ending
We spent weeks blaming the phone connection for dropped agent sessions. The Windows event log said the killer was our own restart, and the fix was two commands in the Claude app.
-
Why one idle SSH session vanished after minutes of agent thinking
The oldest SSH tab dropped first whenever an agent spent several minutes thinking.
-
Why tmux stayed alive while claude.exe quietly vanished under WSL
At 11:03 PM, a new remote-control session would register, print its join URL, and die a few seconds later.
-
claude --resume worked in a terminal and failed in a web app: the LocalSystem homedir trap
claude --resume <sid> worked in a terminal and returned No conversation found when a web app ran the exact same command, on the same machine, against a session file that demonstrably existed.
-
Chrome Was Alive on Port 9222 and Invisible Everywhere Else
Chrome was up, CDP answered, automation worked fine, but the window was nowhere. The process was running in Session 0, a Windows desktop no human is ever logged into.
-
LastResult 0x0 after switching a scheduled task off the desktop
A PowerShell window flashed up on our desktop every 30 minutes, on the dot. It was a black console that appeared and vanished while we were in the middle of something else.
-
Version 3.5.38 Stranded Itself in a Staging Directory
We pin Cursor to the taskbar. We reboot. It's gone. Not minimized, not hiding behind another window, gone.
-
Why renaming the old project drive failed after the sweep commit landed
A single environment variable was supposed to finish a drive migration. The folder rename still failed with Access Denied. The blockers were background services we had forgotten were running.
-
Eight Live Sessions Pinned a 1.7 GB Directory I Wanted to Move
Our C: drive dropped to 230 MB free out of 231 GB, and one of the fattest things we could move off it was a 1.7 GB .claude directory full of transcripts and a chrome-debug profile.
-
Our Fix for a System Freeze Caused Its Own 16-Minute Freeze
Two hard freezes in roughly two hours turned a 7.5 GB always-on box into an unavailable agent runner. The immediate temptation was to call it a memory leak. It was not.
-
Building ops-cli background After the Third Popup That Week
The PowerShell window popped up again at 1:48 PM. Third time that week. We went looking for what spawned it and found something worse: nothing on the machine could tell us what was actually running.
-
The Usage Dashboard Said 60 Percent and Nothing Was Running
The alert said 60 percent. Our usage dashboard for 2026-09-10 showed spend at nearly double the trailing week's daily average, and the first instinct was to hunt for a runaway agent, a stuck loop, a session that forgot to stop.
-
When Your Safety Guards Fail Open, Everything Overloads
The queue depth read 300. That number made the slowness real instead of anecdotal, and it turned out to be a symptom of something different from what we went looking for.
-
The 39% CPU saturation log behind six concurrent agent sessions
A powershell.exe window opened on top of an editor every time any agent session finished a turn, and on the morning we chased it six sessions were open at once.
When the bug is not where it looks
-
Two overflow:hidden Ancestors and a Click That Hit the Iframe
The report said "critical bug on the live event, the design dropdown is not working," and the menu started at y=96 while both of its ancestors' clip boxes ended at y=95.
-
Two Sessions and 245 Prompts Chasing One Collapsed iOS Banner
A long mobile-rendering investigation blamed the platform before inspection showed a responsive reorder had left the banner frame without the child that supplied its usable height.
-
Tiptap v3 Fires onFocus During Init, So Subscribe to the DOM Instead
A formatting toolbar opened when a Tiptap editor initialized instead of when a person interacted with it. Moving the listener to the editor DOM node avoided the unwanted initial state in this integration.
-
A Tiptap Editor That's Invisible Until You Add ProseMirror's Base CSS
A headless Tiptap editor mounted successfully but offered no visible empty state. A minimum height, deliberate wrapper styling, and the placeholder extension made the editing surface discoverable.
Blaming the wrong thing: network and certificate faults
-
The Registry Value 0x20 Did What the Adapter Checkbox Never Could
The pings were clean. Zero packet loss to anywhere we tested, DNS resolved in milliseconds, and a TCP handshake to port 443 completed fast.
-
Why localhost Resolved to ::1 and Stalled Every Request
A morning IPv6 tweak made localhost resolve to ::1 and every request hang for ten seconds. Timed curl probes found the dead hop in a three-hop proxy chain.
-
Nothing Was Wrong With the App Code, Yet Requests Stalled
What looked like a mysterious site hang turned out to be a boring Windows hosting default with exactly the wrong behavior for an AI-facing app. The fix took five minutes. Encoding it took another ten.
-
Two Expired Certificates, 72 Hidden Hostnames, and a Morning of Wrong Questions
Two custom domains showed a security error, and we spent the morning auditing what the customers had set up. Both were fine. The certificates had expired, and 72 more were on the way.
-
A Customer Bought Back Their Domain 12 Minutes After Our Notice
Eleven customer web addresses were already dead. Visitors got nothing. We treated that as a DNS problem, and Claude was ready to turn the same label into a batch notice.
-
1,282 Hostnames Audited and a Single Extension Caused the Real Trouble
At 2:54 p.m., the live production log showed the certificate auto-heal worker trying to process cf-heal-state.json as a command.
Plans built on numbers nobody measured
-
Why strategy A failed the 99.9% gate at 60Hz before any code ran
Exit code 4. That was Spike.Meter's verdict on the first real matrix run against the gate we'd written into the phase-1 spec for a Windows shell decorator: two physical pixels of error for 99.9% of...
-
Why ScheduleWizard accounted for 7 of 24 timeouts in one window
524s on the main site, measured over five complete days: 6 on the 24th, 9 on the 26th, 1 the day after. The Dev Notes on the ticket had called it one account's data hitting a slow path.
-
Why a suspiciously tidy 3x in the usage report deserved distrust
The counter said our subagents had burned exactly three times the tokens they had. Exactly three, to the digit, which is what a real measurement almost never gives you.
-
Why schedules broke around March 4, 1973: a getTime() digit-count sort bug
A schedule sorted correctly for years until one getTime() value gained a digit. A number-to-string promotion made the sort lexicographic, with a breaking point in 1973.
When the problem isn't where it points
-
The Actions deploy that failed with BlobNotFound all night
The deploy had been dead all night. Not a code error, not a failed test: a GitHub Actions billing block, silent from inside the repo and only visible on GitHub's org settings page.
-
Five Hours Chasing 2000.0 Minutes While One Annotation Sat Unread
The alert said the failure was a cache issue. The job had run zero steps, in two seconds, twice in a row, and the actual reason was sitting in a GitHub check-run annotation our triage script never read.
-
172 accounts held a session length the settings page could not show
A session message blamed the wrong account. Tracing it found an open redirect. Our first fix missed a second copy.
-
Four Verdicts, Four Failures, and a Drive at Zero
On 2026-09-21 the drive that holds our work was at zero bytes free. A triage agent read four failed jobs that morning, one at a time, and no verdict said what the jobs had in common.
Testing the funnel before trusting the data
-
Our Signup Conversion Drop Left No Error, No 500, and No Event
New signups had gone bad, and the first thing we typed into a fresh agent session was a request to check whether some unknown bug was killing conversions. That session opened at 5:37 PM.
-
A 59% Signup Jump Right After Shipping Structured Data
A 59% week-over-week signup jump looked like proof an SEO fix had worked, until the referral data showed most signups weren't being attributed to anything at all.
-
A 50-character email limit hiding behind a slow homepage
A long email address used to get quietly cut short on our signup form, and then rejected.
-
697 Seconds Versus 3 Seconds on the Same Password Reset Email
The p90 queue delay on our password reset email was 697 seconds during a campaign burst and 3 seconds outside one. Users had been telling us that resets and signup confirmations felt slow.
Measure first, then choose the fix
-
What the Measurements Said Before Anyone Touched the Cache Location
When a workstation slowed under concurrent work, the visible AI workload was an easy explanation. The useful lesson was to measure the whole system, test the hypothesis, and make reversible infrastructure changes that protect data and account for the actual workload.
-
Uniform state is a clue, not a conclusion
When supposedly independent copies share the same incidental history, the pattern may reveal a shared origin. The safe response is to compare evidence, protect active work, and clean up only after the explanation is verified.
Checks and guards that do hidden work
-
The photo upload rule we deleted was doing a second job
Loosening a filenames rule that seemed to only reject normal photos also reopened a door it had been keeping shut.
-
137,208 Bytes Instead of 1.43 MB After One Conditional
The upload log for August 29th shows a phone photo landing at 1.43 MB, a JPEG the customer picked at 438 KB on disk.
Hitting one of these walls in your own codebase or your own machine? Start an assessment or talk to us.