--- title: "The SSL bug was easy. The bot traffic made it a day-long incident." canonical: https://dxdev.com/blog/2026-03-27_ssl-bots-five-hotfix/ datePublished: 2026-03-27 --- My activity log for March 27 is blank. My calendar has no incident entry. But the git log has six releases shipped between morning and early afternoon. The commits are the only record the day happened at all. That gap between "calendar thinks it was quiet" and "git log shows an incident arc" is worth examining, because it captures something true about how incidents work. You don't stop to write things down. You just fix. ## the first fix was mechanical A ticket opened as a certificate renewal failure on a custom domain. The ACME challenge requires the certificate authority to hit a specific path on the domain to verify ownership. My request filter was blocking that path. That was the bug. The first patch let the challenge path through: one filter rule and one deploy. The certificate still did not renew. ## the noise was the harder problem Bot traffic was also affecting renewal. The request filter I loosened had been blocking the ACME challenge, but it had also kept out some unwanted traffic. After the change, enough traffic reached the server to drive CPU high and interfere with the renewal process. The next three releases each blocked a set of known bot signatures, then more, then more again. Each one was a targeted filter update and a fresh deploy. Each deploy took a few minutes. None of them felt decisive while I was making them. That's the thing about cascading incidents. A bug has a root cause you can aim at. An incident has a stack of causes that each look like the root cause until you fix them and the needle doesn't move. You're in a loop: apply fix, wait, check CPU, check whether cert renewed, repeat. The loop requires a tight feedback cycle, and my feedback cycle was not tight. ## flying on memory Across those first four fixes, I was SSHing into the box, tailing IIS logs in a terminal window, and checking CPU in Task Manager. That's workable for a single-variable incident. For a cascading one, it's exhausting and slow. What I needed was a status surface baked into the product itself, not a separate monitoring host or a notebook I was updating by hand. I wanted something in the admin UI that showed me, in real time, whether the current release had actually moved the needle. I didn't have that. So I built it during the incident. The fifth release added a status endpoint and a log view to the admin side of the application. That was the right call. It was also something that should have existed before the incident started. I had been running this application without a built-in status surface because I'd been managing state in my head and in SSH sessions. That works until an incident is four fixes deep and you can't tell, from memory alone, whether the last deploy did anything. ## the smallest fix explained the most The sixth release was a single line. The new log view defaulted to oldest-entry-first. Every time I opened it I was looking at records from days prior, scrolling to the bottom to find the failure I'd just triggered. The fix jumped the view to the bottom on load. That one line removed repeated scrolling from each check cycle. Over several hours, that friction accumulated. More importantly, its absence is a measurement of how rough the status tooling was when I shipped it. Jump-to-bottom is such an obvious affordance that skipping it tells you the feature was written in incident mode, not in design mode. I'm not embarrassed by that. I'm noting it because it's the honest version of how observability actually gets built in solo operations: not ahead of the incident, but during it, by someone who is simultaneously trying to fix the thing the tooling is supposed to help them see. Somewhere in the middle of that stack, a small fix for custom domain and page URLs slipped in too. That's the other thing about incident days. Adjacent work rides along on the same deploy train because the train is already running. ## what the blank activity log means I didn't write an entry because I was in the loop. Fix, deploy, check, repeat. There was no natural pause. When the incident resolved, I moved on. The calendar knows nothing about any of it. The commits are the artifact. "Bot block pass 2." "Jump log to bottom." Those titles are the closest thing to documentation I produced that day. They're thin, but they're sequenced and timestamped, and they let me reconstruct the arc later when I came back to write this. This is worth saying explicitly: during an incident, commits are load-bearing documentation. They are not just delivery artifacts. They are evidence that what you changed, you changed deliberately, in a particular order, for a reason that someone thought to name. The activity log and calendar are better surfaces when you have the headspace to update them. When you don't, the commit trail is what's left. In this incident, the hard part was not only finding the initial bug. It was learning quickly whether each subsequent change improved the system. That feedback loop is not polish to defer; it is part of operating a system under pressure. ## Related - [Six Production Releases in One Day: Anatomy of a Hotfix Train](six-production-releases-before-lunch-hotfix-train): how small release units made rollback scope easier to reason about during a separate high-tempo day - [The Most Valuable Line in an Agent's Ruleset: Verify the Fix in the Browser Before Claiming It's Fixed](close-the-verify-loop-browser-cursor-rule): the same distinction between a change made and an outcome observed