The Only Record of March 27 Was a Git Log

Six releases shipped between morning and early afternoon on March 27, and our activity log for that day is blank. The calendar has no incident entry either. The git log is the only record the day happened at all.

That gap between “calendar thinks it was quiet” and “git log shows an incident arc” says something true about incidents. You don’t stop to write things down. You just fix.

the first fix was mechanical

A ticket opened as a certificate renewal failure on a custom domain. The ACME challenge requires the certificate authority to hit a specific path on the domain to verify ownership. Our request filter was blocking that path. That was the bug.

The first patch let the challenge path through: one filter rule and one deploy. The certificate still did not renew.

the bot traffic was the harder problem

Bot traffic was also affecting renewal. The filter we loosened had been blocking the ACME challenge, but it had also kept out some unwanted traffic. After the change, enough of that traffic reached the server to drive CPU high and interfere with the renewal process.

The next three releases each blocked a set of known bot signatures, then more, then more again. Each one was a targeted filter update and a fresh deploy. Each deploy took a few minutes. None of them felt decisive while we were making them.

A bug has a root cause you can aim at. A cascading incident has a stack of causes that each look like the root cause until you fix them and the needle doesn’t move. We were in a loop: apply the fix, wait, check CPU, check whether the cert renewed, repeat. A loop like that needs a tight feedback cycle, and ours was not tight.

flying on memory

Across those first four fixes, we were SSHing into the box, tailing IIS logs in a terminal window, and checking CPU in Task Manager. That is workable when only one thing is going wrong. With several causes stacked up, it was exhausting and slow.

What we needed was a status surface inside the product itself, in the admin UI, showing in real time whether the current release had moved the needle. A separate monitoring host or a hand-updated notebook would have meant one more thing to keep current mid-incident.

We didn’t have that, so we built it during the incident.

The fifth release added a status endpoint and a log view to the admin side of the application. It was the right call, and it should have existed before the day started. We had been running the application without a built-in status surface because state lived in memory and in SSH sessions. That holds up until an incident is four fixes deep and memory alone can’t tell you whether the last deploy did anything.

the smallest fix explained the most

The sixth release was a single line. The new log view defaulted to oldest-entry-first. Every time we opened it we were looking at records from days prior, scrolling to the bottom to find the failure we had just triggered. The fix jumped the view to the bottom on load.

That one line removed repeated scrolling from each check cycle, and over several hours that friction had added up. Skipping something as obvious as jump-to-bottom also shows how rough the tooling was when we shipped it. The feature was written in incident mode, not in design mode.

That is how observability tends to get built in practice: during the incident, by whoever is also trying to fix the thing the tooling is supposed to help them see.

Somewhere in the middle of that stack, a small fix for custom domain and page URLs slipped in too. Adjacent work rides along on incident days because the deploy train is already running.

why the activity log is blank

We didn’t write an entry because we were in the loop. Fix, deploy, check, repeat. There was no natural pause, and when the incident resolved we moved on. The calendar knows nothing about any of it.

The commits are what’s left. “Bot block pass 2.” “Jump log to bottom.” Those titles are the closest thing to documentation we produced that day. They are thin, but they are sequenced and timestamped, and they let us reconstruct the arc later when we came back to write this up.

During an incident, commits are load-bearing documentation. They show that what you changed, you changed deliberately, in a particular order, for a reason someone thought to name. The activity log and the calendar are better surfaces when you have the headspace to update them. When you don’t, the commit trail is the record.

The SSL bug was the easy part. The hard part was learning quickly whether each change improved the system, and that feedback loop is something to build before the pressure arrives.

Related

• Six Production Releases in One Day: Anatomy of a Hotfix Train: how small release units made rollback scope easier to reason about during a separate high-tempo day
• The Most Valuable Line in an Agent’s Ruleset: Verify the Fix in the Browser Before Claiming It’s Fixed: the same distinction between a change made and an outcome observed

dxdev LESSON · paste into your AI coding agent
LESSON: Observability gaps turn fixable bugs into all-day incidents
SOURCE: dxdev.com/blog/ssl-bots-five-hotfix
WHAT HAPPENED: A certificate renewal failure turned out to have a trivial root cause, but the fix inadvertently loosened a traffic filter. Bot traffic then degraded the service enough to prevent renewal from completing. Without a built-in status surface, the operator had to fly on memory, SSH sessions, and Task Manager across six successive releases before the incident resolved. A log view built during the incident defaulted to oldest-entry-first, costing 30 seconds per check cycle, which reveals how rough purpose-built tooling is when written under pressure.
THE RULE: Tight feedback between a deploy and its observable effect shortens incident response. Establish that feedback surface before an incident where possible, rather than improvising it under pressure.
CHECK MY CODE, then report PASS / FAIL with file:line for each:
  1. Does the application expose a built-in status or health endpoint that reflects the current state of the service (not just an external ping target)?
  2. Does any log or event view default to the most recent entries first (newest-first or jump-to-bottom on load), rather than oldest-first?
  3. When a filter, firewall rule, or rate-limit is relaxed, is there a compensating check or test that verifies the change does not open a secondary traffic vector?
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.