---
title: "Three crash causes behind one alert that nobody read anymore"
canonical: https://dxdev.com/everyday-ai/the-alarm-that-trained-everyone-to-stop-reading-it/
datePublished: 2026-09-10
---
A bot whose whole job was to notice and fix small problems in a piece of engineering tooling kept crashing. Each crash sent the same alert: a crash message and a timestamp, several times over, the moment it happened. After enough of those, nobody reads them the way they read the first one. That's what an alert costs when it fires without being useful. It teaches you to skip the next one, including the one that mattered.

We set aside an afternoon and pulled up the actual causes instead of reacting to the symptom each time.

## Three ordinary causes in different disguises

The first cause was a leftover marker from an earlier step that had died partway through and never cleaned up after itself. Every attempt after that hit the same marker and refused to proceed, over and over, until someone went in and cleared it by hand.

The second was a set of routine internal checks that kept failing. Nothing was actually broken. The machine was busy with something unrelated at the moment each check ran. A check that [fails once under normal load](/everyday-ai/not-every-warning-deserves-the-same-reaction/) and gets treated as a hard failure produces an alert saying "something is broken" when what happened was "the system was busy." So those checks got a retry before they were allowed to escalate.

The third was the simplest and least interesting. The process itself sometimes died and stayed dead until a person noticed and restarted it. That one needed no diagnosis. It needed to restart itself.

## The fix that looked done and wasn't

The first fix, for the leftover marker, was built the right way on paper. Before clearing anything, check whether something is still using it right now, and clear it only if that check comes back clearly clean. Refuse to guess.

It still didn't fire, on more than one separate occasion, on exactly the cases it was built to protect. The reason took a while to find. The check that confirmed nothing was using the marker relied on a single outside query, and that query was timing out under the same kind of load that causes the leftover marker in the first place. When the query timed out, the check refused to guess and left everything alone, which sounds safe in isolation. In practice the fix's own safety net went blind at the moment the underlying problem was most likely to be happening. A cautious check that can't get an answer under real load is useless right when it's needed most.

The fix for the fix added a second, cheaper way to check when the first one couldn't answer. It also added one narrower rule for when both stayed unclear: if the marker was empty and old enough that nothing legitimate could still be using it, that counted as a second, different kind of evidence, not a guess.

## Where the routine stuff goes now

Alongside fixing the mechanism, the bigger change was deciding where routine, self-resolved problems go. If the process restarts itself and comes back clean, if a check fails once and passes on retry, if a leftover marker gets cleared automatically, none of that interrupts anyone. It gets logged and rolled into a single daily summary. What still pages right away is only the thing the automatic layer tried and failed to resolve, which turns out to be a much smaller list than "anything that looked wrong at any point."

## What the assistant did and what a person decides

An AI assistant was useful for the grouping work: looking at a pile of repeating alerts, sorting them by actual cause instead of message text, and proposing a safety check that only acts on a confirmed answer. It shouldn't be the one deciding that a stalled or inconclusive check is close enough to a clean one. It also shouldn't stop paging someone without that change being visible somewhere a person can see it.

A person decides which failures are safe to auto-resolve and log for later, and which must always interrupt someone immediately, no matter how routine they start to look. That's a judgment about real risk, and it doesn't belong to the system fixing itself.

Ask whether each automatic fix still answers under load, and decide in advance which failures may be handled quietly and which must always page a person.