We killed the assistant’s listening process on purpose, left its credential completely untouched, and sent a test alert. It went out under the assistant’s own name, formatted correctly and delivered successfully, and the dead process never read it.

That was the failure we had gone looking for. We had a rule for who a message should come from when something needed a person’s attention right away. It should go out under that person’s own assistant, the one built to act on whatever the message was about, in preference to a shared company identity that could only relay the words. The rule we wrote to enforce it was simple: if that assistant’s own credential worked, send under its name.

The credential is just an API key. It works whether or not anything is listening on the other end. So the send succeeds with the process that was supposed to act on it sitting dead, and the system reports that the right identity delivered the message. Nobody finds out that the one thing the identity was for, someone or something awake to respond, was not true. We checked for this deliberately, before it had a chance to bite anyone.

What the old check asked

if send_as(identity, message).ok:
    return "delivered"   # true even if nothing is listening

send_as().ok only tells you the API accepted the request. It says nothing about whether anything downstream will act on it. A credential working and a process being alive to use it are two separate facts, and we had built a check for exactly one of them.

Asking whether the process is alive

The fix replaced the question entirely. Instead of asking whether authenticating as the identity succeeds, it asks whether the process behind that identity is currently alive. The answer comes from a heartbeat that the process publishes on its own, on a schedule, somewhere the sender can read it regardless of whether sending would succeed:

if not heartbeat.fresh_within(identity, max_age=HEARTBEAT_WINDOW):
    fall_back()   # any read failure counts as "cannot act", no exceptions
else:
    send_as(identity, message)

If that heartbeat is missing, stale, or unreadable for any reason at all, the system falls back. It does not try to distinguish between the process being down, the heartbeat being unreachable, or some other read failure. To the person waiting on the other end, every one of those is the same outcome: they think someone is going to see this, and no one is.

Collapsing the causes into one outcome was the part we almost skipped. Our first instinct was to treat “definitely down” and “can’t tell” as different cases, maybe log the second as a warning and send anyway. But the person on the receiving end doesn’t experience the difference. A message that lands nowhere because the assistant crashed costs them exactly as much as one that lands nowhere because a status check timed out. Refusing to send under that identity, and saying so out loud, is the only version that doesn’t eventually cost someone a response they were counting on.

Every fallback says it is one

The fallback is not silent, and that mattered as much as the check. When the primary identity can’t prove it’s alive, the message still goes out, just under a different, clearly labeled identity, with a line at the top saying why: the usual assistant appears to be offline and this is a backup. If even that fails, it drops to a plain text message, also labeled. Every rung announces itself.

An unannounced fallback is the version that goes wrong for months without anyone seeing it. The day the primary identity’s token finally expires for real, everything downstream still looks fine. The label is what makes the failure visible.

Breaking it on purpose again

We tested the whole chain by breaking the primary identity deliberately and confirming that a real message arrived through the backup path, labeled correctly, rather than trusting the code because it looked right. send_as().ok is still true the instant the process behind it dies. The heartbeat check is the only one of the two that would have caught it, and it is the one we did not write until we went looking for the gap.

What to ask of your own alerting

Check whether your setup confirms that a person or agent is alive to act on a message, or only that the message was accepted for delivery. A working credential proves the second and says nothing about the first. Then make every fallback path label itself, so a failure shows up in the message instead of staying hidden.