The status alert said the relay had been down for 22 minutes, and the relay was a box we thought we understood.

It terminates TLS for the bare domain of every customer site. That is over a thousand domains, listed in a flat file at /opt/relay/apex-allowlist.txt, all served by Caddy on a 458 MB droplet. The www hosts run through a separate path and were fine. Only the bare apexes were dead. Port 22 answered and port 80 refused the connection, so the box was up and Caddy was gone. The kernel OOM killer had taken it, and nothing brought it back.

Getting it back

We asked the agent working the incident for the first move. It wanted a go-ahead before touching prod, and it offered to open a ticket for the root cause. We said “no tix, fix now.”

systemctl start caddy, then a check that 80 and 443 were listening, then curls against real customer apexes. They returned 301 to www with a clean TLS verify. The outage ran from 02:01 to 02:23 UTC.

The root cause was two missing things on a small box:

  • No swap. Any memory spike went straight to the OOM killer.
  • No restart policy. The packaged Caddy unit has no Restart=, so a killed process stays killed.

The fix was a systemd drop-in at /etc/systemd/system/caddy.service.d/10-restart.conf:

[Unit]
StartLimitIntervalSec=600
StartLimitBurst=20


[Service]
Restart=on-failure
RestartSec=5s

We set the start limit generously on purpose. A crash loop on this box is still better than a permanent outage. We added a 1 GB swapfile with vm.swappiness=10, so swap works as a cushion and not a paging target. We applied the same hardening to the second droplet and resized the relay for real headroom, because swap on a box this small only delays the next kill.

Killing it on purpose

A config file is a claim, so we killed the thing. We recorded the main PID, sent SIGKILL, and polled port 443 from our machine for the gap. Refused, then back after 5.9 seconds. The PID had changed and NRestarts read 1.

The kill command printed Failed to send signal SIGKILL to auxiliary processes: Invalid argument. That reads like a failed test, but the main process died anyway and the restart policy fired. The polling loop gave us the real answer, so we trust the port over the exit message.

The line in our incident note

In our own incident note we wrote: “The alert did its job, probe-backed down, paged on the first build, exactly as designed.”

That was true of the alert, and we let it stand in for the whole system. A green verdict on the monitor told us nothing about the jobs the monitor doesn’t watch. The relay’s allowlist is refreshed by a sync job, and while we were reading the allowlist file to size the incident, we found that job had not run since June. That is about two months of silence. No alert fired, no error was logged, and nothing was down. The file still had all its lines, so it looked fine.

A silent failure and a loud one are different problems. A crash that leaves a service dead is loud, and a restart policy fixes it. A job that stops running and leaves stale but plausible output behind needs a check that asks whether the output is fresh. The restart policy we had just proven covers the first kind and does nothing for the second.

A ticket filed where the job never lived

We had said no ticket for the outage, but this one deserved a follow-up, so we filed one for the allowlist sync in the platform’s tracker.

Later that link was gone. The ticket had never belonged there. The sync never lived in the platform repo. It runs as a scheduled task on one of our own machines, outside anything we deploy. We had filed the work in the place we assumed it lived, and the assumption cost us the trail. Whoever followed that ticket would have searched a repo that has never contained the job.

So the follow-up went stale for the same reason the job did. Both lived outside the system that would have noticed. The fix for the ticket was to strip the dead link and record the real location. The fix for the job is what we still owe: a freshness check that fails loudly when the allowlist’s last successful sync is older than a day.

Rebuilding the alert path

The outage also exposed how alerts reach us. The old path was one channel, and we were the single point of delivery. We rebuilt it into a ladder that says out loud which rung is speaking:

  1. Our own agent for that venture, checked through a status beacon that says whether it is listening.
  2. That venture’s always-on bot, announced as the backup.
  3. SMS, announced as the last resort.

All three rungs were exercised end to end, including a real text. Status pages now send from the agent when it is listening and fall back with the reason stated when it can’t act.

The ladder guards against our not hearing an alert. It does nothing about an alert that never fires, and the allowlist job proved that after the fact. A crash cost us 5.9 seconds once the restart policy was in place. The job nobody was watching cost two months of stale data, and the only thing that catches that is a check on when it last succeeded.