Why one idle SSH session vanished after minutes of agent thinking

The oldest SSH tab dropped first whenever an agent spent several minutes thinking.

At first we blamed the iPhone.

That was a reasonable diagnosis. We were running multiple SSH sessions from a phone, one tab would sit open while an agent worked, and mobile operating systems are not famous for protecting background network connections. The failure looked like backgrounding. We would return to the oldest tab, type a command, and get nothing useful back. The newer tabs were still alive. If we moved between apps, the oldest session was usually the one we had to reopen.

The explanation was plausible enough that we nearly stopped there.

The ordering was the clue

The failures had a strict order. The oldest connection died first, and newer sessions survived. A tab that was actively printing output survived. A tab that had gone quiet while the agent reasoned was the one that disappeared.

If iOS backgrounding were the whole story, I would expect the visible app state to be the main variable. A session could fail after the phone moved it out of the foreground, regardless of when the SSH connection started. What we actually had looked like an idle-timeout curve. The connection with the longest silence was consistently the first one removed.

The agent’s thinking phase made the pattern easy to miss. A long reasoning step looks like work, but it is quiet from the terminal’s perspective. No prompt is being typed and no command is producing output. The session is open, the agent is busy, and the network path sees an old TCP connection that has stopped sending packets.

We checked the wrong layer first

We began with the client because that was the part we could see. The phone was in hand, the tab was on screen, and the broken connection appeared after the SSH app had spent time in the background. That is how a plausible story gets ahead of the evidence.

We went to the logs instead.

The sshd configuration showed that server keepalives were disabled. Nothing was sending periodic server-originated traffic to check whether an apparently idle client still had a usable path. The Tailscale logs pointed the same way. The connection was not being picked at random. The quiet path was aging out.

The chain looked like this:

1. We opened several SSH sessions through Tailscale.
2. One session went quiet while an agent thought for minutes.
3. The NAT state for that quiet TCP flow aged out.
4. sshd had no server keepalive configured to generate traffic before that happened.
5. The next interaction exposed a connection that had already been removed somewhere between client and server.

Step 3 is the one that matters. TCP does not get a vote when a NAT device decides its state entry is stale. From each endpoint’s point of view, the session can look open until one side tries to send something. That is why the tab did not announce its death when the timeout happened. It looked fine until we came back and used it.

The oldest tab was not unlucky. It had simply been silent longer than every other session.

The fix belonged on the server

There were several ways to respond.

We could have changed the phone app’s keepalive setting. That might have worked for that one client, but it would have made the behavior depend on every SSH client we use. We could have added a local loop that typed harmless commands into idle terminals, which is more machinery to maintain and blurs the line between an interactive session and a healthy network path. We could have called it an iOS limitation and accepted periodic reconnects. That last option lost because the logs had already contradicted the diagnosis, and the same quiet-connection failure could hit another client on another device using the same path.

The control point that held up was sshd. We enabled a nonzero ClientAliveInterval in its configuration.

The exact interval is a policy choice. It needs to be shorter than the relevant idle timeout without turning every dormant terminal into needless traffic. What mattered was the mechanism: sshd now sends periodic client-alive requests during an otherwise silent session, so a quiet terminal no longer looks like an abandoned TCP flow.

We did not have to redesign the agent workflow, replace the phone client, or build a reconnect supervisor. One server setting restored the missing signal.

What the logs changed

Before we read them, we had a neat explanation: the phone backgrounded an app, so the app lost its connection. It fit the device in front of us and it fit the timing well enough. It also pushed us toward a client-side fix.

After we read them, the phone turned out to be incidental. The decisive facts were the disabled server keepalive, the Tailscale path, the NAT timeout, and the age of the quiet connection. That sequence explained why the oldest tab died first, why a busy terminal stayed alive, and why the disconnect only surfaced after the agent finished thinking.

The connection had already died while the tab looked idle. We just had not sent it anything yet.

If remote sessions to your servers drop after sitting idle, check whether the server sends keepalive traffic before blaming the device or the client app. In our case it did not, so we turned on a nonzero ClientAliveInterval. And when a disconnect follows an ordering like oldest first, read that ordering as the timeout showing you which clock is running.

dxdev LESSON · paste into your AI coding agent
LESSON: An Ordered Failure Pattern Is Evidence, Not a Coincidence
SOURCE: dxdev.com/blog/ssh-keepalive-oldest-connection-dies-first
WHAT HAPPENED: Multiple SSH sessions were open at once, and the oldest one consistently died first whenever it sat quiet for several minutes while a long-running process produced no terminal traffic. The first diagnosis blamed mobile OS backgrounding, which was plausible but did not explain why the failure had a strict age-based order rather than a foreground-or-background one. Reading the server config and network logs showed the SSH server had no server-side keepalive, so a NAT device along the path aged out the idle connection, and the session was already dead before the next command exposed it. The fix was one server setting, a nonzero client-alive interval.
THE RULE: When an intermittent failure has a consistent order, such as oldest-first, longest-idle-first, or largest-first, that ordering is diagnostic evidence pointing at a timeout or resource-aging mechanism, and it should be checked against logs before accepting the first plausible client-side story.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any long-lived connection (SSH, database, websocket, queue consumer) that goes idle during a long processing step, with no server-side keepalive or heartbeat configured to keep the path alive.
  2. Any intermittent-failure investigation that stopped at the first plausible explanation without checking whether the failures follow a specific order that a coincidence would not produce.
  3. Any NAT, proxy, or load-balancer boundary on the connection path whose idle-timeout value has never been compared against the application's own expected quiet periods.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.