8 live sessions in 3,673 rows
Out of 3,673 session rows spread across six endpoints, each answering “what’s running right now” in its own way, only 8 mattered: the live agent sessions actually in flight. Nothing on the page pointed at those 8 directly. Every time visibility broke, the fix was a seventh way of looking at the same pile.
The day before, we had run the dashboard hard. We spun up a dozen subagents in a loop because a central hub was too slow to wait on, and then had no way to tell which of the dozen were working and which had stalled. You can’t trust parallel agents you can’t see. You either babysit them one at a time, which defeats running them in parallel, or you give up on the dashboard and go back to a single terminal. We had been doing the second one for weeks.
Checking the query first
With a dashboard that feels broken, the first move is to check the query. We did, and the data was fine.
What we found instead was a subagent recorder that had been silently dead since a code change on August 22, a full week earlier. Nobody noticed because there was no one canonical place to notice it from. There were six. Two composer components at the bottom of the page were independently reading overlapping slices of the same session table and rendering slightly different subsets. A session could show in one panel and be absent from another, and that read as “flaky” when the real cause was two sources of truth.
None of this was new. A design review of this same dashboard back in June had already named the fix: one aggregation layer, a single read-only endpoint that fans out to the existing resolvers in parallel. It came with an explicit requirement that a slow downstream source return a partial or error state for its own zone instead of hanging the whole page. The recommendation was sound and it was never fully built.
What we built instead, over the following weeks, were targeted patches on the old endpoints. Each bug had a small, obvious, local fix, and the aggregation layer was a bigger job that kept losing to whatever was on fire that day. Patching had won every previous time, so we did it again. By August 29 there were six of those local fixes stacked up. Each was correct in isolation, and together they gave six different answers to the same question.
One view on the front door
The change that shipped was one canonical live-session view on the dashboard’s front door, replacing the six panels. It did not add a new endpoint. Building it forced three things that had been avoidable as long as there were six places to patch around them:
- The dead subagent recorder got rewired into the same path the new view reads from, so a recorder outage now shows up as an empty or stale view instead of silently not existing.
- The two competing composer components got merged into one. That meant first deciding which of their two definitions of “active session” was correct, which is the decision we had put off for weeks by keeping both.
- A blog module that had nothing to do with session state, but had been sitting on the dashboard since some earlier “just add it here” moment, moved to its own page. The dashboard had grown to 4,481 pixels tall, most of it content unrelated to the question the page was supposed to answer. Pulling the module out collapsed the page to 1,256 pixels.
Patching the sixth endpoint to agree with the other five was the option that lost, and it is the one that had won every time before, which is why there were six endpoints. The way out was to delete panels.
Verifying on the live page
“The dashboard says it’s fixed” was the same sentence that had been wrong six times before, so a green test suite was not the check. We drove the deployed page. We opened the live site and confirmed the recorder was populating the view in real time. We confirmed the merged composer showed one consistent session list instead of two. We confirmed the blog page rendered independently at its new address.
Two gaps turned up because we did the work instead of just closing the ticket. First, the canonical view has no fallback if you land on the front door without the keyed link that authenticates it. That is a gap in the page, and it got its own ticket rather than a follow-up comment. Second, a duplicate post surfaced on the gate board during the blog move. It had nothing to do with session visibility, and we only saw it because we were finally looking at that corner of the page directly instead of through a patch.
Six endpoints each did their assigned job correctly, and that correctness let the layering hide for months. Before approving another patch to a status page, check whether it reduces the number of places you have to look to trust the answer. When several fixes are each right on their own but the views still disagree, consolidate to one view and verify it on the live system.