---
title: "rtk Wasn't Broken, It Was Unreachable: How I Found 366 Silent Failures in My AI Agent Sessions"
canonical: https://dxdev.com/blog/audit-ai-agent-transcripts-like-flaky-test-suite/
datePublished: 2026-06-02
---
A command-line tool was referenced in every single one of my AI agent sessions, with a 200-line usage guide in the system prompt, supposedly saving me 60 to 90 percent of my tokens on routine git and build operations. It had actually run ten times across 4,446 transcripts, and one of those ten was the investigation that found the problem. The other 366 invocations resolved to "command not found" and silently fell back to the slow path.

The tool is `rtk`, a Rust binary that filters verbose command output down to the parts a model actually needs. The pitch is real: `rtk git status`, `rtk cargo build`, `rtk vitest run`, each one strips the noise and hands back a compact version. My global instructions tell every agent to prefix commands with it. The agents believed they were using it. The output they got back said otherwise, except nobody was reading closely enough to notice.

It started with a flat question to one of my sessions: "we stopped using rtk, why?" The honest first move was "I don't know yet, let me check," not inventing a plausible-sounding reason. That instinct is the whole post.

## The probe

`where rtk` in bash returned nothing. `Get-Command rtk` in PowerShell, nothing. The binary itself existed at `C:\Users\<you>\.cargo\bin\rtk.exe`, freshly built. It was sitting in a directory that appeared in neither the User nor the Machine PATH registry entry. Every `rtk <subcommand>` an agent issued resolved to exit code 127, command not found, and then the wrapper logic silently fell back to running the plain command. You type `rtk git status`, you get plain `git status`, and nothing anywhere tells you the optimization never happened.

That is the entire failure. Not a crash, not an error in the output the agent reads. A 200-line CLAUDE.md section created the illusion of heavy use. The reality was a silent fallback, every single time, for as long as the binary had been off PATH.

## The myth-busting number

Out of 4,446 transcripts, only 10 had ever issued a real `rtk ` invocation that resolved to the actual binary. Grepping the transcripts for the failure signature told the rest of the story: `rtk: command not found` showed up 366 times. The tool I thought was carrying my token budget had been exit-127ing 366 times and falling back to the unoptimized path on every one.

## The fix, and the band-aid I didn't take

The tempting band-aid was a single symlink in `.local\bin` pointing at `rtk.exe`. That fixes exactly one tool and leaves every other thing in `~/.cargo/bin` equally unreachable. The right fix was to add `C:\Users\<you>\.cargo\bin` to the persistent User PATH, which makes `rtk` and every other cargo-installed binary resolvable at once.

The careful part: PATH is a string you can clobber. I read the existing User PATH first (81 entries), appended the cargo bin directory, and wrote it back as 82 entries, not 1.

```powershell
[Environment]::SetEnvironmentVariable('Path', "$trimmed;C:\Users\<you>\.cargo\bin", 'User')
```

PATH is read once at process launch, so the persistent change does nothing for the already-running shell. The durable fix needs a VS Code relaunch to take effect for new sessions. To prove the fix was correct without waiting, I patched the live session's env in place and ran `rtk git status`, which returned the compact filtered output immediately.

## Once you know failures leave a fingerprint, sweep for all of them

Fixing the PATH took five minutes. What it opened up was the question: if `rtk` had been failing this way for months and leaving a clear signature in the transcripts, what else had?

The `rtk` find proved something useful: a silent failure still leaves a greppable fingerprint in the transcript, even when it eats its own error output. So I stopped looking at one tool and swept all 4,446 transcripts for failure classes.

AI coding sessions write a full JSONL transcript of every command and its output. That is a corpus. You can mine it for *recurring* failure, not just the one bug in front of you. I counted distinct sessions per failure class, then extracted the actual culprits:

- `command not found`: 105 sessions. Top offenders by raw count were `rtk` (366) and my project CLI (62).
- `is not recognized` (the PowerShell twin of the same problem): 48 sessions.
- Python tracebacks: 113 sessions.
- `UnicodeEncodeError`: 143 hits across 59 sessions.
- `no such file`: 191 sessions. Lock-blocked: 146.

Two of these were live and systemic. Some only looked bad. The project CLI count, for instance, was historical noise from before it was installed; it resolves fine now. Triage discipline matters here as much as it does on a flaky test suite: separate "live and systemic" from "looked bad, actually fine" from "genuine runtime noise" like a handled `JiraError` that was caught and logged on purpose.

## The encoding crash that ate 59 sessions

The `UnicodeEncodeError` class was the second real one, and it was nastier than the PATH bug because it crashed mid-output. On Windows, when a Python script prints a checkmark, an arrow, or an accented name through a captured pipe, the process can die with `UnicodeEncodeError: 'charmap' codec can't encode character`. The codec was `charmap` in 142 of 144 hits.

The repro is one line:

```
python -c "print('✓')"
```

Piped or captured, stdout encoding comes back as `cp1252`, because the Windows console codepage leaks into captured output when `PYTHONUTF8` is unset. The glyph hits the cp1252 encoder, the encoder has no mapping, the process throws, and the traceback eats the output that would have shown you what happened. It had fired in 59 distinct sessions and was still firing the day I found it.

The obvious fix is `PYTHONUTF8=1` in the persistent User env. Before: stdout enc = cp1252, crash. After: stdout enc = utf-8, the glyphs print, exit 0. But a codex review pass caught why that fix is a trap: the env var is workstation-local. It heals my desktop and silently regresses on my always-on Linux box or any fresh clone of the repo. The durable fix lives in code, not in the environment: a UTF-8 stdio helper that calls `_stream.reconfigure(encoding="utf-8", errors="replace")` once at the CLI boundary. It now lives in a shared lib module, wired into the project CLI entry point, and ships as a tracked fix.

I verified it the way that actually proves something: I *removed* `PYTHONUTF8` from the env, confirmed the helper still flipped stdout to UTF-8 and printed both glyphs at exit 0, and only then called it fixed. The input-side twin of this bug (a UTF-8 BOM in PowerShell-written JSON throwing `JSONDecodeError: Unexpected UTF-8 BOM`) got filed separately as a companion ticket.

## The class that dwarfed everything

The biggest finding wasn't a tool or an encoding. It was that the single most common self-inflicted failure across all my sessions was shell commands that fail to *parse*. PowerShell `Missing` errors: 562. `unexpected EOF`: 78. `Unexpected token`: 52. `ParserError`: 48. The cause was agents writing complex inline one-liners with nested quotes and `&&`, which is not valid in PowerShell 5.1. It happened three times inside the very investigation that was cataloging it.

The fix for that one wasn't more code. It was a guardrail promoted into the global instructions: anything beyond a trivial one-liner gets written to a script file and run as a file, because a `.sh` or `.ps1` parses cleanly every time. A class of failure that shows up 562 times is not a series of typos, it's a missing rule. Your transcripts already counted them for you. The only question is whether you ever read the tally.

## Related

- [RTK Ate My JSON and Hid My Merge Commit: When a Token-Saving Filter Lies to Your Agent](rtk-token-filter-lied-to-my-agent-lossless-on-load-bearing-paths): the follow-on discovery made possible by transcript auditing
- [Reverse-Engineering Claude Code's Session JSONL (So You Can Mine Your Own Transcripts)](reverse-engineer-claude-code-jsonl-transcript-format): the format reference for the transcript data this audit relies on
- [Your Committed Output Is a Free Golden Test](committed-output-is-a-free-golden-test): using existing artifacts as a verification baseline for agent behavior
- [A Hash-Chained Manifest That Catches a Single Injected Byte, in 85 Lines, Zero Deps](hash-chained-manifest-tamper-evident-85-lines): making agent output tamper-evident via structural verification
- [Same SHAs in Two Repos: Why Commit-Count Metrics Lie in a Clone Workflow](dedupe-by-sha-not-repo-mirrors-clones-inflate-metrics): another metrics-layer lie that transcripts can surface
- [My agent scheduled a daily job against a repo that didn't exist](agent-scheduled-job-against-hallucinated-repo): a silent agent error that transcript auditing would have caught