A command-line tool was referenced in every one of our AI agent sessions, with a 200-line usage guide in the system prompt, supposedly saving us 60 to 90 percent of our tokens on routine git and build operations. It had actually run ten times across 4,446 transcripts, and one of those ten was the investigation that found the problem. The other 366 invocations resolved to “command not found” and silently fell back to the slow path.

The tool is rtk, a Rust binary that filters verbose command output down to the parts a model actually needs. The pitch is real: rtk git status, rtk cargo build and rtk vitest run each strip the noise and hand back a compact version. Our global instructions tell every agent to prefix commands with it. The agents believed they were using it. The output they got back said otherwise, but nobody was reading closely enough to notice.

It started with a flat question to one of our sessions: “we stopped using rtk, why?” The first move was “we do not know yet, let us check,” not an invented, plausible-sounding reason. That instinct is the whole post.

The probe

where rtk in bash returned nothing. Get-Command rtk in PowerShell returned nothing. The binary itself existed at C:\Users\<you>\.cargo\bin\rtk.exe, freshly built. It sat in a directory that appeared in neither the User nor the Machine PATH registry entry. Every rtk <subcommand> an agent issued resolved to exit code 127, command not found, and then the wrapper logic silently fell back to running the plain command. You type rtk git status, you get plain git status, and nothing anywhere tells you the optimization never happened.

That is the entire failure. There was no crash and no error in the output the agent reads. A 200-line CLAUDE.md section created the illusion of heavy use. The reality was a silent fallback, every time, for as long as the binary had been off PATH.

What the transcripts said

Out of 4,446 transcripts, only 10 had ever issued a real rtk invocation that resolved to the actual binary. Grepping the transcripts for the failure signature told the rest of the story: rtk: command not found showed up 366 times. The tool we thought was carrying our token budget had exited 127 on every one of those and fallen back to the unoptimized path.

The fix, and the band-aid we skipped

The tempting band-aid was a single symlink in .local\bin pointing at rtk.exe. That fixes exactly one tool and leaves everything else in ~/.cargo/bin equally unreachable. The right fix was to add C:\Users\<you>\.cargo\bin to the persistent User PATH, which makes rtk and every other cargo-installed binary resolvable at once.

PATH is a string you can clobber, so the careful part was the write. We read the existing User PATH first (81 entries), appended the cargo bin directory, and wrote it back as 82 entries.

Terminal window
[Environment]::SetEnvironmentVariable('Path', "$trimmed;C:\Users\<you>\.cargo\bin", 'User')

PATH is read once at process launch, so the persistent change does nothing for the already-running shell. The durable fix needs a VS Code relaunch to take effect for new sessions. To prove the fix was correct without waiting, we patched the live session’s env in place and ran rtk git status, which returned the compact filtered output immediately.

Sweeping for every other silent failure

Fixing the PATH took five minutes. It also raised a question: if rtk had been failing this way for months and leaving a clear signature in the transcripts, what else had?

A silent failure still leaves a greppable fingerprint in the transcript, even when it eats its own error output. So we stopped looking at one tool and swept all 4,446 transcripts for failure classes.

AI coding sessions write a full JSONL transcript of every command and its output. That is a corpus, and you can search it for recurring failure rather than only the one bug in front of you. We counted distinct sessions per failure class, then extracted the actual culprits:

  • command not found: 105 sessions. Top offenders by raw count were rtk (366) and our project CLI (62).
  • is not recognized (the PowerShell twin of the same problem): 48 sessions.
  • Python tracebacks: 113 sessions.
  • UnicodeEncodeError: 143 hits across 59 sessions.
  • no such file: 191 sessions. Lock-blocked: 146.

Two of these were live and systemic. Some only looked bad. The project CLI count, for instance, was historical noise from before it was installed; it resolves fine now. Triage matters here as much as it does on a flaky test suite: separate “live and systemic” from “looked bad, actually fine” from “real runtime noise” like a handled JiraError that was caught and logged on purpose.

The encoding crash behind 59 sessions

The UnicodeEncodeError class was the second real one, and it was nastier than the PATH bug because it crashed mid-output. On Windows, when a Python script prints a checkmark, an arrow, or an accented name through a captured pipe, the process can die with UnicodeEncodeError: 'charmap' codec can't encode character. The codec was charmap in 142 of 144 hits.

The repro is one line:

python -c "print('✓')"

Piped or captured, stdout encoding comes back as cp1252, because the Windows console codepage leaks into captured output when PYTHONUTF8 is unset. The glyph hits the cp1252 encoder, the encoder has no mapping, the process throws, and the traceback eats the output that would have shown what happened. It had fired in 59 distinct sessions and was still firing the day we found it.

The obvious fix is PYTHONUTF8=1 in the persistent User env. Before: stdout enc = cp1252, crash. After: stdout enc = utf-8, the glyphs print, exit 0. But a codex review pass caught why that fix is a trap: the env var is workstation-local. It heals our desktop and silently regresses on our always-on Linux box or any fresh clone of the repo. The durable fix lives in code. A UTF-8 stdio helper calls _stream.reconfigure(encoding="utf-8", errors="replace") once at the CLI boundary. It now lives in a shared lib module, is wired into the project CLI entry point, and ships as a tracked fix.

We verified it the way that proves something: we removed PYTHONUTF8 from the env, confirmed the helper still flipped stdout to UTF-8 and printed both glyphs at exit 0, and only then called it fixed. The input-side twin of this bug (a UTF-8 BOM in PowerShell-written JSON throwing JSONDecodeError: Unexpected UTF-8 BOM) got filed separately as a companion ticket.

Commands that never parsed

The biggest finding was that the most common self-inflicted failure across all our sessions was shell commands that failed to parse. PowerShell Missing errors: 562. unexpected EOF: 78. Unexpected token: 52. ParserError: 48. The cause was agents writing complex inline one-liners with nested quotes and &&, which is not valid in PowerShell 5.1. It happened three times inside the very investigation that was cataloging it.

The fix was a guardrail promoted into the global instructions: anything beyond a trivial one-liner gets written to a script file and run as a file, because a .sh or .ps1 parses cleanly every time. A class of failure that shows up 562 times points to a missing rule.

What to count

A tool your agents are told to use can fall back silently, so count how often it actually ran. Agent transcripts already record these failures, so search them for recurring error signatures and read the tally. Then fix each cause in code or in a standing rule, since a change to one workstation’s environment only heals that workstation.