78 of 12,866 Requests Came Back as IIS 502 Pages

On a phone at 17:44 UTC, 78 of 12,866 requests that day came back as IIS’s own 502 page. All of them were hits on a Chrome upload endpoint’s transcript replay. The app server was running under tsx-watch. So every file save restarted Node mid-request.

We moved it to the built production server, node dist/index.js with NODE_ENV=production on port 3010. We stopped the spine loader from painting the raw upstream error body over the UI. We added a client-side self-retry. We also made the server refuse to start at all if PORT wasn’t set explicitly. Before, it fell back to a default and then 502’d forever behind the proxy.

The mic button found the dead feature

That same day we shipped RichInput.tsx, a single composer with type, mic, attach, paste and drag-drop. We wired it into every input surface on the app: gate cards, the capture rail, the closure queue, the task chat panel.

Wiring in the mic button is what surfaced the problem. Voice transcription had been failing for weeks. Someone had pulled the OpenAI key out of the server’s service environment to close an earlier issue. The reasoning was sound. The same Windows service also runs a scheduled classification worker, and a key sitting in process.env would get handed to that unattended worker along with everything else the service does. So the key came out, and voice started failing closed. Nobody noticed, because nothing in the app was calling it yet.

The wrong turn: putting the key back in the env

The first fix was the obvious one. The endpoint clearly needed the key, and it was right there in .secrets/openai.env. So we re-added OPENAI_API_KEY to the NSSM service env and restarted the service. Transcription worked.

Then we remembered why it had been pulled. The classification worker runs in the same process and reads the same env, unauthenticated, on a schedule. Re-adding the key fixed voice and silently re-armed a background job that could burn OpenAI spend with no user in the loop and no visibility if it went sideways. We pulled the key back out within the same session, before the scheduled worker’s next tick. Then we started over from a disk read instead of an env var.

Reading the key from disk

server/pwa/voice.ts now reads the key from disk, inside the request handler, and never assigns it to process.env. It mirrors a pattern already in place for speakTts.

That closed the leak path but not the risk. A live-mic endpoint plus a valid key is an open spend spigot with no guardrail. So the rewrite added three things beyond the bare fs.readFile:

• A key gate: the read only succeeds if the caller is authenticated against the same session-key check the rest of the bridge API uses.
• A test-runner refusal: the handler checks for the CI/test environment marker and returns an error instead of hitting the network, so a stray test run can’t spend real money.
• A 1 hour/day audio cap per session, plus a visible spend ledger that the transcription write path appends to on every call, so the number is never a mystery days later.

Upload for boxes with no session

Also that day, POST /api/upload went in for boxes that don’t have a session yet. It saves to <cwd>/.cockpit-uploads/<sid>/<ts>-<safeName>. The filename is stripped of path separators, nulls and control characters. Then it is clamped to 180 characters.

We verified it live before calling it done. Transcription returns 200 with a ledger entry written. Upload returns 403 unauthenticated and 200 authenticated, with the file actually landing on disk. The full suite ran 129 green.

What it cost

Pulling a key to close one hole, the scheduled worker, created a second and invisible one: a dead feature nobody was watching. We caught it only because an unrelated UI pass happened to touch the same wire. So when a credential is pulled, check the features that depended on it. And before it goes back, check whatever else shares that environment.

dxdev LESSON · paste into your AI coding agent
LESSON: Removing A Credential To Close One Hole Can Silently Kill Other Features, So Audit What Depended On It
SOURCE: dxdev.com/blog/silent-feature-death-and-cost-guards
WHAT HAPPENED: A key had been removed from the service environment to stop an unattended classification worker from inheriting it. Voice transcription then broke, but nothing in the app was calling it yet, so no one saw the failure. A UI pass that added a mic button to the shared composer exposed the dead feature. The first fix was to re-add the key to the service env, which worked but re-armed the background worker to spend money with no one watching, so it was pulled again within the same session. The final version reads the key from disk inside the request handler, never puts it in process.env, and adds a session-key gate, a test-runner refusal, a 1 hour per day audio cap per session, and a visible spend ledger.
THE RULE: When you remove a credential to close one exposure, inventory every feature that depended on it and add a check that makes its failure visible. Scope access to the one code path that needs it, with a cap and a ledger, instead of widening the shared environment.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Every secret removed from a shared environment has a recorded list of the features that used it, and each of those features has a test or alert that fails loudly when the secret is missing.
      2. Paid third-party keys are read inside the specific request handler that needs them and are never assigned to process.env, where unrelated scheduled jobs on the same process could inherit them.
      3. Every endpoint that spends money enforces an authentication gate, refuses to run under the test or CI environment, applies a per-session usage cap, and appends each call to a spend ledger someone can read.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.