--- title: "78 of 12,866 Requests Came Back as IIS 502 Pages" canonical: https://dxdev.com/blog/silent-feature-death-and-cost-guards/ datePublished: 2026-08-23 --- On a phone at 17:44 UTC, 78 of 12,866 requests that day came back as IIS's own 502 page. All of them were hits on a Chrome upload endpoint's transcript replay. The app server was running under `tsx-watch`. So every file save restarted Node mid-request. We moved it to the built production server, `node dist/index.js` with `NODE_ENV=production` on port 3010. We stopped the spine loader from painting the raw upstream error body over the UI. We added a client-side self-retry. We also made the server refuse to start at all if `PORT` wasn't set explicitly. Before, it fell back to a default and then 502'd forever behind the proxy. ## The mic button found the dead feature That same day we shipped `RichInput.tsx`, a single composer with type, mic, attach, paste and drag-drop. We wired it into every input surface on the app: gate cards, the capture rail, the closure queue, the task chat panel. Wiring in the mic button is what surfaced the problem. Voice transcription had been failing for weeks. Someone had pulled the OpenAI key out of the server's service environment to close an earlier issue. The reasoning was sound. The same Windows service also runs a scheduled classification worker, and a key sitting in `process.env` would get handed to that unattended worker along with everything else the service does. So the key came out, and voice started failing closed. Nobody noticed, because nothing in the app was calling it yet. ## The wrong turn: putting the key back in the env The first fix was the obvious one. The endpoint clearly needed the key, and it was right there in `.secrets/openai.env`. So we re-added `OPENAI_API_KEY` to the NSSM service env and restarted the service. Transcription worked. Then we remembered why it had been pulled. The classification worker runs in the same process and reads the same env, unauthenticated, on a schedule. Re-adding the key fixed voice and silently re-armed a background job that could burn OpenAI spend with no user in the loop and no visibility if it went sideways. We pulled the key back out within the same session, before the scheduled worker's next tick. Then we started over from a disk read instead of an env var. ## Reading the key from disk `server/pwa/voice.ts` now reads the key from disk, inside the request handler, and never assigns it to `process.env`. It mirrors a pattern already in place for `speakTts`. That closed the leak path but not the risk. A live-mic endpoint plus a valid key is an open spend spigot with no guardrail. So the rewrite added three things beyond the bare `fs.readFile`: - A key gate: the read only succeeds if the caller is authenticated against the same session-key check the rest of the bridge API uses. - A test-runner refusal: the handler checks for the CI/test environment marker and returns an error instead of hitting the network, so a stray test run can't spend real money. - A 1 hour/day audio cap per session, plus a visible spend ledger that the transcription write path appends to on every call, so the number is never a mystery days later. ## Upload for boxes with no session Also that day, `POST /api/upload` went in for boxes that don't have a session yet. It saves to `/.cockpit-uploads//-`. The filename is stripped of path separators, nulls and control characters. Then it is clamped to 180 characters. We verified it live before calling it done. Transcription returns 200 with a ledger entry written. Upload returns 403 unauthenticated and 200 authenticated, with the file actually landing on disk. The full suite ran 129 green. ## What it cost Pulling a key to close one hole, the scheduled worker, created a second and invisible one: a dead feature nobody was watching. We caught it only because an unrelated UI pass happened to touch the same wire. So when a credential is pulled, check the features that depended on it. And before it goes back, check whatever else shares that environment.