Theme
Silent failures
Why green dashboards lie, and how to build checks that catch what is actually broken.
109 posts by DX, in two lanes: plain-language stories and the technical build log.
Everyday lane
Plain-language stories. No jargon. 24 posts.
Start here
One Change Overnight Locked Every Customer Out. Nobody's Phone Rang.
Every customer's own web address stopped working for eleven hours. We had a monitor. It was checking the front gate while the door customers use was locked. Six questions to ask whoever runs your website.
Alerts that went quiet or cried wolf
-
One Change Overnight Locked Every Customer Out. Nobody's Phone Rang.
Every customer's own web address stopped working for eleven hours. We had a monitor. It was checking the front gate while the door customers use was locked. Six questions to ask whoever runs your website.
-
A Running Monitor and 25 Domains It Never Looked At
A six-hour migration moved 25 brand domains with no downtime. The bigger find was a health monitor that had been checking the wrong addresses the whole time, unrelated to the move itself.
-
Why a customer's account showed expired weeks after the charge went through
A customer paid on time and their account still showed expired. The system had no error to point to, because the message telling it a payment happened had never arrived at all.
-
Why Our Balance Check Filed a Real Charge Under the Wrong Day
A guard built to warn us the moment free usage ended stayed silent through two real charges, for two different reasons.
Certificates, routing and misleading verification
-
What a Green Status on a Domain Migration Actually Measured
A migration was marked done and trusted for four days. The tool that verified it was asking the same system it was supposed to be checking, so it could only ever confirm what that system already believed.
-
Why a Pending Domain Showed No Error for Thirteen Hours
A migrated domain sat unfinished for thirteen hours with an empty error box. The record we had already checked off was the thing that had gone stale.
-
The Alarm That Paged 55 Times on Its First Morning
A watchdog built to catch a real problem judged everything by one signal. Forty-four of its 55 pages were nothing. The other eleven, wearing the identical label, were a real emergency.
Backups and jobs that quietly did nothing
-
A folder rename that turned up one saved copy labeled initial
A routine folder cleanup revealed that a weekly backup had stopped long before anyone noticed.
-
The Timer Fired on Schedule and the Newest Copy Predated Our Changes
A repository move uncovered scheduled work that looked active but was no longer producing a recent backup.
-
Line 10 and Line 2160 of the Same File
A tiny setting made around 60 kinds of requests, most of them edits, look successful even though nothing was saved.
-
Why a healthy-looking diagnostic page kept printing undefined
A small page looked healthy but gave people the wrong answer because a copied shortcut had lost a safety step.
-
Two to three minutes of 404 after removing a setting we were sure was dead
A website check looked reassuring until removing an old setting revealed that it was answering the whole time.
-
What the Tracking Screen Showed for 205 Customer-Managed Domains
A nightly check kept hitting its one-hour limit and stopping without a word. The screen showed results for the domains it reached, which looked exactly like full coverage of all 205.
Reports and dashboards that mislead
-
Draft, Reviewed, Confirmed Were Three Different Columns
When work happens in parallel, a larger activity number can look impressive without showing whether the work was useful, reviewed, or ready for the people affected by it.
-
Nothing Improved Between a Report of 46 and a Report of 33
A recurring-failure count dropped by thirteen between two runs. Nothing in the underlying work had actually changed. The counter had been wrong the whole time.
Shipped checks that never actually ran
-
Three Blank Fields and a Tool Nobody Thought to Check
A tool can be well built, pass its demo, and still never do its job on a single real case afterward. The only proof is checking a recent one.
-
Why a 120 Pixel Minimum Height Was Only Half the Fix
A working editing box can still be a bad experience when it gives people no sign that it exists.
-
We Removed a Safety Setting on Purpose and the Check Stayed Green
We built a check to make sure a safety setting was always turned on. The first version passed even after we removed the setting, because a comment nearby still mentioned it.
-
We Tested an Unreachable Recipient and the Send Still Said Success
A working login proves the door opens. It does not prove anyone is on the other side to answer when something urgent comes through.
-
Three crash causes behind one alert that nobody read anymore
The same crash alert kept firing, over and over, until it stopped meaning anything. The causes underneath it turned out to be three ordinary things wearing different disguises.
Tests and tasks that silently never ran
Metrics that are real but misleading
Tuning alarms to cut false positives
More in this theme
Technical lane
The build log: the logs, the code, the fix. 85 posts.
Start here
Every Customer Domain Was Down for Eleven Hours and the Monitor Stayed Green
One IIS binding carried every customer domain. An AI coding agent removed it overnight while setting up something else. Our monitor was in place and never fired, and the reasons stack up layer by layer.
Alerts that went quiet or cried wolf
-
Every Customer Domain Was Down for Eleven Hours and the Monitor Stayed Green
One IIS binding carried every customer domain. An AI coding agent removed it overnight while setting up something else. Our monitor was in place and never fired, and the reasons stack up layer by layer.
-
A 458 MB Droplet, a SIGKILL, and 5.9 Seconds of Refused Connections
The status alert said the relay had been down for 22 minutes, and the relay was a box we thought we understood.
-
The Discord Alert Key That Swallowed a Real 28-Credit Charge
The manus-watch commit says it missed 152 real credits. The commit before it says a shared alert key ate a real 28-credit charge.
-
Why the 17h 29m Session Made Our Daily Review Card Wrong
We'd already built our internal ops-tracking system's Discord integration: daily, weekly, and monthly review cards, auto-posted on a schedule, pulled from the same session logs that drive the work ...
-
We Diffed Every Row and the Bug Was in the Script Order
The alert had gone quiet, which was the problem. A rollover that hits a duplicate-key error is supposed to fail loud and then heal itself on retry.
-
Silence from a renewal job with no error line, no failed run, and 520 correct skips
A customer's site had gone unrenewed-reminded for nearly seven years, despite renewing several times since. The cause traced back to one commented-out check from 2019.
-
Our Monitor Was Right About the Condition and Wrong About Everything Else
A monitor built to catch our own outages paged a customer's dead third-party server as ours, six times in nine days, and we defended it before we were shown wrong.
-
Why off-edge went quiet while stalled kept paging
Fifty-five Discord DMs were sitting in the channel by 7 AM, one bot message per line, all from the same script: the Cloudflare cert watcher that scans every domain on the platform and flags anythin...
-
Leftover Browser Tab Sent Duplicate Emails
One task-bot email arrived three times on October 1, and the lock we built to stop duplicates had done nothing.
Certificates, routing and misleading verification
-
Why Our Cloudflare Verify Tool Never Queried Public DNS for www
A customer told us their site was down at 3:57 PM on June 12th, four days after we'd migrated their domain to Cloudflare and marked it done.
-
Cloudflare Error 1014: The Cert Issued. The Page Didn't Load.
One proxied=False to proxied=True flip closed the ticket, but it took two wrong root causes and a live domain to find the flag, and it rewrote a design doc that had been wrong for months.
-
Four of 25 domains stalled in pending_validation with no error to read
A migrated domain sat in pending validation with an empty error field and a correctly published DNS record. We blamed the certificate authority. The token we were checking against had been rotated.
-
We Deleted One IIS Binding and www.customer.example.com Went to 404
A CDN cutover returned a clean 200, so we called it verified. Then we deleted what we thought was dead weight and the page dropped to 404. The 200 had been the interim binding answering the whole time.
-
Why 44 Stuck Certificates Were Fine and 11 Were an Outage
55 certificates were stuck, and every alert line read ssl status 'pendingvalidation' with a live cert.
-
The Only Record of March 27 Was a Git Log
A certificate-renewal failure became a six-release incident as bot traffic and slow feedback compounded the original problem. A retrospective on why observability belongs in an incident response loop.
-
A second #Fields line appeared mid-file with 15 columns instead of 14
Orange-cloud a legacy IIS app and every IP your origin sees becomes a Cloudflare edge IP, with no errors anywhere. How to find and fix the lying logs.
-
Why TXT Validation Left 956 Customer Hostnames Waiting on Someone Else's DNS
A bulk migration had put 956 domains on the one renewal method that needs a DNS record from the customer every 90 days. Moving them was easy. The watchdog we built for the next one spent its first real morning raising an alarm over the wrong 55.
-
What a repair script cannot see in a 90-day certificate renewal cycle
Nearly a thousand certificates sat on a validation method that needed refreshing every 90 days, and nothing would have noticed a specific one about to fail before it actually did.
-
What an August 7 Cutover Audit Turned Up in Our Cloudflare Registrations
At 8:36 on July 20, we went in to answer whether five client-managed domains would break at an August 7 cutover. Two would. Three would not.
-
Why a 200 OK Killed Our Certificate Root Theory
Our support lead spent hours on the phone with a customer because our own screen was telling her he was wrong.
-
Why a green drift check on v27 hid a crawler bypass at v28
The drift check ran on 2026-08-31 and said live 31 vs doc 27, four behind. It was correct. It was also the only thing it looked at.
-
Ten Custom Hostnames Pending, Eight Active, and One Stub Worker
The admin console had a clean button. Click it, and each domain got queued for Cloudflare custom-hostname registration.
-
The 24-Hour Cache That Made Working Domains Look Broken
A customer's domain resolved correctly, served over HTTPS, and our staff Detail page still said "Awaiting DNS".
Backups and jobs that quietly did nothing
-
The Weekly Restic Backup Had Only One Snapshot in S3
We split a sprawling AI ops monorepo into five sibling repos. The rename exposed runtime automation that had been dead for weeks.
-
The Latest Backup Artifact Was Older Than Anyone Assumed
A repository move exposed external automation and a backup job that no longer produced the expected artifacts. The lesson is to treat any substrate migration as an inventory-and-verification exercise, not a search-and-replace task.
-
The legacy server-variable bug that printed `undefined` instead of a value
A diagnostic page returned the literal word `undefined` instead of a client value. Copies of shared logic had drifted away from the defensive helper.
-
A 6:33 AM Tailscale DNS Error and a Missing .gpg File
At 6:33 AM, the daily sweep logged one line and moved on: archive drop failed, a remote host unreachable over Tailscale, DNS resolution failing. The obvious read was that a PC was off.
-
The weekly undeployed change warning that sent us into the wrong records
We chased the weekly undeployed change warning for weeks before we found out it was a false positive. One service was absent from the checker's own sidecar list.
-
A Loopback Call to localhost:3000 That the IIS Proxy Never Saw
We found the failure during a production messaging-worker cutover, before we trusted SMS as the way an autonomous job talks back to a human.
-
Counting Scan Results Against 205 Expected Domains Told a Different Story
At 8:56 AM, the migration console was telling a partial truth about 205 client-managed domains.
-
Why Our Clean Review Missed Eleven Defects in the Invoice Loop
The invitation was supposed to be on its way, but it had never sent for any client. The billing screen could not tell anyone whether a payment had landed.
-
One Alert Said ERROR Six Times While Both Outcome Lines Read 100%
The alert landed at 12:53 on a Saturday, and it said ERROR six times.
-
A Gmail 550 5.7.25 Buried in the Mail Server Log
Between 10 and 20 percent of our signup codes and password resets to Yahoo, AOL and Gmail had been hard-rejected for about ten days, and nothing in our code had changed in that time.
-
Steam Installs Filled 452GB of the Drive Called Backups
The backup dashboard on our main desktop has said "up to date" for something like two years straight.
Monitors and gates that fail silently
-
The Publish Log Said 554 to 731 and Nothing Else Was Wrong
The publish log for a 23 hour 53 minute run showed zero errors, and the live post count went from 554 to 731.
-
What GHSA-mwp4-54f8-5fhr kept doing to a client's main branch every morning
We opened gh run list on a client's API repo expecting one failure. We got a pattern. dependency-audit failed on 2026-08-04 at 09:24 UTC, on schedule, on main.
-
Our Alerts Watched for Failed Builds While the Plan Quietly Stopped Running
The alert never fired. What fired instead was a teammate asking twice, five days apart, whether his preview build had actually gone out.
-
One Malformed Seed, 43 Missing Posts, and a Misleading ENOENT
Five nights running, the nightly blog drafter produced zero posts. Five nights running, the only signal we had was a CI triage bot repeating the same line every 45 minutes: publishPost returned ok:...
-
One Regex Boundary Held 43 Blog Posts Back
At 6:58 AM on the sixth day, we found the nightly blog drafter had been dead since the previous week.
-
The Second Claude Account Was Missing Three Tools Without an Error
Automatic failover moved crons off a dry Claude seat onto a fresh one. A drill found it only reached nine of fourteen.
-
The Weekly Limit Message That Arrived on Standard Output
A Claude seat ran out of weekly quota, and our ticket bot kept calling it for hours. Nothing in its logs read as a quota failure, because as far as the bot could tell, no failure had happened.
-
The Idle Account Dropped Out of Routing While Another Burned at 69%
Our router couldn't see one of our two paid Max 20x accounts for 809 minutes, about 13.5 hours, and none of its default output said so.
-
Four monitors watched the admin and none of them saw error 0174
An admin page rendered a fatal error inside a normal-looking response for five days. The uptime watcher and the weekly crawl missed it, and the error log had no row for it.
-
Why a scheduled codex exec job found no credentials as SYSTEM
Twenty-six rewritten posts were staged, the stuck promote branch was merged, and nothing published. The run had gone a full pass with no error in the output that anyone read.
-
A 6:46 AM Alert and a Blank Brief That Never Existed
The alert fired at 6:46 AM. By 8:03 AM we were still looking at a blank brief that had refused to render, and the cron that generated it had already exited zero an hour earlier like nothing was wrong.
-
Our red CI was a lie: a deleted workflow haunted every push while nothing real ran
Every push to our internal API repo showed a red failure, so we assumed CI was running and just unhappy with our code. That assumption was wrong twice over.
Reports and dashboards that mislead
-
The Review Record Missing From a Chronological Activity Log
Elapsed time, concurrent process time, human attention, and verified outcomes are different measures. Once work happens in parallel, a useful record must preserve provenance, uncertainty, review state, and the limits of what any productivity metric can claim.
-
calendar_day Raised an Exception Every Night and Nobody Heard
We built a health dashboard so cron jobs could not rot in silence. The cron job rotted in silence anyway, because the dashboard was watching the wrong thing.
-
The Run That Finally Exited 4 After Every Run Exited 0
Exit code 4 came back on the first run where the meter was actually allowed to say no.
-
Roughly 21 percent of the approvals were guessed
Forty-three: that was the number that kicked off the post-mortem. One day, we'd closed forty-three agent windows manually, one at a time, no batch action, no shortcut.
-
One Command Could Have Rewritten Ten Retired 2020 Tickets
A daily alert flagged 17 tickets as inconsistent. Sixteen were false alarms, and the fix for them was about to run.
-
847 of 1,586 Claude sidecars wrote rows and nothing printed an error
Building a memory layer on a year of agent conversation sidecars: the enricher, the theme aggregator that exposed a 14 percent slice, the nightly schedule, and what shipped unfinished.
-
Why a Fifteen Minute Idle Sleep Swallowed Our One-Off Refresh Request
A status check came back two and a half days stale, because the refresh gate only recognized a page someone was still watching.
-
The 38 Minute Bug Hiding Behind an active_watchers Loop
The session tree had been showing yesterday afternoon's activity since 11:03 PM, when we opened it planning to check on a quick fix.
-
51.8K to 14.9K and the Markdown Still Rendered Fine
A refactor silently dropped a class of records while the rendered file still looked plausible. A large diff against a known-good committed fixture exposed the regression, then semantic checks explained it.
Errors that hide inside working interfaces
-
What Connecting Session Recordings Turned Up in One 2h Late-Night Session
The session-replay connection took one late session, 11:27 PM to about 1:27 AM, and by the end of it a popup crash fix was live in production. Nothing in our error logs had pointed at that crash.
-
Release 3.358 and the Missing Middle of Our Funnel
signup, begincheckout, and purchase were the only 3 events in a customer journey with 8 decisions that mattered.
-
The onboarding event that fired on every page load for 10 days
On July 22, we found an onboarding event that had fired on every page load since July 12. For 10 days, our signup funnel had been counting navigation as progress.
-
Twelve Days After Client-Side Reporting Shipped, the Chart Jumped
An admin error log sat over its limit for most of a day. The count was real. The story it seemed to tell wasn't.
-
The Error Log Showed 32 Rows and We Chased the Wrong Thing
The error log printed 32 rows in one hour. We chased it.
-
The Button Clicked and Nothing Happened: IIS Ate a JSON Error and Handed Back HTML
Our backend returned a 400 with a JSON error body. IIS replaced it with HTML, the client swallowed the parse error, and the button did nothing.
-
A 2016 clipboard.js Build Stopped Working on Some Browsers
A copy button broke with no deploy because clipboard.js and the browser both claimed window.Clipboard. A three-level fallback fixed it without a version bump.
-
78 of 12,866 Requests Came Back as IIS 502 Pages
78 of 12,866 requests that day came back as IIS's own 502 page, on a Chrome upload endpoint's transcript replay.
-
What 39 Hours of Expired-Session Errors Turned Out to Be
The browser had been dead for 39 hours, and it was still asking our staff toolbar who it was, over and over, with a login that had expired long before.
-
The Day We Taught Failures to Speak, Everything Started Shouting
A customer reported a dead Save button on Locations about two hours after we shipped a fix for a dead Save button on another admin page. So we checked whether our fix was involved.
Shipped checks that never actually ran
-
Why item_close.py Never Reached the Code That Set the Sprint
We got a screenshot with one line under it: tickets keep skipping the current sprint. Also the developer field is empty. Also, if there's code involved, the fix version isn't there either. Not once.
-
The Media Query Space That Made 102 Rules Vanish
A compile script said a release had dropped 117 style rules. The audit built to verify that invented its own, different missing count before either alarm cleared.
-
Why send_as().ok Returned True After We Killed the Assistant
An alert system checked whether a token worked before deciding who should send a message. That check passed even when the thing actually supposed to read and act on the message was not running at all.
-
A Slash Command in a Headless claude -p Seed
The alert sat in a shared alerts channel for over an hour: our partner asked a question, and nothing answered back. Not an error, not a retry, just silence where a response should have been.
Tests and tasks that silently never ran
-
The Test That Passed 497 Seconds of Runs Without Ever Touching the Guard
The full test suite takes 497 seconds. For as long as anyone had written it down, the number in our notes was 26 minutes, and nobody had ever timed it.
-
check:rls exits 1 after every listed check passes
The alert fired three times in a row: check:rls exiting 1 after every listed check printed pass.
-
The Commit That Wasn't: When autocommit=False Rolls Back Silently
On July 22, every production mutation sent through our internal SQL tool's --write path rolled back when the connection closed. The command executed its SQL and returned without an exception.
-
A 530 from the archive MCP that pointed at the wrong component
At 10:59 PM, our archive MCP was returning Cloudflare 530 instead of serving requests.
-
Seven Health Checks, One Hourly Gate, and an Exit Code 3 on Windows
A teammate's agent sat broken for two days, and the way it finally got noticed was someone squinting at Discord reactions and thinking they looked wrong. Then it spammed the channel.
-
Thirteen Scheduled Tasks Could Not Report a Failure
An audit of our scheduled-task wrappers found thirteen that exited 0 no matter what the Python underneath returned. Task Scheduler could never see a real failure, including a failed backup.
-
Grepping 4,446 Transcripts Turned Up a PATH Entry and a charmap Crash
A silent PATH misconfiguration meant our token-optimization tool had actually run 10 times across 4,446 AI agent sessions. Grepping our own transcripts like a flaky test suite found it, and two other live bugs.
Metrics that are real but misleading
-
The Dashboard Count of 352 and the Ticket It Wrongly Started
A dashboard number read as 'happening right now.' It was a lifetime total going back more than two years.
-
Making Dashboard Metrics Honest
On the 23rd, the dashboard was correct, and it was still misleading us.
-
Same SHAs in Two Repos: Why Commit-Count Metrics Lie in a Clone Workflow
A commit-count script treated mirrors and local clones as separate work, turning 23 unique commits into an inflated total of 134. The fix was to aggregate unique commit IDs, not repository directories.
-
Our Pre-Deploy Name Check Only Read Text, and 13 Live Images Carried the Name
A text-only grep and a text-only scanner both passed while a real name sat in the EXIF data of 13 images on a pseudonymous site. An independent check found it.
Tuning alarms to cut false positives
-
Three Hours and 11 Minutes Chasing False-Down Pages on the Status Board
At 9:58 AM on July 22, our status board was still raising false-down pages when one monitoring signal disagreed with another.
-
Why a 2500 ms Threshold Still Wasn't Enough to Stop the Pages
At 4:59 PM, a Cloudflare edge alert marked a site degraded because one probe came back 53 ms over the slow threshold. The site was up the whole time. The next cycle was clean.
Alerts and instructions that never land
-
Three model-driven writers retired, one append-only ledger, one unregistered hook
A hook logged work whenever a ticket key showed up in text. A second reviewer caught the flaw. It went live a day later, with receipts.
-
The Instruction That Reached the Model on Zero Prompts
The footer reached the model on 0 of 278 prompts.
More in this theme
Hitting one of these walls in your own codebase or your own machine? Start an assessment or talk to us.