--- title: "\"Bamboo is broken\" was wrong: a deploy that races the filesystem under CPU pressure" canonical: https://dxdev.com/blog/bamboo-not-broken-deploy-races-filesystem-under-cpu-pressure/ datePublished: 2026-04-23 --- A memory file from an earlier session told me "Bamboo broken, bandaid in place." That one note steered two whole sessions into the wrong fight. The deploy was never broken. It was starving downstream of a bot swarm, losing a filesystem race under CPU load. Getting there took most of an evening and two parallel incidents. ## The symptom The deploy died with a single clean line: ``` Could not find www\default.asp. Aborting. ``` The deploy pipeline I run is Bamboo. A build produces an archive, the deploy stage expands it onto the IIS box, and a verify step confirms the expected files landed before the deploy is marked good. Push to live is about five minutes end to end. The failing check lived in the deploy extract script on the build server. So the deploy expands the archive, then immediately checks that `www\default.asp` exists, and the check says it does not. The file is in the archive. The expand reports success. The very next line says the file is missing. ## The false trail The first session that night did not treat this as a bug to understand. It treated it as a known-broken CI pipeline and applied an emergency bandaid, because the memory said so. "Bamboo broken, bandaid in place" sounds like useful context, but it pointed both sessions in the wrong direction. Once you accept that the pipeline itself is broken, every piece of evidence gets read as confirmation. You stop asking why the file is missing and start asking how to work around Bamboo. The calendar for the day has it in black and white: one session titled "Fix Bamboo build git checkout failures," then a separate one titled "Apply Bamboo CI emergency bandaid." Two sessions, both fighting the tool, neither asking the right question. The right question was: why is a file that the expand step just wrote not visible to the verify step that runs immediately after? ## The actual root cause `Expand-Archive` finishes and returns. The PowerShell that follows checks for the file. On a healthy box, the file is there by the time the check runs, every time, which is exactly why this never reproduced when I tried it by hand later. The write has flushed and the directory entry is visible before the next statement executes. When the box is CPU-starved, that ordering stops being reliable. The expand can return before the filesystem has flushed the new entries into a state the verify step sees, and the `Test-Path` style check loses the race. It is a classic write-then-immediately-read race that is invisible at idle and only surfaces under load. And the box was under load. The same evening, in a completely separate window, I was working a related incident ticket: the site was slow and CPU was pinned. That turned out to be a distributed bot swarm, 55-plus unique IPs each hitting one page once or twice, all clustered into a single hosting ASN (Alibaba), an attack the top-IP-by-volume playbook is blind to. When I finally landed a block that relieved CPU, the production box dropped from pegged to calm in seconds. One Windows Firewall rule against an Alibaba IP range took CPU from 73 percent to 11 percent and active connections from around 1000 down to 128. Here is the part that took the whole evening to see. The deploy failures and the slow site were the same problem showing up in two places. CPU pressure from the swarm was starving the deploy box badly enough to make the extract script lose its flush race. Once the firewall rule relieved the CPU, the deploys ran clean again. No pipeline change required. ## The fix looks like a one-liner The deploy does not need a rebuild, a migration, or a new CI tool. It needs the verify step to stop assuming the filesystem is instantaneous. A short `Start-Sleep` plus a retry loop between the expand and the existence check in the extract script is the entire fix. Wait, re-check, and only abort if the file is still missing after the filesystem has actually had a chance to settle. That converts a load-sensitive race into a deterministic check. Worth saying plainly: that retry loop is the proposed fix, not a shipped one. What actually made the deploys go green again was relieving CPU at the app and firewall layer. The race went away because the box stopped being starved, not because the verify step got hardened. The extract script also lives on the build server outside source control, so it is not something a commit can prove. Treat the retry loop as the right shape that still needs to be written and proven under a fresh swarm, not a closed ticket. While I was in there, I also stopped trusting the deploy's own success flag. The deploy status told me a deploy succeeded that had not put the right file on disk. So the standing rule now is to verify a deploy by grepping the deployed file for a known marker comment that I know is in that release, not by trusting the green checkmark Bamboo paints. The pipeline's report of success is a claim, not a measurement. The hard part was never the fix. It was getting past a confident, stale note long enough to notice the deploy was not broken at all. It was just standing downstream of something on fire. ## Related - [It Looked Exactly Like a Bot Swarm. It Was SQL Server Parameter Sniffing.](sql-server-parameter-sniffing-looked-like-a-bot-swarm): two unrelated symptoms tracing to one shared root cause - [The runtime was half dead: use a migration as the audit you'd never run](the-runtime-was-half-dead-migration-as-audit): discovering what a system is actually doing under load vs. what you assumed - [My red CI was a lie: a deleted workflow haunted every push while nothing real ran](ghost-workflow-zero-second-failure-masked-no-ci-running): stale state in a CI system producing confident-looking but wrong signals - [One Private Dependency, Five Different Failures](five-layer-deploy-break-one-private-dep): a single upstream problem manifesting as multiple unrelated-looking deploy failures - [The rate-limiter that banned our power users: 302 to 200 redirects look like an attack](rate-limiter-banned-power-users-302-200-redirects): a working system misbehaving because of an upstream condition, not a code bug - [The Self-Updating Deploy Script That Has to Fail Once to Fix Itself](self-updating-deploy-script-fails-once-to-fix-itself): deploy pipeline behavior that only makes sense once you understand execution order