Letting AI Read Your Site Takes Four Layers
I keep coming back to how truthful the green monitor was. It reported correctly on the one redirect it checked, while the customers were on the next hop, which it never made. A green check tells you about the request it sent and nothing else.
Letting AI Read Your Site Takes Four Layers
Our public pages blocked GPTBot twice, with a 403 from the edge and a robots.txt line of ‘Disallow: /’, while answer bots like OAI-SearchBot and Claude-SearchBot got a 200. Four layers each had a say: robots.txt, Cloudflare, an IIS rewrite rule that still listed CCBot, and the app’s own bot handling. The fix on 10/2/26 deleted the gptbot, claudebot and ccbot groups so they inherit the ’*’ rules, removed CCBot from the origin rule, split answer bots from training crawlers in the app, and set Cloudflare’s AI Training policy to Allow while the heavy stats pages stayed closed. If you open one layer and not the others, nothing visible changes, because the crawler is rejected before it ever reads your file.
Every Customer Domain Was Down for Eleven Hours and the Monitor Stayed Green
At 9:01 pm on 10/5/26 one IIS catch-all HTTPS binding vanished, and every customer custom domain returned a bare 404 until 8:11 the next morning. An AI coding agent setting up a new internal site removed it with a command that named only the site and protocol, though the exact cmdlet is inferred from timing and the transcript. The monitor probed only the apex domain, answered by a healthy relay, and the health page read healthy for any finished run, so nothing paged. Our monitor now probes www through the CDN, the health page requires a passing run, and changes go through a wrapper that restores on drift, but the inbox alert for a spike in customer emails is not built yet.
A Staging Test Passed Two of Three Checks, and Both Passes Were False
On 9/23/26 a staging fix for duplicate customer emails broke a staff page, and the staging smoke test then reported two of three checks as PASS. Both passes were false, because staging and production share the live database and the dbo.monitor rows were written by production, still running the old code. The lock failed because BeginTrans over our ODBC bridge had not raised the transaction count, giving return code -999 that the error message mislabeled as a timeout, so the fix became a literal BEGIN TRANSACTION in the same SQL batch as the lock call. The record does not name who wrote the retest rows, so for any staging result you trust, find which process wrote each row you counted as evidence.
If you have a monitor or smoke test that checks only one hop or one table, reply and tell me what it actually proves.
Not sure where AI would actually help in your business? Our AI workflow readiness assessment is the place to start.
Want the plain-English version for running a business? Subscribe to Everyday AI.
Ben Downey