Letting AI Read Your Site Takes Four Layers

GPTBot was blocked twice. Our edge returned a 403 “Attention Required!” page, and our robots.txt said Disallow: /. OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and Googlebot all got a 200.

A teammate had raised an AI SEO concern about our public pages. The ticket for it ended up touching four layers: robots.txt, Cloudflare, the origin server and the app itself.

Starting from the 403

The robots.txt side was easy to find. A ticket from late May had added a section labelled “aggressive SEO and AI-training crawlers blocked outright”. It included gptbot, claudebot and ccbot, each with Disallow: /.

The 403 was less obvious. GPTBot, ClaudeBot, PerplexityBot and Perplexity-User were blocked at Cloudflare, but the answer-bot agents were not. The first job was to find which mechanism was doing it. The candidates were the AI-bot setting, a WAF custom rule, a managed rule, or a UA list. We used read-only API calls only, because changes on the production zone need a human sign-off.

We read the origin too. The IIS rewrite rule Block-Known-Heavy-Bots returns a custom 403 (“Crawler blocked”) for a list of user agents: SemrushBot, DotBot, MJ12bot, PetalBot, Bytespider, TikTokSpider, YandexBot, and so on. A comment in it says GPTBot is intentionally not listed, but CCBot is. So an edge allow alone would still have ended in a 403 for that crawler, one layer further in.

The robots.txt edit

The first plan was one named group for the AI crawlers, with Allow lines for the public marketing and help pages and Disallow: / for everything else. That plan has a trap. RFC 9309 says a named group replaces the * group instead of adding to it, so the new group would have to restate every exclusion the * group carries. The same file already had a comment about this from an earlier bug. bingbot had a group containing only Crawl-delay: 10, so nothing restricted it, and it crawled /admin/ and every scoresheet popup until someone repeated the restrictions inside its group.

The change that shipped is simpler. The commit deletes the gptbot, claudebot and ccbot groups, so those crawlers fall back to the * group and inherit its exclusions. In the same commit, CCBot came out of the origin rule.

Keeping them off the heavy pages

Opening robots.txt does not mean every crawler should get every page. The schedule, scores and stats modules are expensive to render. The same commit made the app flag the AI crawler user agents as bots, so cache writes and those three modules stay off their requests.

Ten minutes later, a second commit split the AI crawlers in two:

• Answer bots (ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Perplexity-User) get full content, the way Googlebot and Bingbot do. A person triggered the fetch, or an AI search index is reading the page.
• Training crawlers stay suppressed and get a light page.

The answer bots are matched by an explicit list of tokens. They are never matched by removing a token from the bot list, because Claude-SearchBot’s user agent also contains the training crawler’s token.

At 3:00 PM a last commit gave Applebot the same full content as Googlebot and Bingbot.

The Cloudflare side

The edge changes were two. A Cloudflare custom rule got trimmed, and the heavy stats pages stay behind the block. The AI Training policy, the setting for crawlers that collect data to train models, was set to Allow.

The layers stack. robots.txt only asks well-behaved crawlers to stay out, the origin rule and the Cloudflare layers decide whether a request is served at all, and the app decides what a served bot gets. If the edge says no, robots.txt can say whatever it likes.

Checking it live

The change shipped as a hotfix. After deploy we checked it live, crawler type by crawler type, against the intent:

• Training crawlers get the light page.
• Answer bots and Apple’s crawler get the full schedules.
• Scrapers like Bytespider are still blocked.

The release note we published says AI search bots can read site content the way Google and Bing do, crawlers that only scrape for training data are still blocked, and so are bots hitting the heaviest stats pages, so the extra traffic won’t slow the site down. That last clause matters most. We opened the gate on purpose for the agents that send people to us, and we kept the heavy endpoints closed.

Every layer gets a vote

The robots.txt change alone would have changed nothing visible, because the edge was rejecting those user agents before they ever read the file. Fixing only the edge would have left Disallow: / in the file, and the origin rule would still have returned a 403 to CCBot.

The ticket was still open at 4:50 PM that day. The reply to the teammate who raised the concern was approved the next morning.

dxdev LESSON · paste into your AI coding agent
LESSON: Read Every Layer That Can Veto A Request Before Changing Any Of Them
SOURCE: dxdev.com/blog/ai-crawler-rules-coordination
WHAT HAPPENED: The block on AI crawlers lived in several places at once. robots.txt carried Disallow: / for gptbot, claudebot and ccbot. Cloudflare blocked GPTBot, ClaudeBot, PerplexityBot and Perplexity-User. The IIS origin rule Block-Known-Heavy-Bots returned its own 403 for a list of user agents including CCBot. Editing robots.txt alone would have changed nothing visible, because the edge rejected those agents before they read the file. The author read all the layers first with read-only calls, then removed the three robots.txt groups, dropped CCBot from the origin rule, trimmed a Cloudflare custom rule, set the AI Training policy to Allow, and taught the app to keep heavy pages off training crawlers while answer bots get full content. After deploy they checked each crawler type live against the intended behavior.
THE RULE: When one access decision is enforced in several stacked layers, inventory every layer before editing any of them, and confirm the combined result by testing each class of client live. A change to one layer can be invisible if another layer still says no.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. List every layer that can accept or reject a request for a given client (robots.txt, CDN or WAF rules, origin rewrite rules, app-level bot checks) and record what each one currently does for each user agent you care about.
      2. After any access-policy change, request the site as each class of crawler (training, answer bot, scraper) and compare the actual status codes and content to the stated intent.
      3. When a robots.txt group is created for a named agent, confirm it repeats every exclusion from the wildcard group, because a named group replaces the wildcard group instead of adding to it.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.