GPTBot vs OAI-SearchBot vs ChatGPT-User: blocking the wrong one deindexes you from AI search

“Does blocking GPTBot keep me out of ChatGPT’s results?” The answer is no, and the reason is the whole point of this post: OpenAI runs three different bots for three different jobs, and the per-bot toggles exist precisely so you can opt out of training without opting out of being found. Treat “AI bots” as one switch and you will either feed the training crawlers you meant to block or deindex yourself from the search surface you meant to keep. Both happen silently. Nobody throws an error when you vanish from an answer engine.

I hit this while putting Cloudflare in front of a 20-year-old Windows/IIS/ASP sports SaaS. The plan was straightforward: turn on bot protection, geo-block the traffic that has zero funnel value, move on. My first move was flipping on Cloudflare’s managed “AI bots” ruleset and treating that as handled, one setting for anything that looked like a scraper. Then I started drilling the actual Security Events and found that ruleset was already blocking real Googlebot traffic on a Pro plan, by default. If I had not gone looking, that block would have sat there quietly until rankings dropped weeks later. It became clear that “block the AI bots” is several separate decisions wearing one label.

Three bots, three jobs, one User-Agent prefix that lies to you

Here is the breakdown that matters, in OpenAI’s case:

• GPTBot crawls your site to train future models. Block it and you opt out of training. It does not affect whether ChatGPT can answer questions about you. People block this one thinking it controls their visibility in ChatGPT. It does not.
• OAI-SearchBot builds the ChatGPT Search index. This is the surface that actually cites you. Block this and you disappear from ChatGPT Search results. If your goal was “stay discoverable in AI search,” blocking OAI-SearchBot is the exact mistake that costs you the thing you were trying to protect.
• ChatGPT-User is the on-demand fetcher. When a user asks ChatGPT a question and the assistant pulls your page live to answer, that is ChatGPT-User. Perplexity does the same thing with Perplexity-User. This is not a crawler walking your site on a schedule. It is a real fetch triggered by a real person asking a real question, and the page it grabs is the page that ends up in the answer.

Three names, three jobs. Opting out of training (GPTBot) is not the same decision as opting out of discovery (OAI-SearchBot), which is not the same decision as opting out of live-answer fetches (ChatGPT-User). If your robots.txt or your WAF rule treats them as one bucket, you are making three decisions with one toggle and you will get at least one of them wrong.

A related trap: do not block by User-Agent string and call it done. “GPTBot” in a UA is trivially forgeable. The thing you actually want to discriminate on is a verified signal, not a string a scraper can type. On Cloudflare that signal is cf.client.bot, which is reverse-DNS and ASN validated, so an attacker forging Googlebot in the UA does not match it. That distinction is what kept the geo-block from deindexing the site, more on that below.

The miss nobody warns you about: link-preview bots

For a content or community site, the most expensive bot you can accidentally block is not a search crawler at all. It is the link-preview fetchers: facebookexternalhit, Twitterbot, Slack, Discord, iMessage. When a coach pastes a team URL into a group chat, those bots fetch the page to render the preview card, the title, the image, the description. That preview is a conversion surface. It is the difference between a link that looks like a real thing worth tapping and a naked URL that looks like spam.

On Cloudflare, Super Bot Fight Mode in Block mode will catch those preview bots unless you carve them out first. Flip SBFM to Block without the carve-out and your social shares stop rendering. The page is up, the link works if you click it, but in the chat it is a dead grey rectangle. Nobody taps a dead grey rectangle.

So the ordering rule is: get the link-preview exemption in place before you flip SBFM to Block. Same logic as the search crawlers. You decide, per bot category, whether this is traffic you want, and you make that decision before you turn on the hammer, not after a customer asks why their team link looks broken on Facebook.

How this connects to the verified-bot rule that saves your SEO

The reason I trust this pattern is I watched the inverse of it nearly happen with Google. While drilling a sampled Cloudflare Security Event, the top blocked source address was in a documented Googlebot range, ASN AS15169 Google LLC, User-Agent Googlebot/2.1. Cloudflare’s own managed “Manage AI bots” ruleset was blocking Googlebot on a Pro plan. Default on. Nobody would have noticed until rankings dropped weeks later.

The fix is counterintuitive and it generalizes to every bot decision in this post: you do not loosen the managed ruleset, you add a higher-priority Skip rule. Expression (cf.client.bot), action Skip, covering all managed rules and all Super Bot Fight Mode rules. Because cf.client.bot is the cryptographically verified good-bot signal, this passes the real Googlebot, OAI-SearchBot, and friends while still letting you block the spoofers. After deploying it I watched live events confirm three Google crawler addresses, all in Google ranges, flip from Block to Skip within about six minutes.

Same thing on the geo-block. The site serves US and Canada youth-sports leagues, so blocking the rest was structurally correct. But the naive rule, (not ip.geoip.country in {"US" "CA"}), kills Googlebot crawling from a foreign datacenter. The load-bearing fix was one extra clause: (not ip.geoip.country in {"US" "CA"}) and (not cf.client.bot). Without the and (not cf.client.bot), you silently deindex yourself from search the moment a crawler hits you from the wrong country. With it, the humans you do not serve get blocked and the bots you depend on sail through.

That is the same shape as the OpenAI bot question. Geo-block is one switch that, applied naively, makes two decisions: block foreign humans (intended) and block foreign-datacenter crawlers (catastrophic). You split them with an explicit verified-bot carve-out. “AI bots” is the same: one label, several decisions, and you split them deliberately.

The developer takeaway

“AI bots” is not one switch. Stop reaching for a single toggle.

Decide separately, per bot, what you actually want:

• Opt out of training by blocking GPTBot. That and only that.
• Stay in discovery by leaving OAI-SearchBot alone. Block it and you are gone from ChatGPT Search.
• Decide on live-answer fetches (ChatGPT-User, Perplexity-User) on their own merits. These are users asking about you in real time.
• Discriminate on a verified signal (cf.client.bot), never on a User-Agent string a scraper can forge.

And do not let bot protection eat your social link-previews. facebookexternalhit, Twitterbot, Slack, Discord, and iMessage are not crawlers consuming your bandwidth, they are the thing that makes a shared link look worth tapping. Carve them out before you flip SBFM to Block, not after. Those previews are conversions. Treat them like it.

Related

• Your CDN’s default bot protection is blocking Googlebot and quietly bleeding your SEO: the Skip rule that protects all verified crawlers from managed ruleset catches
• Geo-blocking everywhere except the US and Canada, and the one clause that saves your SEO: the geo rule that needs the same verified-bot carve-out to avoid deindexing
• The UA blocklist is the right-sized fix for an identified crawler (and the wrong one for a swarm): when a UA-level block is correct and when it is not
• Cloudflare Pro is a box of levers, not a managed service: understanding what the managed AI bots ruleset actually does and how to override it
• JS Detections: the one Pro-tier Cloudflare lever that actually catches HTTP-replay proxies: the next layer of bot defense after the per-bot categorical decisions are set

dxdev LESSON, paste into your AI coding agent
LESSON: One AI bot label can hide three different jobs
SOURCE: dxdev.com/blog/gptbot-vs-oai-searchbot-blocking-wrong-ai-crawler
WHAT HAPPENED: A single managed ruleset toggle labeled "AI bots" was flipped on and treated as one decision covering all AI crawler traffic. In fact the vendor runs separate bots for training ingestion and for live search-answer fetches, each with its own opt-in/opt-out meaning. The single toggle silently blocked a live search crawler alongside the training crawler it was meant to stop, and nothing errored; the site just began losing search-surface visibility. Reading the actual traffic events, not the dashboard's green summary, is what surfaced the specific bot being blocked.
THE RULE: Before you block "AI bots" as one category, confirm how many distinct bot roles the label actually covers and whether blocking one silently blocks a role you meant to keep.
CHECK MY CODE, then report PASS / FAIL with file:line for each:
  1. Any firewall, WAF, or bot-management rule that groups multiple distinct crawler user-agents or roles under one enable/disable toggle without per-role control.
  2. Any bot-blocking change that was verified only by a dashboard summary or rule-count, not by inspecting individual blocked-request events after deployment.
  3. Any documentation or comment describing a vendor's bot category as a single purpose when the vendor's own docs list multiple distinct bot roles under that name.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.