Block Training Bots if You Want — Don’t Accidentally Block Being Cited

You can block training crawlers if your policy says so — but do not treat every AI user-agent as “ChatGPT.” OpenAI’s GPTBot (training), OAI-SearchBot (ChatGPT search / citation index), and ChatGPT-User (user-initiated fetch) are different agents with different consequences. Blocking the wrong one removes you from being cited while you congratulate yourself for “opting out of AI.”

This spoke is the robots.txt decision layer of the Answer Engine Optimization playbook. Cross-check fetchability in the AEO audit checklist.

The short answer

• GPTBot ≠ ChatGPT search. Blocking GPTBot opts out of OpenAI training use per OpenAI’s docs; it does not, by itself, opt you out of ChatGPT search surfacing.
• OAI-SearchBot is the search crawler. Sites opted out “will not be shown in ChatGPT search answers” (OpenAI), though they may still appear as navigational links.
• ChatGPT-User is a user-triggered fetch, not automatic crawl; OpenAI says robots.txt rules may not apply, and it is not the Search opt-out control.
• Google-Extended is a product token for Gemini training/grounding uses — not the crawler that controls Google Search indexing.
• Default for most brands that want citations: allow search/retrieval bots; decide training bots as a policy call.

Should I block GPTBot in robots.txt?

Only if you intend to opt out of OpenAI foundation-model training collection — not because you “don’t want to show up in ChatGPT.”

Per OpenAI’s crawler overview (developers.openai.com/api/docs/bots, verified 2026-08-07):

User agent	Job	If you Disallow
GPTBot	Crawl content that may be used to train OpenAI generative AI foundation models	Signals content should not be used in that training
OAI-SearchBot	Surface sites in ChatGPT search features	Not shown in ChatGPT search answers (may still appear as navigational links)
ChatGPT-User	User actions in ChatGPT / Custom GPTs fetch a page	Live user-directed fetches may fail; OpenAI notes robots.txt may not apply; not the Search control
OAI-AdsBot	Validate landing pages submitted as ChatGPT ads	Only visits ad landing pages; not used for foundation-model training

OpenAI states these settings are independent. You can allow OAI-SearchBot while disallowing GPTBot.

Which AI bots are training vs search/retrieval?

Use this operator table. Confirm against each vendor’s current docs before you ship — names and scopes change.

Vendor	Training-oriented	Search / index oriented	User-initiated fetch
OpenAI	GPTBot	OAI-SearchBot	ChatGPT-User (robots.txt may not apply)
Anthropic	ClaudeBot	Claude-SearchBot	Claude-User (Anthropic states robots.txt is honored)
Google	Google-Extended token (Gemini training & grounding uses)	Googlebot for Search (separate)	N/A as a single “user bot” in the same sense
Perplexity	Check current Perplexity bot docs for training vs answer crawl	Answer crawl / citation bots per their docs	User fetch agents may differ

Wrong mental model: “AI bot = one switch.” Right mental model: training policy vs citation eligibility vs live fetch.

Can blocking crawlers remove me from ChatGPT or Perplexity answers?

ChatGPT search: Yes — if you block OAI-SearchBot. OpenAI is explicit that opted-out sites are not shown in ChatGPT search answers. Blocking GPTBot alone is the wrong lever for that outcome.

ChatGPT user fetches: Blocking ChatGPT-User may interfere with on-demand reads, but OpenAI says robots.txt may not apply to those user-initiated actions, and that agent is not used to decide Search inclusion. Control Search with OAI-SearchBot.

Perplexity / others: Blocking that vendor’s search/answer crawler (whatever their current user-agent is) can remove you from retrieval. Blocking a training-only agent does not automatically equal “invisible in answers.” Verify the agent name in their docs — do not copy a 2024 gist blindly.

Goal	OpenAI control
Stay in ChatGPT search answers	Allow OAI-SearchBot
Opt out of training	Disallow GPTBot
Manage live user fetches	Understand ChatGPT-User limits; do not use it as Search opt-out

What about ClaudeBot and Google-Extended?

ClaudeBot (Anthropic): Training-oriented collection. Anthropic documents separate agents for search (Claude-SearchBot) and user fetches (Claude-User). Blocking ClaudeBot is a training-policy choice; blocking Claude-SearchBot is the visibility risk for Claude search-style surfacing. Confirm on Anthropic’s current help-center crawler page before editing production robots.txt.

Google-Extended: A robots.txt product token for whether Google-crawled content may be used for specified Gemini model training and grounding uses (Gemini Apps / Vertex AI grounding, per Google’s documentation and Search Engine Journal coverage of those docs). It is not a substitute for Googlebot, and Google states it is not a method for managing how content appears in Google Search. Blocking Google-Extended does not equal “remove me from Google” or “turn off AI Overviews.” AI Overviews eligibility still rides on normal Search indexing and snippet controls (for example nosnippet), not on this token.

Token / bot	Controls Search ranking?	Controls ChatGPT search?	Typical policy use
Googlebot	Yes (crawl/index path)	No	Search visibility
Google-Extended	No (per Google)	No	Gemini training/grounding opt-out
GPTBot	No	No (training)	OpenAI training opt-out
OAI-SearchBot	No	Yes	ChatGPT search eligibility

What robots.txt pattern should most brands ship?

For brands that want answer-engine citations and are okay deciding training separately:

# Citation / search retrieval — keep allowed
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Training — policy decision (example: opt out)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Gemini training/grounding token — policy decision
User-agent: Google-Extended
Disallow: /

# Do not blanket-ban everything unknown with User-agent: * Disallow: /
# unless you also Allow the bots you need.

Adjust the training lines to Allow: / if your legal/policy team wants training inclusion. The important part is splitting the decisions.

Decision list:

1. Do we want ChatGPT search citations? → OAI-SearchBot
2. Do we want Claude search-style indexing? → Claude-SearchBot
3. Do we allow foundation-model training? → GPTBot / ClaudeBot / Google-Extended
4. Did a WAF or CDN already block these at the edge? → fix that next

Failure mode: the “block all AI” CDN default

What breaks: a security dashboard enables “block AI crawlers” globally. robots.txt says Allow for OAI-SearchBot, but Cloudflare (or similar) returns 403 first. ChatGPT search never indexes you. You spend a quarter “doing AEO” on content that cannot be fetched.

What it costs: zero citations despite perfect answer-first pages.

What you do instead:

• Fetch https://yoursite.com/robots.txt anonymously
• Confirm separate User-agent blocks exist (not one muddy *)
• Check CDN / WAF bot scores for OpenAI and Anthropic published IP ranges
• curl -A "OAI-SearchBot" (and vendor equivalents) on About, offer, and top answer pages — expect 200
• Re-test after every security policy change

Fetchability is step 8 in the AEO audit checklist for a reason.

Will blocking GPTBot hurt Google rankings?

No — not as a Google ranking lever. GPTBot is OpenAI’s training crawler. Google Search crawl/index is Googlebot. These are different systems. Blocking GPTBot does not tell Google to demote you. Conversely, allowing GPTBot does not boost Google rank.

Do not conflate “AI” into one SEO myth.

How fast do robots.txt changes take effect?

OpenAI documents that for search results, it can take about 24 hours from a site’s robots.txt update for their systems to adjust. Other vendors differ; assume hours to a few days for automated crawlers, then re-verify with log lines and live answer tests.

Change	Expect
Allow OAI-SearchBot after accidental block	~24h for OpenAI search systems to adjust (per OpenAI), then longer for re-crawl of key URLs
Disallow GPTBot	Future training collection should respect the signal; already-trained model memory is a separate, slower clock
CDN unblock	Immediate for new fetches; old index residue clears on re-crawl

Training residue in model weights is not cleared by robots.txt. robots.txt governs future crawl/use signals — not a memory erase.

How to verify bots can fetch key pages

1. Confirm robots.txt allows the search agent on the path.
2. Confirm CDN/WAF allows the vendor’s published IPs / verified bot.
3. Confirm the page returns 200 without login.
4. Confirm the page is not noindex if you also care about Google AI Overviews.
5. Re-run five ChatGPT search prompts that should cite you; log whether your URL returns.

• robots.txt split training vs search
• WAF exceptions documented
• About + offer + top 5 answer URLs fetch clean
• Prompt panel archived post-change

Also keep /llms.txt consistent with what you allow crawlers to read — see llms.txt done properly.

Impostor bots and log hygiene

User-agent strings are trivial to spoof. Before you panic about “GPTBot ignoring robots.txt,” match the request IP to the vendor’s published ranges (OpenAI publishes JSON lists for GPTBot, OAI-SearchBot, and ChatGPT-User). Impostors wearing the UA are common.

Check	Action
UA says GPTBot, IP not in gptbot.json	Treat as impostor; do not rewrite policy on fakes
UA + IP match, hits Disallow path	File a vendor report if persistent; verify your robots.txt is reachable
robots.txt itself blocked by WAF	Fix that first — crawlers that cannot read rules cannot honor them

Anthropic has warned that IP-blocking their bots can prevent them from reading robots.txt at all. Prefer robots.txt signals over silent IP bans when you want a clean opt-out.

Policy worksheet for legal + marketing

Fill this once; store it next to the deploy checklist:

1. Training inclusion: allow / disallow (per vendor)
2. Search / citation inclusion: allow / disallow (per vendor)
3. User-initiated fetch: allow / monitor / restrict at edge
4. Review cadence: quarterly or on vendor-doc change
5. Owner: named eng + named marketer

• Worksheet signed off
• robots.txt matches worksheet
• CDN rules match worksheet
• Change log entry with date

If marketing wants citations and legal wants training opt-out, that is a normal, supported split — not a conflict that requires blocking everything.

FAQ

What is OAI-SearchBot vs GPTBot?

GPTBot crawls for OpenAI foundation-model training. OAI-SearchBot crawls to surface sites in ChatGPT search features. OpenAI treats them as independent robots.txt controls. Blocking training does not equal blocking search — and blocking search is how you disappear from ChatGPT search answers.

What about ClaudeBot and Google-Extended?

ClaudeBot is Anthropic’s training-oriented crawler; use Claude-SearchBot when the question is Claude search indexing. Google-Extended is Google’s product token for certain Gemini training and grounding uses — it does not control Google Search indexing or ranking. Do not block Googlebot thinking you only touched “AI.”

Do WAFs silently block AI bots?

Yes, often. CDN “AI scraper” defaults and bot-fight scores can 403 OAI-SearchBot while your robots.txt looks fine. Verify with user-agent curls and vendor IP lists after every security change.

Will blocking GPTBot hurt Google rankings?

No. Google rankings depend on Google’s crawlers and ranking systems, not on whether OpenAI’s training bot can fetch you. Keep Googlebot decisions separate from GPTBot decisions.

How fast do robots.txt changes take effect?

OpenAI notes roughly 24 hours for search systems to adjust after a robots.txt update. Plan for re-crawl lag beyond that. Training opt-outs affect future collection; they do not rewrite model memory overnight.

How do I verify bots can fetch key pages?

Allow the right agents in robots.txt, allow them at the CDN/WAF, confirm HTTP 200 on canonical answer URLs with the bot user-agent, then re-test live ChatGPT search prompts and log citations.

CTA

Split training policy from citation eligibility — then prove the fetch with a 200, not a vibes-based robots.txt screenshot.

Lane overview: /visibility. Next step: a visibility audit.