AI Crawlers and robots.txt: What to Allow for ChatGPT, Perplexity, Claude and Google
If an answer engine cannot fetch your pages, it cannot cite them. robots.txt and your CDN decide that before any "AEO content strategy" matters. This guide summarizes what major vendors document as of late 2026 and a practical pattern for local and B2B sites.
Search bots vs training bots
Most vendors split agents:
- Search / retrieval bots help the product find and link pages when answering questions.
- Training bots collect content that may be used to train foundation models.
- User-triggered fetchers load a page because a person asked; robots.txt may apply differently.
You can often allow search while blocking training. That is a policy choice, not a ranking hack.
OpenAI
Per OpenAI's bots documentation:
OAI-SearchBot— surfaces sites in ChatGPT search features. Blocking it means you will not be shown in those search answers (navigational links can still appear).GPTBot— used in connection with training generative AI foundation models.ChatGPT-User— user-initiated fetches; OpenAI notes robots.txt rules may not apply the same way as for automatic crawlers.
Perplexity
Per Perplexity's crawler docs:
PerplexityBot— surfaces and links sites in Perplexity search results; not for foundation-model training crawls per their docs.Perplexity-User— user-triggered fetches; generally ignores robots.txt because a user requested the page.
They publish IP lists and WAF notes — update allowlists from their JSON endpoints, not from a blog copy.
Anthropic (Claude)
ClaudeBot— training-related collection.Claude-SearchBot— search quality / indexing for search responses.Claude-User— user-initiated retrieval.
Googlebot and related crawlers remain the path into Search, including AI Overviews eligibility. Google's AI features documentation says a page must be indexed and snippet-eligible — no extra AI-only file required. Google-Extended is a separate control aimed at some Gemini training/grounding uses; do not confuse it with blocking Googlebot.
A practical robots.txt pattern
Many AEO-minded sites allow search agents and keep a separate decision on training:
User-agent: *
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Example: block training-oriented bots (optional policy)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
Copy only what matches your policy. Test with your CDN logs. AEOswitch.com currently allows the major search and assistant bots listed in our robots.txt.
CDN and WAF gotchas
Cloudflare bot fights, "browser integrity" challenges and aggressive rate limits can block legitimate crawlers even when robots.txt says Allow. If engines never cite you, check:
- robots.txt Disallow rules and wildcards
- WAF rules matching AI user-agents
- Geo or ASN blocks
- Whether
llms.txtexists (helpful hint file; not a substitute for crawl access)
Training opt-outs without cutting yourself out of answers
Publishers sometimes want to limit training use while remaining eligible for search-style answers. That only works if you keep search agents allowed and you understand each vendor's split. Re-read the official tables when you change policy — bot names and roles get updated.
Also remember: user-triggered fetchers may still retrieve a page when a person pastes your URL or the model decides to open it, even if automatic training crawlers are blocked.
Checklist before you declare "we're blocked"
- robots.txt fetched over HTTPS and HTTP
- www vs apex both covered
- Staging Disallow rules not copied to production
- WordPress or plugin SEO tools not injecting conflicting rules
- CDN "block AI scrapers" style toggles reviewed one by one
How to verify
- Open
https://yoursite.com/robots.txt - Confirm search bots you care about are not Disallowed
- Fetch a key URL from a clean network (or ask your host for bot logs)
- Re-check after CDN rule changes
Our free basic AEO audit looks for robots.txt and llms.txt hints when it can fetch them. A coming AI crawler robots.txt checker will go deeper.
Where this sits in an AEO project
Crawler access is usually week-one work. It is cheap to verify and expensive to ignore. Pair it with consistent business details and answer-first pages before spending months on outreach.
Should I allow all AI bots?
Not necessarily. Many sites allow search/indexing bots and block training bots. Decide based on your content policy, then implement consistently across robots.txt and the CDN.
Does blocking GPTBot hide me from ChatGPT answers?
OpenAI documents OAI-SearchBot for search appearance. GPTBot is framed around training. Blocking one is not the same as blocking the other — read the current OpenAI bots page when you change rules.
Is llms.txt required?
No. It is an optional hint file. Google says you do not need special AI text files for AI Overviews. Crawl access and clear pages matter more.
Why do user agents ignore robots.txt sometimes?
Vendors often say user-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User) follow a person\'s request and may not honor robots.txt the same way automatic crawlers do.
Can you audit our crawler access?
Yes. Crawler access is part of every full AEO visibility audit ($49) and of Audit + Fixes work.
Related: AEO by the numbers · Book a $49 audit
