robots.txt for AI Crawlers: GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot
Which AI crawler does what, how to allow the ones you want citations from, and how to block training without disappearing from answers.
Most site owners who “blocked AI” did it by pasting a snippet that blocks everything. Then they wondered why the assistant stopped citing them. The crawlers are not interchangeable: some fetch live answers, some collect training data, and blocking one has nothing to do with the other.
Who is crawling your site
| Crawler | Owner | Purpose | Blocking cost |
|---|---|---|---|
| GPTBot | OpenAI | Training corpus collection | You lose training inclusion; live search still works |
| OAI-SearchBot | OpenAI | Index for ChatGPT search answers | You disappear from ChatGPT search citations |
| ChatGPT-User | OpenAI | User-triggered page fetch inside a chat | Links users paste stop resolving |
| ClaudeBot | Anthropic | Crawling for Claude | Lower chance of Claude citing you |
| PerplexityBot | Perplexity | Answer index | Removed from Perplexity answers |
| Google-Extended | Gemini training and grounding | No Gemini grounding; Google Search unaffected | |
| Applebot-Extended | Apple | Apple Intelligence training | No effect on Siri or Spotlight |
| CCBot | Common Crawl | Open dataset used by many models | Widest indirect exclusion |
A sane default policy
For a content site that wants citations, allow the answer crawlers and decide separately about training. For a members-only or scraped-content-risk site, block aggressively but accept the visibility loss.
# Allow answer engines
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
# Training crawler: optional
User-agent: GPTBot
Allow: /
# Never crawl these
User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xmlNever write User-agent: * with Disallow: / and then expect an allow rule further down to override it for a specific bot. Order matters, and the most specific matching group wins — so put bot-specific groups first and test afterwards.
Verify instead of guessing
- Fetch
https://yourdomain.com/robots.txtin a private window and confirm it is not cached from an old version. - Check that your CDN is not serving HTML with a 200 status for missing files — a soft 404 of robots.txt confuses crawlers badly.
- Confirm your sitemap URL in the file actually returns XML with
<loc>entries. - Re-run a visibility audit and confirm the crawler checks pass.
Static sites and GitHub Pages
On GitHub Pages you cannot set custom headers, but a plain robots.txt in the repository root is served as text and works fine. Keep the file small, keep it honest, and do not add comments that describe pages you removed two years ago.
Build the whole file in one click with the AI crawler robots.txt Builder. It toggles each bot, adds private-path rules and appends your sitemap.
The business decision behind the technical one
Blocking AI training is a legitimate choice, and so is allowing it. What is not legitimate is an inconsistent policy — blocking GPTBot while paying an agency for “ChatGPT visibility”. Pick a position, write it down, and make the robots file match it.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT answers?
Not directly. Live answers come from search crawlers such as OAI-SearchBot. Training inclusion is a separate path.
Will AI crawlers respect robots.txt?
The major vendors document that they do. Smaller scrapers may not, in which case rate limiting and terms of service are your tools.
Is Google-Extended the same as Googlebot?
No. Googlebot crawls for Search. Google-Extended controls Gemini training and grounding only.