robots.txt for AI crawlers
Three versions of one robots.txt, each defined by the crawlers it blocks. Pick one, copy it and put it at the root of your site.
Updated
Pick a version
Allow all AI crawlers
Every crawler in the table may read every page. The file blocks nothing.
Our checker reads this file as all 20 crawlers allowed. Open it in the checker to see each result.
robots.txt
# Allow all AI crawlers
# Crawler names checked 14 Oct 2026
# Every crawler
User-agent: *
Allow: /
# Replace with your own sitemap address
Sitemap: https://example.com/sitemap.xml
Allow search and answers, block training
AI search, web search and the fetches a person asks for may read your pages. The 8 crawlers for training, data collection or an undocumented purpose are blocked.
Our checker reads this file as 12 crawlers allowed and 8 blocked. Open it in the checker to see each result.
robots.txt
# Allow search and answers, block training
# Crawler names checked 14 Oct 2026
# Training and data collection
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /
# Not documented by the vendor
User-agent: Bytespider
User-agent: cohere-ai
Disallow: /
# Every other crawler
User-agent: *
Allow: /
# Replace with your own sitemap address
Sitemap: https://example.com/sitemap.xml
Block AI crawlers, keep web search
All 16 AI crawlers are blocked: AI search, the fetches a person asks for, training and the undocumented ones. bingbot and Applebot stay allowed because they also run Bing and Apple search. Link previews stay allowed too.
Our checker reads this file as 4 crawlers allowed and 16 blocked. Open it in the checker to see each result.
robots.txt
# Block AI crawlers, keep web search
# Crawler names checked 14 Oct 2026
# AI search
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Meta-WebIndexer
Disallow: /
# Fetches a person asks for
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Meta-ExternalFetcher
Disallow: /
# Training and data collection
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /
# Not documented by the vendor
User-agent: Bytespider
User-agent: cohere-ai
Disallow: /
# Every other crawler
User-agent: *
Allow: /
# Replace with your own sitemap address
Sitemap: https://example.com/sitemap.xml
How the three differ
Each version allows or blocks whole groups of crawlers, and each crawler sits in one group. The groups follow what each vendor says its crawler is for.
| Crawler group | Allow all | Block training | Block AI crawlers |
|---|---|---|---|
| AI search4 crawlers | Allow allAllowed | Block trainingAllowed | Block AI crawlersBlocked |
| Web search that also feeds AI answers2 crawlers | Allow allAllowed | Block trainingAllowed | Block AI crawlersAllowed |
| Fetches a person asks for4 crawlers | Allow allAllowed | Block trainingAllowed | Block AI crawlersBlocked |
| Training and data collection6 crawlers | Allow allAllowed | Block trainingBlocked | Block AI crawlersBlocked |
| Not documented by the vendor2 crawlers | Allow allAllowed | Block trainingBlocked | Block AI crawlersBlocked |
| Link previews and ads2 crawlers | Allow allAllowed | Block trainingAllowed | Block AI crawlersAllowed |
Before you upload it
- The file replaces your whole robots.txt. If your current file has rules for private paths, copy those lines under
User-agent: *first. - Put it at the root of your site, at /robots.txt. Anthropic asks (opens in a new tab) site owners to do this “for every subdomain that you wish to opt out from”.
- Change the last line to your own sitemap address, or delete it.
- Changes are not instant. OpenAI says (opens in a new tab) “it can take ~24 hours from a site’s robots.txt update for our systems to adjust”. Perplexity says (opens in a new tab) “it may take up to 24 hours for our systems to reflect changes”.
- None of the three files block Googlebot. On AI features in Google Search, Google says (opens in a new tab): “AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search.”
The 20 crawlers, in their vendors’ words
These are the crawlers our free audit checks. Text in quotes is the vendor’s own. Plain grey text is our note, used where the vendor says nothing.
Each vendor page was read on 14 Oct 2026.
AI search
Crawlers that find pages to show and link in an AI product's answers.
| User agent | Who runs it | What it is for | What it says about robots.txt | Source |
|---|---|---|---|---|
| OAI-SearchBot | Who runs itOpenAI | What it is for“used to surface websites in search results in ChatGPT's search features” | What it says about robots.txt“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” | SourceOverview of OpenAI Crawlers (opens in a new tab) |
| Claude-SearchBot | Who runs itAnthropic | What it is for“navigates the web to improve search result quality for users” | What it says about robots.txt“Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization” | SourceDoes Anthropic crawl data from the web? (opens in a new tab)Page dated 7 Apr 2026 |
| PerplexityBot | Who runs itPerplexity | What it is for“designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.” | What it says about robots.txt“To ensure your site appears in search results, we recommend allowing PerplexityBot in your site's robots.txt file” | SourcePerplexity Crawlers (opens in a new tab) |
| Meta-WebIndexer | Who runs itMeta | What it is for“navigates the web to improve Meta AI search result quality for users” | What it says about robots.txt“Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses.” | SourceWeb crawlers (opens in a new tab) |
Web search that also feeds AI answers
Search engine crawlers. Their vendors say the same crawl serves their AI answers.
| User agent | Who runs it | What it is for | What it says about robots.txt | Source |
|---|---|---|---|---|
| bingbot | Who runs itMicrosoft | What it is for“Bingbot is our standard crawler and handles most of our crawling needs each day.”“Bing and Copilot search experiences rely on the same core crawling, indexing, and ranking foundation as traditional search.” | What it says about robots.txt“Robots.txt files can be configured to tell Bing crawlers how to interact with your website.” | SourceWhich crawlers does Bing use? (opens in a new tab)Webmaster Guidelines (opens in a new tab) |
| Applebot | Who runs itApple | What it is for“Enabling Applebot in robots.txt allows website content to appear in search results for Apple users”“The data crawled by Applebot may also be used to help train Apple foundation models” | What it says about robots.txt“Applebot respects standard robots.txt directives in general search crawls that are targeted at Applebot.” | SourceAbout Applebot (opens in a new tab)Page dated 4 Sept 2026 |
Fetches a person asks for
Someone asks the AI a question and it opens your page to answer. Three of the four vendors say these may not follow robots.txt.
| User agent | Who runs it | What it is for | What it says about robots.txt | Source |
|---|---|---|---|---|
| ChatGPT-User | Who runs itOpenAI | What it is for“When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent.” | What it says about robots.txt“Because these actions are initiated by a user, robots.txt rules may not apply.” | SourceOverview of OpenAI Crawlers (opens in a new tab) |
| Claude-User | Who runs itAnthropic | What it is for“When individuals ask questions to Claude, it may access websites using a Claude-User agent.” | What it says about robots.txt“Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query” | SourceDoes Anthropic crawl data from the web? (opens in a new tab)Page dated 7 Apr 2026 |
| Perplexity-User | Who runs itPerplexity | What it is for“When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response.” | What it says about robots.txt“Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” | SourcePerplexity Crawlers (opens in a new tab) |
| Meta-ExternalFetcher | Who runs itMeta | What it is for“fetches individual links at a user's request” | What it says about robots.txt“may bypass robots.txt because it performs fetches that were requested by the user” | SourceWeb crawlers (opens in a new tab) |
Training and data collection
Crawlers and tokens that decide whether your pages go into model training or an open dataset.
| User agent | Who runs it | What it is for | What it says about robots.txt | Source |
|---|---|---|---|---|
| GPTBot | Who runs itOpenAI | What it is for“used to crawl content that may be used in training our generative AI foundation models” | What it says about robots.txt“Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models.” | SourceOverview of OpenAI Crawlers (opens in a new tab) |
| ClaudeBot | Who runs itAnthropic | What it is for“collecting web content that could potentially contribute to their training” | What it says about robots.txt“When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets.” | SourceDoes Anthropic crawl data from the web? (opens in a new tab)Page dated 7 Apr 2026 |
| Google-Extended | Who runs itGoogle | What it is for“a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models”It has no user agent string of its own. Google crawls with its usual crawlers. | What it says about robots.txt“Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.” | SourceGoogle's common crawlers (opens in a new tab)Page dated 14 Jul 2026 |
| Applebot-Extended | Who runs itApple | What it is for“Applebot-Extended does not crawl webpages.”“Applebot-Extended is only used to determine how to use the data crawled by the Applebot user agent.” | What it says about robots.txt“Web publishers can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended”“Webpages that disallow Applebot-Extended can still be included in search results.” | SourceAbout Applebot (opens in a new tab)Page dated 4 Sept 2026 |
| Meta-ExternalAgent | Who runs itMeta | What it is for“crawls the web for use cases such as training foundation AI models or improving products by indexing content directly” | What it says about robots.txt“In order to block these crawlers, add a disallow for the relevant crawler to robots.txt.” | SourceWeb crawlers (opens in a new tab) |
| CCBot | Who runs itCommon Crawl | What it is for“an open repository of web crawl data that is universally accessible and analyzable by anyone”Common Crawl does not state AI training as CCBot’s purpose. | What it says about robots.txt“Add these lines to your robots.txt file and our crawler will stop crawling your website” | SourceCCBot (opens in a new tab)FAQ (opens in a new tab) |
Not documented by the vendor
Our audit checks these names, but the vendor publishes no page that describes them.
| User agent | Who runs it | What it is for | What it says about robots.txt | Source |
|---|---|---|---|---|
| Bytespider | Who runs itByteDance | What it is forByteDance does not state Bytespider’s purpose. | What it says about robots.txtThe vendor does not say. | SourceNone found |
| cohere-ai | Who runs itCohere | What it is forCohere's crawler page lists no crawler by this name. It says:“We do not use Cohere bots or user agents for the purpose of crawling or scraping web content to train generative AI foundation models at this time.” | What it says about robots.txt“It is Cohere's policy to require that crawlers be designed to respect robots.txt.” | SourceCohere Web Crawlers (opens in a new tab) |
Link previews and ads
Meta crawlers that build link previews and serve advertising. Meta does not describe them as AI crawlers.
| User agent | Who runs it | What it is for | What it says about robots.txt | Source |
|---|---|---|---|---|
| FacebookExternalHit | Who runs itMeta | What it is for“crawl the content of an app or website that was shared on one of Meta's family of apps, such as Facebook, Instagram, or Messenger” | What it says about robots.txt“might bypass robots.txt when performing security or integrity checks” | SourceWeb crawlers (opens in a new tab) |
| Meta-ExternalAds | Who runs itMeta | What it is for“crawls the web for use cases such as improving advertising and other business-related products and services” | What it says about robots.txt“In order to block these crawlers, add a disallow for the relevant crawler to robots.txt.” | SourceWeb crawlers (opens in a new tab) |
See what your site blocks today
The free audit reads your live robots.txt and shows which of these 20 crawlers it blocks.
Run free audit