robots.txt for AI Search: OAI‑SearchBot, Claude, and Perplexity
Do not treat every AI-related user agent as the same bot. OpenAI, Anthropic, and Perplexity publish separate agents for search, user-requested fetches, and model training. Your robots.txt policy should reflect the purpose you actually want to allow.
This guide covers the main AI crawlers and fetchers that vendors document, how robots.txt controls them, and an example configuration you can adapt.
Current Crawler Roles You Need to Know
The catalog below was reviewed against official vendor documentation on 11 Oct 2026. Crawler names and behavior can change, so verify the linked sources before changing a production policy.
| Bot (operator) | Purpose | Default Behavior |
|---|---|---|
| OAI-SearchBot (OpenAI) | PurposeSearch crawler used to surface sites in ChatGPT search | Default BehaviorCrawls unless blocked |
| GPTBot (OpenAI) | PurposePotential model-training collection. Not the ChatGPT search control | Default BehaviorCrawls unless blocked |
| ChatGPT-User (OpenAI) | PurposeUser-initiated page retrieval. Not automatic search crawling | Default Behaviorrobots.txt may not apply to every user-initiated request |
| Claude-SearchBot (Anthropic) | PurposeSearch crawler used to improve Claude search results | Default BehaviorCrawls unless blocked |
| Claude-User (Anthropic) | PurposeFetches pages when a Claude user asks a question | Default BehaviorAnthropic says its bots honor robots.txt |
| ClaudeBot (Anthropic) | PurposeCollects web content that could contribute to model training | Default BehaviorCrawls unless blocked |
| PerplexityBot (Perplexity AI) | PurposeSurfaces and links sites in Perplexity search. Per Perplexity, not used for foundation-model training | Default BehaviorCrawls unless blocked |
| Perplexity-User (Perplexity AI) | PurposeSupports user actions in Perplexity | Default BehaviorPerplexity says it generally ignores robots.txt rules |
| Google-Extended (Google) | PurposeToken for Gemini training and some grounding uses. No effect on Google Search inclusion | Default BehaviorToken only, with no separate user-agent string |
| Googlebot (Google) | PurposeGoogle's crawler for Search, including its AI features | Default BehaviorCrawls unless blocked |
| Amazonbot (Amazon) | PurposeImproves Amazon products. Amazon says data may train its AI models | Default BehaviorCrawls unless blocked |
| Meta-ExternalAgent (Meta) | PurposeCrawls for uses such as AI training or indexing to improve products | Default BehaviorCrawls unless blocked |
| CCBot (Common Crawl) | PurposeBuilds Common Crawl's open repository of web crawl data | Default BehaviorCrawls unless blocked |
| Applebot-Extended (Apple) | PurposeControl for whether Applebot data may train Apple's generative AI models | Default BehaviorDoes not crawl itself. Per Apple, disallowing it does not remove pages from Apple search |
The critical distinction is between three kinds of bot:
- Search crawlers such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot
- User-fetch agents such as ChatGPT-User, Claude-User, and Perplexity-User
- Training crawlers such as GPTBot and ClaudeBot
Blocking a training crawler is not the same as opting out of search.
How robots.txt Controls AI Bot Access
The robots.txt file sits at the root of your site, for example https://example.com/robots.txt. It tells web crawlers which parts of your site they may or may not access. It uses a simple protocol: you specify a User-agent and then list Allow or Disallow rules beneath it.
Here is a basic example that blocks GPTBot entirely:
User-agent: GPTBot
Disallow: /
And here is one that allows GPTBot to crawl everything except your admin area:
User-agent: GPTBot
Allow: /
Disallow: /admin/
Rules are evaluated per user-agent. If a bot does not find its own user-agent string in your robots.txt, it falls back to the wildcard rule (User-agent: *). If there is no wildcard rule, the bot assumes everything is allowed.
A few important technical details:
- robots.txt is advisory, not enforcement. Well-behaved bots from major companies honor it. Malicious scrapers ignore it entirely. It is not a security mechanism.
- Rules match URL paths. Under RFC 9309, patterns can use
*and$wildcards, but robots.txt works on URLs, not on what a page contains. - More specific rules win. The longest matching rule applies, so
Allow: /blog/takes precedence overDisallow: /for the same user-agent when the path matches. - Changes are not instant. Crawlers may cache robots.txt. RFC 9309 says they should not use a cached copy for more than 24 hours unless the file is unreachable. OpenAI says it can take about 24 hours for a robots.txt update to affect ChatGPT search.
The Case for Allowing AI Bots
Allowing a documented search crawler keeps the public pages technically eligible for that vendor's search experience. Eligibility does not guarantee indexing, citation, ranking, or referral traffic.
Measure actual referrals and mentions separately. Robots access is only one observable prerequisite, not a visibility metric.
Some crawling is also tied to features you already depend on. Google says there are no additional requirements to appear in AI Overviews or AI Mode beyond being indexed and eligible to be shown in Google Search with a snippet. Blocking Googlebot is not a realistic option for most sites, and it would stop Google crawling your pages for Search.
The Case for Blocking AI Bots
The main concern with allowing AI bots is content being used to train models without compensation or attribution. When a training crawler such as GPTBot or ClaudeBot collects your pages, the content may be used to train models. A model trained on your content may produce answers that draw on it without linking back to you. For publishers and content creators, this can feel like having your work used without permission.
There are also competitive concerns. If you run a research firm, consulting practice, or premium content business, your intellectual property is your product. Allowing AI bots to ingest your proprietary analysis means that analysis may appear in AI-generated answers available to everyone, including your competitors' customers.
Bandwidth and server load are practical concerns for smaller sites. Check your server logs for heavy automated traffic. If your hosting infrastructure is limited, aggressive crawling can affect site performance for real users. Anthropic documents support for the non-standard Crawl-delay directive to limit crawl frequency.
Finally, there is the legal and ethical dimension. Whether and on what terms AI companies may use publicly accessible web content for model training is contested, and the answer can differ by country. Some site owners prefer to block training-only bots as a precaution.
A Sensible robots.txt Starting Point
For many public marketing sites, a reasonable starting point is a selective policy: allow documented search crawlers, decide separately on user-fetch agents, and make an explicit choice about training crawlers.
Here is a robots.txt configuration that implements this strategy:
# Allow documented search crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Keep ordinary Google Search eligible, including its AI features
User-agent: Googlebot
Allow: /
# Optional user-fetch policy (OpenAI and Perplexity say their user-fetch agents may not follow robots.txt)
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
# Example training preference: opt out without blocking search
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Default: allow everything else
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
This example separates access by purpose. It does not promise visibility. It may not fit these kinds of site:
- Publishers
- Private documentation
- Regulated content
- Sites with different licensing requirements
For three complete files to copy, each defined by the crawlers it blocks, see robots.txt for AI crawlers.
For a deeper look at the scope of a technical audit, see our AEO guide.
Recommended default: allow search-facing AI crawlers, keep standard search crawlers open, and make an explicit decision on training-only bots. After editing robots.txt, run the AEOprobe audit to verify the actual bot verdicts instead of assuming the directives behave as intended.
Adjust based on your business model. If you run a media site that monetizes through traffic, allowing search-facing bots is almost certainly the right call. If you sell proprietary research, you may want to be more restrictive. The key is making a deliberate choice rather than leaving your robots.txt on its default settings and hoping for the best.
How to Check Your robots.txt AI Bot Rules
Many site owners have not reviewed what their robots.txt says about AI bots. A robots.txt written before AI crawlers existed gives every AI bot full access by default unless a wildcard rule says otherwise.
AEOprobe's free audit checks robots.txt against its current, versioned crawler catalog and separates search, user-fetch, and training roles. It reports allowed, blocked, partial, and inherited wildcard behavior.
The free audit covers up to 3 pages and needs no sign-up. Enter your URL, and you will see:
- Which tracked agents are allowed by robots.txt
- Whether you have a wildcard rule that inadvertently allows or blocks everything
- Whether your sitemap is declared in robots.txt, so crawlers can find it
- An overall AI Bot Access grade from A+ to F
Beyond robots.txt, the audit reports on crawlability, canonicals, sitemaps, structured metadata and rendering. It also runs general web-quality checks. These are diagnostics, not a guarantee of citation.
Run your free AEO audit now and see what your robots.txt allows for each tracked AI bot.
Questions
Does blocking GPTBot prevent my site from appearing in ChatGPT search?
No. OpenAI documents GPTBot for potential model-training collection. The robots.txt control for ChatGPT search is OAI-SearchBot. These settings are independent.
Can I block AI training but still appear in AI search results?
Yes, where vendors publish separate controls. For example, OpenAI allows a site to permit OAI-SearchBot while disallowing GPTBot. Anthropic similarly separates Claude-SearchBot from ClaudeBot. Google states that Google-Extended does not control ordinary Search.
How often should I review my robots.txt AI bot rules?
A regular check, such as quarterly, is a reasonable habit. Vendors add and rename crawlers, and OpenAI notes that the version numbers in its user-agent strings may change. AEOprobe checks a versioned crawler catalog, so you can audit your configuration whenever you change robots.txt or a vendor updates its documentation.
What happens if I do not mention an AI bot in my robots.txt at all?
If a specific bot's user-agent is not listed in your robots.txt, it falls back to the wildcard (User-agent: *) rule. If your wildcard rule says Allow: / or has no Disallow directives, the bot can crawl your entire site. A permissive or missing wildcard rule means unlisted AI bots get full access by default. This is why it is important to explicitly address the bots you want to block.
Related posts
What Is Answer Engine Optimization (AEO)? The Complete Guide for 2026
AEO is the technical groundwork that lets search and AI answer products reach, index, and understand your website.
Read moreAEO vs SEO: What Is Different and What to Do About It
AEO builds on SEO. This guide covers the crawlers to manage separately and the 11 categories an AEOprobe audit scores.
Read moreAI Search Audit: A Technical SEO Audit for AI Readiness
An AI search audit is a technical SEO audit that checks whether AI and search crawlers can reach and read your site.
Read moreBest AEO Tools 2026: 7 Tools Compared (Diagnostic vs Monitoring)
Seven AEO tools compared by job, from technical diagnostics to AI visibility monitoring. Prices are checked on each vendor’s page.
Read moreCheck what AI crawlers can actually reach
Run a free audit to find public crawler, indexing, markup, and page-structure issues.
Run free audit