AI crawler access checker
Enter a public URL to see its robots.txt rules, redirects and HTTP response. Results group crawlers by purpose: search, user fetch, training, an open crawl dataset, or when the vendor does not state a purpose. Link previews and ads have their own group.
The live check sends a simulated crawler user agent. It is a diagnostic request, not proof that an official crawler can pass your CDN, firewall or bot protection.
Start from a ready file
Our guide has three robots.txt files to copy, each defined by the crawlers it blocks. Each one opens here with its results.
robots.txt for AI crawlersQuestions
Does allowing a crawler in robots.txt get my site cited by AI search?
No. robots.txt only says whether a crawler may fetch a path. Selection and citation depend on the product, and no setting guarantees either.
Why are crawlers listed by purpose?
Vendors document different uses for their crawlers. Results distinguish search, user-requested fetches, training, an open crawl dataset, and crawlers whose vendor does not state a purpose. Link previews and ads appear in a separate group.
Does a robots.txt rule stop every Meta crawler?
No. Meta says Meta-ExternalFetcher and FacebookExternalHit may bypass robots.txt. To stop them, use a rule on your server or firewall.
Does the live check prove a real crawler can reach my site?
No. The live check uses a simulated user agent from our network. A CDN, firewall or bot-protection rule can still treat official crawler traffic differently.
Check the rest of your site
Crawler access is one part of it. The free audit also checks indexing, server-rendered content, canonicals and your sitemap.
Run free audit