Technical AI Search Readiness Methodology

Evidence levels

Confirmed requirement: A documented access or eligibility control, such as HTTP reachability, a named search crawler policy, or an explicit noindex directive.
Web fundamental: A durable SEO, accessibility, security, or reliability practice that benefits users and crawlability without being presented as a citation factor.
Heuristic: An observable characteristic that may make a page easier to inspect or understand, but whose ranking or citation effect is not documented.
Experimental: An emerging convention or inferred signal that is not a documented requirement. The AI readiness category carries this label and still counts for 10 of 100 points in the overall score. llms.txt and agentic-browsing observations are reported but not scored.

Level	What it means
Confirmed requirement	A documented access or eligibility control, such as HTTP reachability, a named search crawler policy, or an explicit noindex directive.
Web fundamental	A durable SEO, accessibility, security, or reliability practice that benefits users and crawlability without being presented as a citation factor.
Heuristic	An observable characteristic that may make a page easier to inspect or understand, but whose ranking or citation effect is not documented.
Experimental	An emerging convention or inferred signal that is not a documented requirement. The AI readiness category carries this label and still counts for 10 of 100 points in the overall score. llms.txt and agentic-browsing observations are reported but not scored.

Crawler purposes

Search agents such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot are reported separately from user-fetch agents and training crawlers such as GPTBot and ClaudeBot. CCBot builds an open crawl dataset for research and analysis. A training opt-out is not automatically a search failure. Link previews and ads are another group: Meta's FacebookExternalHit builds the preview of a shared link, and Meta-ExternalAds serves advertising.

The audit reports robots.txt policy for these 20 agents. Googlebot is not one of them. Google-Extended is a robots.txt token only, with no user-agent string of its own, and Google says it does not affect inclusion in Google Search. ByteDance does not state Bytespider's purpose. Cohere's crawler page lists no bot by that name and says it does not use crawlers to train foundation models at this time. The report marks both purposes as not stated by the vendor.

A robots.txt rule is not a reliable block for every agent. Meta says Meta-ExternalFetcher and FacebookExternalHit may bypass it. The report shows the rule it read and adds a note.

Purpose	Tracked agents
Search	`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `bingbot`, `Applebot`, `Meta-WebIndexer`
User fetch	`ChatGPT-User`, `Claude-User`, `Perplexity-User`, `Meta-ExternalFetcher`
Training	`GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `Meta-ExternalAgent`
Open crawl dataset	`CCBot`
The vendor does not say	`Bytespider`, `cohere-ai`
Link previews and ads	`FacebookExternalHit`, `Meta-ExternalAds`

Score interpretation

The grade is a summary of checks in one audit version. Compare it only when scope and methodology match, and use the individual findings to decide what to fix. Experimental observations such as llms.txt do not become urgent merely because they are absent.

The AI readiness heuristics category looks at site-side content structure and source context. It does not observe visibility in any AI product. It is labelled Experimental and weighs 10 of 100 points in the overall score, so its findings can move the grade even though they are not documented requirements.

Rechecks

A recheck inspects the selected page and keeps its result linked to the to-do. Host-level observations, including robots.txt and sitemaps, are labelled separately. A recheck costs credits, shown before you run it, and the credits are given back if it fails. Comparing technical findings or search data does not establish that a fix caused a ranking or traffic change.

Known limitations

Imported AI visibility reports

AI visibility shows impressions from your imported Google AI report. Imports are snapshots you supply. They do not change the readiness score. Google can export unavailable values as zero. We show those values as unknown.

Only a complete property-level date series supplies the summary total. Page counts are never added to that total. Overlapping imports stay separate. Comparisons need matching scope and duration, complete dates and confirmed final data. A difference does not show that a content change caused it.

Public crawl limitations

A public crawl cannot confirm a URL's status in a vendor's private index.
robots.txt permission does not prove a vendor crawled the page.
Structured data does not guarantee a search feature or citation.
A simulated user-agent fetch is not proof of access from an official vendor IP range.

Primary sources

Found a change? Send a correction.