We Audited 50 Sites for AI-Readiness — Here’s What We Found

We ran AEOprobe audits on 50 websites across five industries to answer a simple question: how ready is the average website for AI-powered search?

The answer: not very. The average AEO score across all 50 sites was 41 out of 100 — a D+ grade. Only 12% of sites scored a B or higher. The vast majority of the web is effectively invisible to AI search engines like ChatGPT, Perplexity, Claude, and Google AI Overviews.

This is the first AEO Index — a benchmark study designed to quantify the gap between where websites are today and where they need to be to compete in AI-driven search. Here is what we found.

Methodology

We selected 50 websites across five industries: SaaS/technology (12 sites), media and publishing (10 sites), e-commerce (10 sites), professional services (10 sites), and local businesses (8 sites). Selection criteria:

• Mix of company sizes — from solo-founder startups to enterprise brands
• All sites had at least some organic search presence (ranking for at least a few keywords)
• No sites were AEOprobe customers at the time of the audit (to avoid bias)
• Each site was audited using AEOprobe's standard 9-category assessment against 14 AI and search crawlers

Audits were conducted between March 10-20, 2026. Scores reflect the state of each site at the time of the crawl.

Overall Results: The AEO Index

Metric	Value
Average AEO Score	41/100 (D+)
Median AEO Score	38/100 (D)
Highest Score	87/100 (A)
Lowest Score	11/100 (F)
Sites scoring A or A+	2 (4%)
Sites scoring B or higher	6 (12%)
Sites scoring D or F	31 (62%)

The distribution is heavily skewed toward the bottom. Nearly two-thirds of sites received a D or F, meaning their content is largely inaccessible to AI search engines. Only 2 sites out of 50 achieved an A-level score.

Score Distribution by Grade

Grade	Score Range	Count	Percentage
A+ / A	90-100 / 80-89	2	4%
B+ / B	75-79 / 70-74	4	8%
C+ / C	65-69 / 55-64	7	14%
D+ / D	45-54 / 35-44	6	12%
F	0-34	31	62%

The F-grade cluster is striking. These are not abandoned or broken websites — many are actively maintained, have decent traditional SEO, and generate real traffic. They simply have not addressed the technical requirements for AI search visibility.

Results by Category

AEOprobe scores sites across 9 categories. Here is how the 50 sites performed in each:

Category	Average Score	% Scoring A	% Scoring F
AI Bot Access	34%	18%	54%
Structured Data	28%	14%	62%
Content Quality	56%	22%	16%
Meta Tags	52%	26%	20%
Sitemap	45%	30%	28%
Performance	61%	36%	10%
Security	48%	20%	24%
Accessibility	44%	16%	30%
llms.txt	6%	6%	88%

Key observations

llms.txt is almost nonexistent. Only 3 out of 50 sites (6%) had an llms.txt file. This is the newest AEO signal and adoption is still extremely low. Early adopters have a clear differentiation opportunity.

Structured data is the second-worst category. 62% of sites scored an F for structured data. Many had no JSON-LD markup at all — not even a basic Organization or WebPage schema. This is a missed opportunity because structured data directly affects how AI engines understand and cite your content.

AI bot access is widely mismanaged. With an average score of just 34%, most sites are actively or accidentally blocking AI crawlers. More on this below.

Performance is the strongest category. Most modern sites have reasonable page speed and HTTPS, which is why performance averaged 61%. This is the one area where existing web standards align naturally with AEO requirements.

Content quality is better than technical readiness. Content averaged 56% — most sites have decent content, they just have not structured it for machine consumption. The content exists; the machine-readable wrapper is missing.

Finding 1: 73% of Sites Block at Least One AI Crawler

This is the most critical finding. Nearly three-quarters of audited sites block one or more AI crawlers in their robots.txt file.

AI Bots Blocked	Sites	Percentage
0 (none blocked)	14	27%
1-2 bots blocked	17	35%
3-5 bots blocked	11	22%
6+ bots blocked	8	16%

The most commonly blocked crawlers:

Crawler	% of Sites Blocking
GPTBot	52%
Google-Extended	44%
ClaudeBot	38%
Bytespider	36%
PerplexityBot	28%
ChatGPT-User	24%
Cohere-ai	22%
Meta-ExternalAgent	20%

GPTBot is the most blocked AI crawler — over half of sites explicitly disallow it. This means these sites cannot appear in ChatGPT's search-powered answers, regardless of how good their content is.

Many of these blocks appear to be copied from online templates or added reactively when AI crawling first became a topic in 2024, without considering the implications for AI-powered search visibility.

Finding 2: Only 12% Have an llms.txt File

The llms.txt standard is designed to give large language models explicit context about a website — its purpose, structure, key pages, and preferred citation format. Despite being a straightforward file to create, adoption is extremely low.

• 3 out of 50 sites (6%) had an llms.txt file
• All 3 were SaaS/technology companies
• No e-commerce, professional services, or local business sites had one

This represents a significant early-mover advantage. Adding an llms.txt file takes about 10 minutes and immediately differentiates your site from 88% of competitors in our sample.

Finding 3: Structured Data Gaps Are Universal

Structured data (JSON-LD markup) is how you give AI engines machine-readable context about your content. Our audit found widespread gaps:

• 38% of sites had no JSON-LD structured data at all
• 24% had only basic Organization schema — missing Article, FAQ, Product, or other content-specific markup
• 18% had schema with errors — missing required properties, invalid types, or syntax errors
• Only 20% had comprehensive schema coverage across their key pages

The correlation between structured data completeness and overall AEO score was strong: sites with comprehensive schema averaged 68/100 overall, while sites with no schema averaged 31/100.

Finding 4: Content Quality Outpaces Technical Readiness

An interesting pattern emerged: many sites have good content that is poorly wrapped in technical signals.

• Content quality averaged 56% — the second-highest category
• Structured data averaged only 28% — meaning the content exists but is not machine-readable
• Meta tags averaged 52% — many sites have partial metadata but miss canonical URLs or Open Graph tags

This suggests that the primary barrier to AEO readiness for most sites is not content quality — it is technical implementation. The content is there. The machine-readable packaging is not.

Finding 5: Industry Performance Varies Widely

Industry	Sites	Avg Score	Best Score	Worst Score
SaaS / Technology	12	52	87	24
Media / Publishing	10	48	81	19
E-commerce	10	38	72	14
Professional Services	10	35	65	11
Local Business	8	29	58	12

SaaS and technology companies lead — not surprising given their technical teams and familiarity with web standards. Even so, the average SaaS score of 52 is still only a C+.

Local businesses trail significantly at an average of 29, largely due to lack of structured data, outdated robots.txt files, and minimal technical optimization.

The gap between industries represents an opportunity: a local business that invests 2-3 hours in AEO optimization would leapfrog most of its competitors.

Finding 6: Security Headers Are Widely Neglected

Security scored an average of 48%, making it the fourth-worst category. Common failures:

• 44% of sites were missing HSTS headers
• 56% lacked a Content-Security-Policy header
• 32% had mixed content (HTTP resources loaded on HTTPS pages)
• 28% were missing X-Frame-Options

Security headers matter for AEO because AI crawlers use them as trust signals. A site without HSTS or with mixed content appears less reliable as a source, which can affect citation priority.

Finding 7: Sitemaps Are Present but Often Broken

72% of sites had a sitemap.xml file, but many had issues:

• 34% of sitemaps contained URLs returning 404 errors
• 48% had stale or missing lastmod dates
• 18% had sitemaps not referenced in robots.txt
• 8% had XML parsing errors

A sitemap that exists but contains broken links or stale dates can actually hurt crawl efficiency — crawlers waste time requesting pages that no longer exist or that have not changed.

Surprising Finding: SEO Rank Does Not Predict AEO Score

We expected sites with strong traditional SEO to also score well on AEO. They did not.

Several sites ranking on page one of Google for competitive keywords scored D or F on AEO. The reasons:

• Blocking AI crawlers in robots.txt (often added in 2024 as a knee-jerk reaction to AI scraping)
• Relying on backlinks and domain authority rather than technical optimization
• No structured data because "Google doesn't need it to rank us"
• Optimized for Google's ranking algorithm but not for AI retrieval pipelines

The inverse was also true: some smaller sites with modest SEO scored well on AEO because they had clean technical implementations, proper structured data, and no AI bot blocks.

AEO is a leveler. It rewards technical correctness over domain authority, which means smaller sites can compete if they get the fundamentals right.

What This Means for Your Site

If the average website scores 41/100, there is a significant opportunity for sites that invest even modest effort in AEO optimization. Here is what we recommend based on our findings:

1. Audit your robots.txt immediately — if you are blocking GPTBot, ClaudeBot, or PerplexityBot, you are invisible to those AI search engines. This is the highest-impact fix.
2. Add structured data to your key pages — at minimum, add Article, Organization, and FAQ schemas. This lifts you above 62% of sites that have none.
3. Create an llms.txt file — 10 minutes of work puts you in the top 12% of sites for this signal.
4. Fix your security headers — HSTS, CSP, and X-Frame-Options take minutes to add in your server config.
5. Validate your sitemap — remove broken URLs, update lastmod dates, and ensure your sitemap is referenced in robots.txt.

The sites that score well are not doing anything exotic. They are executing on fundamentals that most sites have simply not addressed yet.

How We Will Track This

We plan to re-audit this same set of 50 sites quarterly to track how AEO readiness evolves across the web. The AEO Index will serve as a benchmark — if you score above 41, you are already ahead of the median. But the bar will rise as more sites optimize.

See how your site scores — run a free AEO audit and compare yourself against the benchmark. Most sites have significant room to improve, and the first-mover advantage is real.