GEO & AI search · updated · 10 min read· By Frederiek Pascal

AI-Readiness: can ChatGPT and other AI crawlers actually read your website?

AI-Readiness is the technical base of GEO: crawler access, server-rendered HTML and structured data. How to check each one and fix the common blind spots.

An AI-Readiness checklist with schema.org, FAQ pages and sitemap, scoring 74 out of 100

Summary

  • AI-Readiness measures how well AI systems can reach, crawl, read and understand your website. It is the technical base under your GEO.
  • The biggest blind spot is access: block the wrong bot by accident in robots.txt and you disappear from ChatGPT search.
  • AI crawlers usually do not execute JavaScript, so content that only appears in the browser is invisible to them.
  • Structured data and llms.txt help, but they are no silver bullet: research shows little direct effect on AI citations.

An AI-Readiness score measures whether AI systems can reach, read and understand your website. It is the technical layer beneath your AI visibility, and the quiet reason brands stay invisible: you can write the best content in your sector, but if a crawler cannot get in or only sees an empty page, you appear nowhere. This article explains what AI-Readiness is, which technical factors really count, and where most sites lose points.

What is an AI-Readiness score?

An AI-Readiness score measures how well AI systems like ChatGPT, Claude, Gemini and Perplexity can reach, crawl, read and understand your website. It is not a measure of whether you get cited, that is your GEO Score, but of the technical condition behind it. If AI cannot process your site, it cannot cite you either.

The difference matters. Your GEO Score tells you whether you appear in the answers, and how to raise it is covered separately. Your AI-Readiness score tells you whether it is technically possible to get in there at all. A low AI-Readiness is often the hidden cause of a low GEO Score: your content is fine, but AI cannot reach it or reads it wrong. Veesie's AI-Readiness Audit checks your website on 20 points and sums them up in a score from 0 to 100, next to your GEO Score.

Can AI reach your site at all? The AI crawlers

The first question is access: does your website let the right AI bots in? Every major AI provider uses separate crawlers with a different role, and that is often where it goes wrong.

According to the official OpenAI documentation, four OpenAI bots are in play, each with its own task: GPTBot crawls content to train models, OAI-SearchBot makes sure your site appears in ChatGPT search, ChatGPT-User fetches pages when a user actively asks for them, and OAI-AdsBot checks advertising pages. Anthropic works in a similar way with ClaudeBot (training), Claude-User (real-time questions) and Claude-SearchBot (search). Perplexity has PerplexityBot for search results and Perplexity-User for answers. How each chatbot then picks its sources differs, as how to appear in AI chatbots explains.

Here is the biggest blind spot. OpenAI confirms it directly: sites that block OAI-SearchBot do not appear in the ChatGPT search answers (ChatGPT SEO covers how ChatGPT picks sources once it can read you). Many websites block AI bots in one sweep to "stop scraping", and in doing so accidentally shut out the very search crawler that makes them findable. The distinction is crucial:

  • If you want to be visible in AI answers, let the search crawlers in: OAI-SearchBot, PerplexityBot and the search crawlers of the other providers.
  • If you do not want your content to train models, block the training crawler specifically (GPTBot via robots.txt), without giving up your search visibility.

A robots.txt that keeps you in AI answers but out of model training looks like this, for example:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

This keeps you visible in ChatGPT search and Perplexity, while you do not hand your content over to model training. The opposite, blocking everything with one blanket rule, is exactly how sites make themselves invisible in AI answers.

For Google there is a separate case: Google-Extended is not a separate crawler but a robots.txt token that controls whether Google uses your content for Gemini training. Google states explicitly that this token does not affect your position in regular Google search results. How Google picks sources for its AI summaries is covered in Google AI Overviews.

Does AI see your content, or only a blank page?

Even when an AI crawler gets in, the question is whether it actually sees your content. Many AI crawlers do not execute JavaScript. According to an analysis by Vercel from late 2024, the crawlers of ChatGPT and Claude fetch JavaScript files but do not run them. Crawlers change, so test your own site rather than rely on a rule of thumb. The result: content that is only built up in the browser with JavaScript does not exist for those bots.

This is a silent killer for modern sites. Online shops and apps that load their product info, prices or text client-side often show an AI crawler a near-empty page. The fix is to keep your most important content in the server-rendered HTML, not only after a JavaScript step. For AI-Readiness there is a simple test: turn JavaScript off in your browser and check whether your core content is still there. What you cannot see then, an AI model usually cannot see either.

A second check is "view page source" in your browser. If your real text, your headings and your key facts are simply present in that HTML, an AI crawler can read them. If you mostly see empty containers and scripts, that is a red flag for your AI-Readiness, however strong your content is on substance.

Does structured data help for AI?

Structured data makes your page machine-readable, but it does not lift your AI citations. That nuance matters, because a lot of nonsense gets sold around it.

Ahrefs studied 1,885 pages that added JSON-LD schema and found no meaningful rise in AI citations on any platform; citations in Google AI Overviews even fell by 4.6%, a statistically significant drop. In other words: adding schema does not push a page higher in AI answers.

Even so, structured data is not useless. Searchviu had five AI systems fetch a page live in late October 2025 and saw that they did not read the JSON-LD: whatever sits only in your schema and not in the visible text does not exist for those systems. Schema stays useful for correct display in search engines and for entity recognition. So put every fact you want cited in the visible text as well. The full Ahrefs figures and the other common misunderstandings about schema and AI are worked out in 8 GEO myths marketers still believe.

What about llms.txt?

llms.txt is a proposed file to point AI to your most important content, but expect little from it for now. The idea comes from Jeremy Howard (Answer.AI), who published the proposal in September 2024: a markdown file at the root of your site that hands AI models your key pages in a single overview.

In practice, adoption has stalled: no major AI provider publicly confirms that it reads or honours llms.txt, and Google's Search Advocate even compared it to the keywords meta tag. An llms.txt takes little effort and does no harm, but do not expect growth from it. Put your energy into access and renderability instead.

The four pillars of AI-Readiness

Veesie's AI-Readiness Audit runs twenty checks, ten classic SEO checks and ten GEO signals, and scores them per category (SEO and GEO). The four pillars below are a way of thinking about AI-Readiness, not four scores in Veesie. The checks cover structured data, content signals and indexability, and a separate check shows which AI crawlers your robots.txt lets in. HTTPS, load speed and external mentions are not part of the score, so you assess those by hand.

PillarWhat it measuresMost common problemPriority
1. Technical accessibilityCrawlers, HTTPS, renderable HTMLWrong bot blocked in robots.txtHigh
2. Structured dataSchema.org JSON-LD, machine readabilityMissing or conflicting schemaMedium
3. Content signalsHeadings, lists, citable structureLong paragraphs without structureMedium
4. AuthorityEntity recognition, external mentionsInconsistent brand name, few external sourcesLow
Pillar1. Technical accessibility
What it measuresCrawlers, HTTPS, renderable HTML
Most common problemWrong bot blocked in robots.txt
PriorityHigh
Pillar2. Structured data
What it measuresSchema.org JSON-LD, machine readability
Most common problemMissing or conflicting schema
PriorityMedium
Pillar3. Content signals
What it measuresHeadings, lists, citable structure
Most common problemLong paragraphs without structure
PriorityMedium
Pillar4. Authority
What it measuresEntity recognition, external mentions
Most common problemInconsistent brand name, few external sources
PriorityLow
  1. Technical accessibility: do you let the right AI crawlers in? Is your core content in the server HTML, with no JavaScript dependency? Is your site fast and on HTTPS?
  2. Structured data: are your pages fitted with correct Schema.org markup for clear machine readability and search engine context?
  3. Content signals: is your content clearly structured with headings, lists and definitions, so a model can pull passages from it?
  4. Authority: are you recognised as an entity through consistent mentions and authoritative sources?

That third pillar, readable structure, is where the human reader and the machine reader meet:

If you want your content to perform well in the modern retrieval systems, you need to structure it in a way that's readable and understandable primarily for your readers but also for machine-readable search engine crawlers.

Zach Chahalis, Senior Director of SEO & Data Analytics at iPullRank, in Technical SEO for AI Search (2025)

A high score across all four does not automatically mean you get cited, but a low score almost certainly means you do not. AI-Readiness is the gateway, your GEO Score measures what makes it through.

How to improve your AI-Readiness in 5 steps

Improving AI-Readiness starts with removing technical blockers, not with more content. Five concrete steps:

  1. Check your robots.txt. Make sure you do not block the search crawlers (OAI-SearchBot, PerplexityBot) by accident. Block only what you genuinely do not want, such as training crawlers.
  2. Test your renderability. Turn JavaScript off and check whether your core content holds up. If not, make sure it is server-side rendered.
  3. Structure your content. Use clear H2 headings, lists and direct definitions, so a model can extract relevant passages.
  4. Add correct structured data. For machine readability and search engine context, not for citations.
  5. Measure and repeat. The AI-Readiness Audit from Veesie (from the Plus plan, and in the free trial) scans the on-page factors automatically and shows an SEO score and a GEO score with a recommendation per check.

If you want to work through this as a checklist you can tick off, use the GEO audit checklist. And if you want to know whether it works, compare your position with the GEO benchmark by sector and track your GEO Score. How your existing SEO carries over is covered in SEO for AI, and GEO vs SEO and GEO marketing place the channel next to classic search and the rest of your marketing mix.

Conclusion: readable first, visible next

AI visibility does not start with better text, but with a site that AI can reach, read and understand. Most blind spots are technical and invisible: a mistakenly blocked crawler, content that only appears with JavaScript, or the wrong expectations of schema and llms.txt.

AI-Readiness is therefore the quiet base under your whole GEO strategy. Veesie makes that base measurable for marketers and agencies, so you first make sure you are readable, and only then fight for the citation. Invisible to the crawler is invisible in the answer. See What is GEO? for the full framework, read 8 tips to improve your GEO Score for the path after your AI-Readiness, follow the five steps in how to do GEO, or start the free trial to measure your own score.

AI-Readinessstructured dataschema markupAI crawlersGPTBotllms.txttechnical SEO

Frequently asked questions

Sources and references

  1. OpenAI: Bots and crawlers (GPTBot, OAI-SearchBot, ChatGPT-User)
  2. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
  3. Perplexity: Perplexity Crawlers (PerplexityBot, Perplexity-User)
  4. Google Search Central: Google crawlers and Google-Extended
  5. Ahrefs: Does schema markup help AI citations? (2026)
  6. Searchviu: Schema markup and AI in 2025, what ChatGPT, Claude, Perplexity and Gemini really see
  7. Vercel: The rise of the AI crawler (JavaScript rendering by AI bots)
  8. llms.txt: the proposal (Jeremy Howard, Answer.AI, 2024)
  9. Search Engine Journal: Google compares llms.txt to the keywords meta tag
  10. iPullRank: Technical SEO for AI Search

Discover your own GEO Score

7 days free, no credit card. You measure ChatGPT, Perplexity and Google AI Overviews; Claude and Gemini are added from Plus.

Get started

Related articles