Can AI read your site?
If your robots.txt blocks ChatGPT's search crawler or Perplexity's, no amount of optimization helps: you can't show up in answers built from content they never read. Check it in seconds.
The ones that decide
Googlebot, OAI-SearchBot (ChatGPT), PerplexityBot, Claude-SearchBot, Claude-User and Bingbot, read from your real robots.txt.
Nothing to install
We only read your public robots.txt, exactly as any search engine would.
Training, kept apart
GPTBot, ClaudeBot, Google-Extended and Applebot-Extended only collect content to train models. We tell you if you block them, but blocking them doesn't take you out of the answers.
Frequently asked questions
Is blocking AI crawlers bad?
It depends on which one. Blocking the training crawlers (GPTBot, ClaudeBot, Google-Extended) is a legitimate business decision: your content stays out of model training and you can still appear in answers. Blocking the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) does take you out of the answers. What you don't want is an oversight inherited from a template or a CDN rule nobody reviewed.
Does blocking Google-Extended remove me from AI Overviews?
No. Google states that Google-Extended doesn't affect your inclusion or ranking in its search, and AI Overviews and AI Mode are part of search: Googlebot feeds them. Google-Extended only controls Gemini training and its use in other Google products.
I get 'Can't verify' — what does that mean?
Your firewall or CDN refused our request, so we couldn't read your robots.txt. It's not a failure of your site; it's that we can't tell. Worth checking: AI crawlers also come from data centers and usually hit the same block.
What about ChatGPT-User or Perplexity-User?
Those are visits the assistant makes when a user asks it to, and their own makers say they may not follow robots.txt. That's why we don't check them: a rule there wouldn't have a reliable effect.
Is allowing them enough to appear in ChatGPT?
No — it's necessary, not sufficient. Access is step one; after that it's content, structured data and whether other sources cite you.
Want this checked for you, every week?
These checks are a snapshot. AEON42 connects your Search Console, repeats them weekly, spots opportunities and tells you what to do. 14-day free trial.