Can AI crawlers read your site?
If your robots.txt blocks GPTBot or PerplexityBot, no amount of optimization helps: you can't show up in answers built from content they never read. Check it in seconds.
The four that matter
GPTBot (ChatGPT), ClaudeBot, PerplexityBot and Google-Extended, read from your real robots.txt.
Nothing to install
We only read your public robots.txt, exactly as any search engine would.
We tell you what it means
Blocking isn't always a mistake — but it should be a decision, not an oversight.
Frequently asked questions
Is blocking AI crawlers bad?
Not necessarily — it's a business decision. If you live off content subscriptions, blocking makes sense. If you want AI to recommend you, blocking is a self-inflicted wound. What you don't want is for it to be an oversight inherited from a template or a CDN rule nobody reviewed.
Does Google-Extended affect my Google rankings?
No. Google-Extended only controls whether your content can feed Gemini and generative answers. Your classic search ranking doesn't change if you block it.
I get 'Can't verify' — what does that mean?
Your firewall or CDN refused our request, so we couldn't read your robots.txt. It's not a failure of your site; it's that we can't tell. Worth checking: AI crawlers also come from data centers and usually hit the same block.
Is allowing them enough to appear in ChatGPT?
No — it's necessary, not sufficient. Access is step one; after that it's content, structured data and whether other sources cite you.
Want this checked for you, every week?
These checks are a snapshot. AEON42 connects your Search Console, repeats them weekly, spots opportunities and tells you what to do. 14 days free, no card.