Why it matters
ChatGPT search, Claude and Perplexity fetch web pages to answer questions. If their crawlers can't get in, or see an empty page, your business can't be part of those answers from your own website.
1. Check your robots.txt
Open yourwebsite.com/robots.txt. Look for groups starting with User-agent: OAI-SearchBot, ChatGPT-User, ClaudeBot or PerplexityBot followed by Disallow: /. Any of those blocks that crawler from your whole site. A User-agent: * group with Disallow: / blocks everyone.
2. Check your page without JavaScript
OpenAI's and Anthropic's crawlers download your HTML but don't run your scripts. In Chrome, open your page, press Ctrl+U (Cmd+Option+U on Mac) to view the source, and search for a sentence from your homepage. If it isn't there, AI crawlers don't see it either.
3. Training crawlers are your choice
GPTBot and Google-Extended collect content for model training. Blocking them is a reasonable business choice and doesn't remove you from ChatGPT search, which uses OAI-SearchBot.
The fast way
GEO Guru's free check does both in about 20 seconds and shows which crawlers are blocked and how many words they can read.
Check your own site now
Free, no sign-up, about 20 seconds.