Search crawlers vs training crawlers
Some AI crawlers fetch pages to answer a user's question right now (OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot). Others collect content for training models (GPTBot, Google-Extended).
A sensible default
Allow the search-type crawlers so AI tools can find and cite you. Decide on training crawlers based on how you feel about your content being used for training.
Common accident
Security plugins and some hosts add blanket AI blocks. Check your robots.txt after installing plugins or moving hosts.
Check it
Run a free GEO Guru check: the report lists every AI crawler and whether it's allowed.
Check your own site now
Free, no sign-up, about 20 seconds.