Should I block AI crawlers in robots.txt?

Short answer

If you want to be recommended by AI assistants, do not block the crawlers that power their search and citation features. Blocking GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, or Google-Extended removes your pages from the retrieval pool those products draw on, which removes your chance of being cited.

Separate training from retrieval

Some crawlers primarily collect training data, others fetch pages at answer time. If your concern is training use, you can allow the retrieval and search agents while restricting the training-focused ones. Decide deliberately rather than inheriting a blanket block from a template.

Check what else is blocking you

robots.txt is only one layer. CDN bot-management rules, firewall rules, aggressive rate limiting, and JavaScript-only rendering block AI crawlers just as effectively. Verify with real fetch tests, not just the file.

Keep reading

Other questions

Find out where you stand in AI answers

Free AI Visibility Audit across ChatGPT, Gemini, Claude, and Perplexity.

Run free AI audit