---
title: "Should I block AI crawlers in robots.txt?"
description: "If you want to be recommended by AI assistants, do not block the crawlers that power their search and citation features. Blocking GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, or Google-Extended removes your pages from the retrieval pool those products draw on, which removes your chance of being cited."
canonical: https://atomikdigital.com/answers/should-i-block-ai-crawlers-in-robots-txt
site: Atomik Digital
language: en-US
format: text/markdown
category: "Getting cited"
---
# Should I block AI crawlers in robots.txt?

**Direct answer:** If you want to be recommended by AI assistants, do not block the crawlers that power their search and citation features. Blocking GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, or Google-Extended removes your pages from the retrieval pool those products draw on, which removes your chance of being cited.

## Separate training from retrieval

Some crawlers primarily collect training data, others fetch pages at answer time. If your concern is training use, you can allow the retrieval and search agents while restricting the training-focused ones. Decide deliberately rather than inheriting a blanket block from a template.

## Check what else is blocking you

robots.txt is only one layer. CDN bot-management rules, firewall rules, aggressive rate limiting, and JavaScript-only rendering block AI crawlers just as effectively. Verify with real fetch tests, not just the file.

## Related

- [How to Appear in AI Search Results](https://atomikdigital.com/blog/how-to-appear-in-ai-search-results)
- [Our robots.txt](https://atomikdigital.com/robots.txt)
---

Source: https://atomikdigital.com/answers/should-i-block-ai-crawlers-in-robots-txt
Provider: Atomik Digital — AI visibility optimization for modern brands — get cited and recommended by ChatGPT, Gemini, Claude, Perplexity, and Copilot.
Contact: info@atomikdigital.com
Plans: https://atomikdigital.com/plans (from $249/month, purchased online)