Which AI Bots Can Crawl Your Site? A Practical Guide to AI Crawler Access

In 2020, your robots.txt had two audiences: Googlebot and everyone else. In 2026, there are more than a dozen AI crawler user-agents you may want to differentiate — some indexing content for AI search retrieval (Perplexity, Claude), others gathering training data (GPTBot, CCBot), and some doing both under different tokens.

This guide covers every major AI bot, what it does, and how to configure robots.txt to control access precisely.

What AI crawlers exist in 2026?

Here are the major AI-associated crawlers active on the web:

| Bot token | Owner | Primary purpose | |---|---|---| | GPTBot | OpenAI | Web crawling for ChatGPT training data | | ChatGPT-User | OpenAI | Real-time retrieval during ChatGPT Browse sessions | | ClaudeBot | Anthropic | Web crawling for Claude training data | | Claude-User | Anthropic | Real-time retrieval for Claude's web browsing | | PerplexityBot | Perplexity | Index crawling for Perplexity search results | | Google-Extended | Google | Controls access to Google's AI training (Bard, Vertex) separately from Search | | Gemini-Google-Bot | Google | Retrieval for Google Gemini AI responses | | Applebot-Extended | Apple | AI features and Siri intelligence — separate from Applebot for web indexing | | CCBot | Common Crawl | Open web crawl used for training many AI models | | meta-externalagent | Meta | Training and retrieval for Meta AI |

The critical distinction is training vs retrieval:

• Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) collect pages for model training datasets. Blocking them affects what future model versions know about your content.
• Retrieval bots (ChatGPT-User, Claude-User, PerplexityBot, Gemini-Google-Bot) fetch pages in real-time when a user asks a question in an AI interface. Blocking them means your content cannot appear in AI-generated answers right now.

You can allow one and block the other for the same AI system — OpenAI provides separate tokens for this purpose.

Why does AI crawler access matter for SEO?

Traditional SEO is about ranking in Google's index. GEO (Generative Engine Optimization) is about being cited in AI-generated answers. These are different outcomes from different signals.

If your robots.txt blocks PerplexityBot, Perplexity cannot include your content in its answers — even if your pages are high-quality and well-structured. The block is absolute.

In data from DeepSEOAnalysis audits:

• Over 30% of sites accidentally restrict at least one major AI crawler
• The most common accidental block: User-agent: * / Disallow: / on staging sites that carried over to production
• The second most common: older robots.txt templates that predate AI crawlers and don't explicitly allow them (relying on default-allow, which is fine, but without explicit Allow rules, some edge cases arise)

An AI-visibility audit flags this as a high-severity GEO gap when AI crawlers are blocked because it's the most disqualifying signal — no access means no citation.

How do you check which AI bots can crawl your site?

The fastest way: use the AI crawler access checker. Paste your URL, and the tool fetches your robots.txt server-side and tests all 10 major AI bots against the root path.

Or manually:

1. Fetch https://yourdomain.com/robots.txt
2. Look for User-agent: groups that reference GPTBot, ClaudeBot, PerplexityBot, Google-Extended, or CCBot
3. Check whether they have Disallow: / or Disallow: (empty, which means no restriction)
4. If you have User-agent: * / Disallow: /, all bots including AI crawlers are blocked

How should you configure robots.txt for AI bots?

There are four common configurations depending on your goals:

Allow everything (simplest, maximizes AI visibility)

User-agent: *
Allow: /

This is the default if no robots.txt exists. Explicitly stating it is cleaner. All bots can crawl all paths.

Allow AI search, block AI training

If you want AI systems to cite you in real-time answers but not use your content for model training:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

This blocks the training-focused tokens while allowing retrieval bots like ChatGPT-User, PerplexityBot, and Claude-User to index your content for real-time answers.

Note: blocking GPTBot does not retroactively remove your content from existing training data. It prevents future crawls from updating the training set.

Allow specific paths for AI bots

If you want AI crawlers to only access your public documentation or blog, not your app or admin areas:

User-agent: GPTBot
Allow: /blog/
Allow: /docs/
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/

Block all AI crawlers

If you don't want any AI access:

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Gemini-Google-Bot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

Be explicit — don't rely on User-agent: * to catch AI bots generically, because some AI bots may still crawl under novel tokens not caught by a wildcard.

What about llms.txt — do AI crawlers read it?

llms.txt is a companion file to robots.txt, not a replacement. robots.txt controls access; llms.txt guides AI systems toward your most citeable content.

Perplexity reads llms.txt when fetching content. Some Claude web retrieval modes use it to understand site structure before deciding what to retrieve. It's a layer on top of access control — you need allowed access (via robots.txt) before llms.txt guidance matters.

Use the llms.txt generator to create a well-structured guide file once your robots.txt is confirmed to allow AI crawlers.

Do AI crawlers respect robots.txt?

Major AI companies (OpenAI, Anthropic, Google, Meta) have publicly committed to respecting the Robots Exclusion Protocol for their crawlers. Perplexity had a controversy in 2024 around respect for robots.txt rules, but subsequently updated their policies.

CCBot (Common Crawl) generally respects robots.txt but indexes at massive scale — if you block it, you're blocking content from appearing in training datasets for many open-source models, not just one company's AI.

Blocking robots.txt is advisory, not enforced. Unknown or poorly-behaved scrapers may ignore it. For content you want to keep private, robots.txt alone is insufficient.

How to verify your robots.txt is working correctly

After updating your robots.txt:

1. Check the tool directly: Use the AI crawler access checker to confirm the correct bots are allowed/blocked as intended.
2. Test specific paths: Use the robots.txt tester to paste your file and test individual user-agents against the paths you care about.
3. Run a full audit: The DeepSEOAnalysis audit checks robots.txt as part of the GEO score — it flags any AI crawler that is blocked and estimates the AI-visibility impact.

robots.txt changes take effect as soon as crawlers next check the file. Most major crawlers re-check robots.txt periodically (Googlebot typically within 24 hours; AI crawlers vary).

---

Frequently asked questions

Can I allow Perplexity but block ChatGPT? Yes. Use User-agent: GPTBot / Disallow: / and User-agent: ChatGPT-User / Disallow: / to block OpenAI, while leaving PerplexityBot without a Disallow rule (or with an explicit Allow: /).

Does blocking GPTBot affect Bing's indexing of my site? No. GPTBot is OpenAI's crawler. Bing uses its own Bingbot user-agent, which is unaffected by GPTBot rules.

What is the difference between GPTBot and ChatGPT-User? GPTBot crawls the web for training data collection. ChatGPT-User fetches pages in real-time when a ChatGPT user activates web browsing in a chat. You can block one independently of the other.

Should I add an explicit Allow: / for AI bots if I want them? If your User-agent: * block allows everything, you don't need separate Allow: / rules for each AI bot. They inherit the wildcard rule. Explicit entries are only needed if you have a more restrictive wildcard or want to grant access to specific paths for specific bots.

Does my robots.txt affect the GEO score in a DeepSEOAnalysis audit? Yes. The GEO score checks whether major AI crawlers are allowed access. A blocked bot is flagged as a warning-severity AI-visibility gap. Allowing all major AI crawlers is the fastest GEO win if any are currently blocked.