Is Your robots.txt Blocking AI? A Guide to AI Crawler Access

Why AI Crawler Access Matters

When someone asks ChatGPT, Gemini, or Perplexity about your industry, those systems answer from two places: what they learned during training, and what they fetch live from the web. Both depend on crawlers being allowed to read your site. If your robots.txt blocks them — often by accident — you remove yourself from the sources AI draws on, and you become invisible exactly where buyers are now asking questions.

The frustrating part: many sites block AI crawlers without realizing it. A blanket rule copied from an SEO template, an overzealous security plugin, or a CDN default can quietly shut the door.

The AI Crawlers That Matter in 2026

Different bots do different jobs, so blocking the wrong one has different consequences.

• GPTBot (OpenAI) — trains and improves models, so future versions can learn your facts.
• OAI-SearchBot (OpenAI) — powers ChatGPT Search results and citations.
• ChatGPT-User — fetches a page live when a user's prompt requires it.
• ClaudeBot (Anthropic) — Anthropic's primary crawler for training and retrieval.
• Claude-SearchBot and Claude-User — search indexing and live user fetches for Claude.
• PerplexityBot and Perplexity-User — Perplexity's index and on-demand fetches.
• Google-Extended — controls whether Google can use your content for Gemini and AI features, separate from normal Search indexing via Googlebot.
• CCBot — Common Crawl, an open dataset that feeds many models indirectly.

Note that Claude-Web and anthropic-ai are retired tokens. If your robots.txt still names them, it is out of date — use ClaudeBot and Claude-SearchBot instead.

How to Check If You Are Blocking Them

1. Open yourdomain.com/robots.txt in a browser.
2. Look for any "Disallow: /" rule, especially under "User-agent: *" or under a specific AI bot name.
3. Watch for a rule that names an AI bot and then disallows it — that is an explicit block.
4. Remember that a rule under "User-agent: *" applies to every bot that does not have its own more specific block.

A single "Disallow: /" under "User-agent: *" blocks your entire site from every crawler that respects robots.txt.

A Recommended AI-Friendly robots.txt

Allow all crawlers by default, then explicitly welcome the AI bots. Each block is a "User-agent:" line followed by "Allow: /":

• User-agent: * → Allow: /
• User-agent: GPTBot → Allow: /
• User-agent: OAI-SearchBot → Allow: /
• User-agent: ChatGPT-User → Allow: /
• User-agent: ClaudeBot → Allow: /
• User-agent: Claude-SearchBot → Allow: /
• User-agent: PerplexityBot → Allow: /
• User-agent: Google-Extended → Allow: /
• User-agent: CCBot → Allow: /

Finish with a "Sitemap:" line pointing to your sitemap.xml. You can still disallow private paths such as checkout, account, and admin — just do not disallow the pages that describe who you are and what you offer.

robots.txt Is Necessary, But Not Sufficient

Allowing crawlers only removes the barrier. Being cited still depends on having crawlable, factual, well-structured content: a clear description of your business, Organization schema, an FAQ that answers real questions, and an llms.txt file that spells out your key facts. Access gets AI in the door; quality is what earns the recommendation.

Verify It

After updating your robots.txt:

1. Re-fetch /robots.txt and confirm the AI bots are allowed.
2. Confirm your sitemap is referenced and reachable.
3. Run a scan on VisibleForAI — the crawler-access audit checks each major AI bot and flags anything still blocked.
4. Re-check in a few weeks to confirm recognition improves as crawlers re-index you.