GPTBot & ClaudeBot AI Crawler Access Audit
When someone asks ChatGPT, Claude, or Perplexity a question your site answers, whether your content surfaces depends on a chain that runs long before the query. A bot had to reach your page, read it without executing JavaScript, and find nothing in your robots configuration blocking the path. Most site owners have never audited that chain. Here is how to check every link in it.
TL;DR: GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are separate robots.txt user-agents — blocking one does not block the others, and a stray wildcard rule can block all of them by accident. Most AI crawlers do not execute JavaScript, so client-side-rendered content is invisible to them even when robots.txt allows access. To know your real exposure, audit four layers: live robots.txt per user-agent, rendered robots meta tags, X-Robots-Tag headers, and llms.txt. VisibilityIQ checks all four and reports each bot independently.
Why AI crawlers are not just another Googlebot
Traditional crawlers operate on a familiar model: crawl, index, rank. Block the crawler and the page does not rank. AI systems layer two additional access patterns on top of that, and the distinction determines what blocking actually costs you.
Training crawls ingest content to train or fine-tune model weights. Content absorbed during training becomes part of the model’s parametric knowledge — what it knows independent of any live lookup. RAG crawls retrieve current content at query time to ground an answer in up-to-date information. Perplexity is heavily retrieval-first: it fetches live pages when answering and cites them directly. Google AI Overviews blend both, drawing on trained knowledge plus live retrieval from the index.
The practical implication is that “AI access” is not one switch. Blocking GPTBot affects training inclusion and ChatGPT browsing. Blocking PerplexityBot affects whether your pages are retrieved and cited in live Perplexity answers. These are distinct effects with distinct consequences, which is why our AI visibility audit never collapses them into a single pass/fail. It reports each bot’s status on its own, the way the audit-engine standard requires — separate user-agent behaviors are never conflated into one finding.
The bots that matter and what each one does
GPTBot (OpenAI) gathers training data and increasingly supports real-time browsing in ChatGPT. Blocking it via robots.txt removes your content from future training runs and from that browsing path.
OAI-SearchBot (OpenAI) grounds search-style answers and Copilot-adjacent features. It is a separate user-agent from GPTBot, so a GPTBot block does not cover it.
ClaudeBot (Anthropic) collects training data and supports Claude’s web search. Its user-agent string is ClaudeBot; older traffic sometimes appears as anthropic-ai. Because Claude’s search reads live pages, a ClaudeBot block removes you from those answers.
PerplexityBot retrieves live content to ground Perplexity’s cited answers. Perplexity is explicitly retrieval-first, so blocking PerplexityBot means your pages will not be cited even for queries where you would otherwise be the best source.
Google-Extended is Google’s opt-out for AI training. Standard Googlebot keeps crawling for search; Google-Extended governs whether your content trains Gemini and powers AI features. Disallowing it does not touch your organic rankings — a point I expand on in the companion post on whether llms.txt is the right control for AI access. For a broader map of how each AI system — ChatGPT, Perplexity, Claude — uses what it crawls and the practical consequences of blocking each one, see the guide to AI visibility and how AI search engines crawl versus traditional Googlebot.
Bytespider (ByteDance) is aggressive and widely blocked for crawl-load reasons; whether you want its access is a business call, but you should know its status.
The robots.txt traps that block AI bots by accident
The robots.txt standard gives you per-user-agent control. A deliberate block of one crawler is explicit:
User-agent: GPTBot
Disallow: /The dangerous failures are the unintentional ones. A wildcard rule meant for crawl hygiene can sweep up AI bots you intended to allow, because a bot with no matching specific group falls back to the User-agent: * group:
User-agent: *
Disallow: /blog/If /blog/ is your citation-worthy content and you never wrote a GPTBot-specific group, that rule blocks GPTBot from exactly the pages you want cited. The inverse trap is assuming a specific allow group is inherited — it is not; robots.txt groups do not merge, so a bot obeys only its most specific matching group. Auditing this by eye across a large robots.txt is error-prone, which is why our site audit parses the live file per user-agent and shows you, bot by bot, exactly which paths each can and cannot reach.
The JavaScript wall most AI bots cannot climb
Even a perfectly permissive robots.txt does not guarantee an AI bot can read your content. Most of them — GPTBot, ClaudeBot, PerplexityBot — fetch raw HTML and do not execute JavaScript. If your primary answer, your headings, or your body copy are injected after hydration, those bots see a shell. This is the same defect that produces a render-parity gap for Googlebot, except the consequence is harsher: Googlebot eventually renders on a delay, while AI crawlers simply move on. That asymmetry is what makes server-side rendering a visibility decision rather than a performance preference — the architectural choice determines whether AI crawlers can read your content at all, independent of what your robots.txt allows.
The fix is to ensure your citation-worthy content — the direct answer, the data, the clear claims — exists in the raw HTML response, not only in the rendered DOM. Server-side rendering or static generation puts it there. A client-side-rendered page can be flawlessly indexable by robots.txt and still be unreadable by every major AI crawler.
Auditing access across all four layers
A complete check covers four layers, and a gap in any one breaks AI visibility regardless of the others.
Live robots.txt. Fetch the file from the actual URL — not the copy in your repo, because CDN rules and edge workers can rewrite it — and parse the rules for each AI user-agent, including wildcard fallbacks.
Rendered robots meta. Inspect the rendered DOM, not just view-source, for <meta name="robots"> values like noai, noindex, or noimageai, since these are frequently injected by JavaScript.
HTTP headers. Check X-Robots-Tag response headers with curl -I; they carry the same directives for non-HTML assets like PDFs and are invisible in the browser.
llms.txt and raw content. Verify /llms.txt is well-formed, and confirm your primary content is present in the raw HTML so non-rendering bots can read it.
The edge case that trips up the most sites is a discrepancy between source control and what is actually served. CDN configurations, edge workers, and reverse proxies can all modify the robots.txt response or inject headers before any crawler sees them — which is exactly why every layer of this audit reads the live, served state.
Robots.txt meta tags for AI: noai and noimageai
Beyond path-level robots.txt rules, a second control surface has emerged specifically for AI: the noai and noimageai meta directives. Placed in a meta robots tag or carried in an X-Robots-Tag header, noai signals that a page’s content should not be used for AI training or generation, and noimageai extends that to images. These are page-level signals rather than crawl blocks — they ask a model not to use the content it fetched, rather than preventing the fetch.
The distinction matters for how you reason about them. robots.txt is the enforceable access control: a respected Disallow stops the crawl entirely. noai is a use directive: the crawler may still fetch the page but is asked not to use it for AI purposes. The two serve different intents — block access versus permit access but restrict use — and a complete AI-access posture often combines them. You might allow a retrieval crawler to fetch a page so it can cite you, while declining training use of the same content. Auditing these directives means reading the rendered meta tags and the response headers, not just robots.txt, because noai frequently lives where a path-only crawl analysis never looks.
There is one more layer worth checking: response headers that signal AI content policy. Some sites emit an X-OpenAI-Content-Policy or similar header to communicate usage terms to OpenAI’s systems. Whether these are honored is vendor-dependent and evolving, but detecting their presence — or absence where you intended them — is part of a thorough access audit, because a directive you believe is set but is not actually served gives you false assurance about how your content is being used.
Selective blocking and the consequences you may not intend
Many sites do not block AI crawlers outright — they block them selectively, and the second-order effects are easy to miss. A common pattern is allowing GPTBot for browsing while disallowing it from a paywalled or members-only path, which is reasonable. The trouble starts when the path rules are broader than intended, or when a bot you meant to allow for retrieval is caught by a training-focused block you wrote for a different purpose.
The distinction that matters is between training and retrieval. Blocking a training crawler keeps your content out of future model weights but does nothing to retrieval-time citation; blocking a retrieval crawler like PerplexityBot removes you from live, cited answers even though your content may already exist in a model’s trained knowledge. Decide deliberately which of the two you are restricting, per bot, because a blanket Disallow: / collapses both and a narrow path rule may collapse neither. Our audit reports each bot’s training-versus-retrieval exposure separately so the consequence of every rule is explicit, never inferred.
There is also a response-code dimension that pure robots.txt analysis misses. A bot can be allowed in robots.txt and still be turned away at the edge — a WAF rule or rate limiter returning 403 or 429 to AI user-agents blocks access just as effectively as a Disallow, and far more quietly. Auditing the actual response code each AI user-agent receives, not just the robots directive, is the only way to catch an access failure that lives in your firewall rather than your robots file.
The bottom line
AI crawler access is not one setting but a chain: live robots.txt rules per bot, rendered meta directives, response headers, and raw-HTML content availability. The most common failures are quiet — a wildcard block you never noticed, or content that simply does not exist until JavaScript runs. Audit each AI bot independently against the served state, get your answers into the raw HTML, and you give the systems behind ChatGPT, Claude, and Perplexity a page they can actually read and cite.