AI crawlers are indexing your site
right now. Do you know what they see?
GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are all crawling the web. Your robots.txt, noai directives, and llms.txt file determine whether your content surfaces in AI-generated answers — or gets excluded entirely.
What the AI Visibility audit covers
Access rules for AI crawlers can be set in multiple places — robots.txt, meta tags, HTTP headers, and the newer llms.txt standard. VisibilityIQ checks all of them and scores the result.
robots.txt AI-Crawler Rules
Parses your robots.txt for Disallow rules targeting GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot, and CCBot. Detects overly broad wildcard rules that accidentally block AI crawlers when that wasn't the intent.
llms.txt Validation
Checks for llms.txt at the expected root path, validates the format against the emerging standard, and scores completeness — title, description, key sections, and link structure. A well-formed llms.txt improves how AI systems understand your site.
noai / noimageai Directives
Detects noai and noimageai directives in HTML meta tags and X-Robots-Tag HTTP headers. Flags pages where these directives are present so you can confirm they reflect intentional policy rather than accidental exclusion.
X-Robots-Tag HTTP Headers
Checks HTTP response headers for AI-specific directives that aren't visible in the HTML source — a common source of unintentional AI-crawler blocking on pages served through reverse proxies or edge workers.
Per-Crawler Accessibility Score
Produces an access score for each major AI crawler — blocked, partially allowed, or fully accessible — across your crawled URL set. Sorted by crawler importance so you know which exclusions matter most.
Structured Data for AI Grounding
Validates that structured data (JSON-LD, schema.org types) survives JavaScript rendering — because AI systems increasingly use structured data as a reliable signal when grounding answers from your content.
AI crawlers audited
| Crawler | Owner | Use case |
|---|---|---|
| GPTBot | OpenAI | Used to train and ground ChatGPT and GPT model responses. |
| ClaudeBot | Anthropic | Anthropic's training and grounding crawler for Claude models. |
| PerplexityBot | Perplexity AI | Powers real-time web answers in Perplexity search. |
| Google-Extended | Google's AI training crawler, separate from Googlebot. | |
| Bingbot/AI | Microsoft | Microsoft Copilot and Bing Chat grounding crawler. |
| CCBot | Common Crawl | Widely used training data source for open models. |
Why competitors charge $99–$699/mo for this
AI Visibility is a relatively new category. Legacy SEO platforms built their businesses on keyword databases and backlink indexes assembled over many years — expensive to build and maintain. When AI-visibility auditing became important, they added it as a premium add-on to protect existing revenue rather than include it at the base tier.
VisibilityIQ was built after AI-crawler access became standard operating procedure for technical SEO. There was no legacy revenue model to protect, so including AI Visibility in the base subscription was the only logical choice. It's not a loss leader — it's a reflection of what technical SEO actually is in 2025.
AI Visibility Dashboard
Per-crawler access score matrix with llms.txt validation status, noai directive inventory, and robots.txt rule parser output — all on a single screen.
Know your AI-crawler access score today.
Included with Site Audit, Rank Tracking, and Competitor Intelligence at $59.95/mo.
Start your audit — $59.95/mo