robots.txt: The One File That Can Erase Your Site From Search
In 2013, Robots.txt became internet history’s most expensive configuration mistake when a developer at a major retailer tested a staging robots.txt with Disallow: / and accidentally pushed it to production. Within 48 hours, every product page, category page, and homepage variant was being dropped from Google’s index. The recovery took weeks. The revenue impact was in the tens of millions.
That specific failure has been documented and re-documented, and yet variants of it happen continuously — not from carelessness but from misunderstanding what robots.txt actually does, how the match rules work, and how the relationship between crawl access and indexation is more nuanced than a simple “blocked means hidden” mental model.
What robots.txt Actually Controls
The Robots Exclusion Protocol has one job: telling crawlers which paths they are and are not permitted to fetch. It controls crawl access, not visibility in search results.
This distinction is not semantic. A URL blocked by robots.txt can and will appear in Google’s search results if Google knows about the URL through external links, a sitemap, or prior crawl history. Google can see that the URL exists without being able to read what is on it. The result: the URL appears in search with no snippet — just the URL and sometimes a description from a Dmoz listing or anchor text from external links. Google explicitly documents this behavior.
The corollary is that Disallow is the wrong tool for removing pages from search results. The correct mechanism for deindexation is a noindex directive — in the <meta name="robots"> tag or the X-Robots-Tag HTTP response header — and that directive requires the page to be crawlable in the first place. You cannot use robots.txt to block a page and then add noindex to the blocked page’s HTML: the crawler cannot read the noindex because it cannot access the page. Blocking a page from crawlers while also wanting it removed from the index requires first allowing it to be crawled, serving the noindex, and then waiting for the next crawl cycle to process the directive. Only after confirmed deindexation should you optionally block crawl access again to conserve quota.
The practical upshot: if a page is showing up in search results and you want it gone, robots.txt is not the fix. If a page is getting crawled and you want to reduce crawl pressure on a path that should never be indexed, robots.txt can help — but only after the noindex is confirmed effective.
How the Match Rules Work
The robots.txt specification uses two directives under each User-agent: group: Disallow: and Allow:. Google applies these with a longest-match-first rule within the same user-agent group.
Given:
User-agent: Googlebot
Disallow: /products/
Allow: /products/featured//products/sale/item matches Disallow: /products/ (11 characters). /products/featured/item matches both Disallow: /products/ and Allow: /products/featured/. The longer match wins: /products/featured/ has 19 characters, so the Allow directive takes precedence and the URL is accessible to Googlebot.
The wildcard character * in a User-agent line matches any user-agent not explicitly named in another group. A User-agent: * group applies to every crawler that does not have its own named group. If you have a User-agent: Googlebot group and a User-agent: * group, Googlebot follows only its named group — the wildcard group does not apply to it at all. This is a common source of unexpected behavior: you add an Allow in the wildcard group thinking it overrides a Disallow, but the crawler you are trying to affect is reading a different user-agent group entirely.
Path matching is case-sensitive. Disallow: /Products/ does not block /products/ on a case-sensitive server. On most Linux servers, paths are case-sensitive, so this matters. On Cloudflare Workers and many modern hosting setups, the routing layer may or may not normalize case before the robots.txt check runs.
The Catastrophic Mistakes
Disallow: / in production. The classic. Usually arrives via a robots.txt intended for a staging or development environment that gets promoted to production. The mitigation is a deployment pipeline that always verifies the robots.txt content in production does not contain Disallow: / before marking a deploy complete. Automated checks for this take under a second and can prevent a multi-day recovery.
Blocking /wp-content/, /static/, /assets/, or /js/ paths. This one is less dramatic but more insidious. WordPress sites frequently see a Disallow: /wp-content/plugins/ or Disallow: /wp-content/uploads/ intended to prevent direct indexation of upload files. The problem is that many themes and plugins serve CSS and JavaScript from these paths. Googlebot, unable to access those resources, renders pages without their stylesheets, executes JavaScript in a degraded mode, and sees a partial version of the page. The result: Google thinks the page is broken or thin, rank drops, and the connection to the robots.txt line takes days or weeks to trace.
Googlebot needs to fetch every resource required to render a page. If a font, a script, or a stylesheet is blocked, the rendered page Google sees is different from what users see. This is a render parity failure caused by a robots.txt mistake. Raw-vs-rendered parity is one of the harder problems to diagnose because the symptom (rankings drop, crawl anomalies) looks unrelated to the cause (a two-line change in robots.txt).
Blocking parameterized URLs that hold canonical content. Sites with session identifiers, tracking parameters, or faceted navigation in their URL structure sometimes add blanket Disallow: /*? to block all parameterized URLs. If your canonical content lives at parameterized URLs — if your product filter system uses query strings to define the canonical variant of a product page rather than clean paths — this directive blocks the content entirely. Only add parameter-based disallows if you have confirmed the clean-URL canonical exists and is the version you want indexed.
Putting noindex in robots.txt. Google dropped support for this in September 2019. Any line reading Noindex: /path in robots.txt is silently ignored by Googlebot. This is documented, but it continues to appear in production robots.txt files — usually because someone added it years ago when Google did support it, and no one has audited the file since. Its presence creates false confidence that content is protected when it is not.
Case mismatch in path patterns. A common mistake on sites that recently migrated from case-insensitive Windows hosting to Linux hosting: paths that were previously case-agnostic are now case-sensitive, but the robots.txt was written assuming the old behavior. Auditing after a hosting change should always include verifying robots.txt path matching against the actual server’s case sensitivity.
AI Bots Are a Separate Conversation
Until relatively recently, robots.txt audits could focus exclusively on Googlebot and Bingbot. That has changed. AI crawlers from OpenAI (GPTBot, OAI-SearchBot), Anthropic (ClaudeBot), Perplexity (PerplexityBot), and Google’s own AI training crawler (Google-Extended) each respect robots.txt independently in their own User-agent: groups.
This creates a decision surface that did not exist three years ago. A site might intentionally allow Googlebot while blocking GPTBot and ClaudeBot from the same content. Or it might have a User-agent: * group with a global Disallow: /content/ that inadvertently blocks all AI crawlers from content the site owner would actually want AI systems to read and potentially cite.
The key failure patterns here:
Accidental AI blocking via wildcard groups. A permissive wildcard group covers AI crawlers by default. But a restrictive wildcard group — User-agent: * / Disallow: /blog/ — blocks all AI crawlers from blog content simultaneously, even if you want them to be able to read and extract information from that content.
Intentional AI blocking for training, with unintended search visibility consequences. A site may legitimately want to block AI training crawlers (GPTBot in training mode, Google-Extended) without blocking AI answer crawlers (OAI-SearchBot, the inference-time crawlers used when someone asks ChatGPT a question and it searches the web). These have separate user-agent identifiers and can be controlled independently. Many sites block GPTBot (training), not realizing this has no effect on OAI-SearchBot (search/retrieval). Auditing AI crawler access requires looking at each user-agent’s group separately.
Crawl-delay directives for AI crawlers. The Crawl-delay: directive is not supported by Google, but several AI crawlers respect it. If you want AI crawlers to pace themselves to avoid server load impact, a Crawl-delay: 10 in the GPTBot group slows their access rate. This is a traffic management tool, not a blocking mechanism, but it is worth knowing it works for some crawlers.
For teams concerned about AI visibility and citation eligibility, how you configure AI crawler access in robots.txt is now a first-order decision alongside your llms.txt setup. AI visibility in ChatGPT and Perplexity covers the access layer in the broader context of how AI systems decide what to cite.
Testing Your robots.txt
The GSC robots.txt tester remains the most practical tool for verifying how Google specifically interprets a given URL against your robots.txt file. It shows whether a URL is blocked or allowed, which rule matched, and which user-agent group applied. Test against both Googlebot and Googlebot-Image if you serve image-heavy content, since they can be addressed independently.
For AI bot testing, there is no equivalent Google-provided tool. Manual inspection of the match rules — running through the same longest-match algorithm described above for GPTBot, ClaudeBot, and PerplexityBot groups — is the current option.
A frequent miss: testing only the paths you believe are blocked without testing resources those paths depend on. Testing /products/ is not sufficient if /static/js/app.bundle.js and /cdn-cgi/scripts/ are also blocked. Every URL the page fetches to render should be checked, not just the page URL itself.
VisibilityIQ parses robots.txt against every user-agent group independently and flags important URLs and resources that are blocked for search crawlers or AI bots. The check includes resources — scripts, stylesheets, fonts — linked from the pages being audited, so a blocked CSS file shows up as a finding against the pages that depend on it, with the specific robots.txt rule that triggered the block.
Keeping robots.txt From Going Stale
Robots.txt files accrete directives over time. A path added to handle a now-resolved crawl issue three years ago may block content that has since become important. A staging environment disallow may have been removed from production but remain in the live file because no one was tracking it. A new content section may have been added without anyone checking whether the wildcard group’s existing Disallow patterns affect it.
A practical maintenance approach: version-control the robots.txt file, review it as part of any deployment that changes URL structure, and schedule a quarterly audit against the live site’s URL topology. The audit should specifically ask: does any current Disallow pattern match URLs that are now expected to be indexed?
For sites undergoing accidental noindex situations, the robots.txt check frequently surfaces alongside the meta robots and X-Robots-Tag checks — different mechanisms, same outcome, and all three need to be confirmed clear for a URL to have a clean indexation path. The overlap is worth looking at together rather than auditing each directive in isolation.
Finally, from a pure risk surface perspective: robots.txt lives at the site root, is fetched without authentication, and is readable by every crawler on the internet. Its contents are public. Do not put information in robots.txt comments that you would not want indexed in web archive databases. Comment out-of-date paths for context if you need to, but treat the file as publicly logged. The disavow-level panic that follows a Disallow: / going live in production is the worst outcome; the second worst is realizing a robots.txt comment has been leaking your internal URL structure for three years.