XML Sitemaps vs Reality: Reconciling What You Declare With What You Link

A site with 40,000 pages submitted a sitemap every week, watched coverage numbers in Google Search Console stay flat for months, then traced the problem to 6,000 URLs in the sitemap that redirected to canonicals elsewhere. Google was filing them under “Discovered — currently not indexed” and moving on. The sitemap was busy, accurate in structure, and largely useless for those URLs.

Sitemaps are widely misunderstood as an indexation lever. They are not. They are a discovery mechanism — a way to tell a crawler where to look, especially for pages with weak internal link equity. Understanding that distinction, and then reconciling what your sitemap declares against what your site actually links and what Google actually indexes, is the foundation of a functioning crawl strategy.

What a Sitemap Actually Does (and Doesn’t Do)

The Sitemaps Protocol defines a structured way to enumerate URLs and optionally annotate them with <lastmod>, <changefreq>, and <priority>. Google is transparent about its treatment of these: <lastmod> is used when it is trustworthy; <changefreq> and <priority> are ignored entirely. What the sitemap actually delivers is a crawl hint — a list of URLs that Googlebot might otherwise miss if its link graph is sparse.

The critical corollary is that sitemap inclusion does not override Googlebot’s quality and canonicalization judgments. A page can be in the sitemap, return a 200 status, and still end up in “Discovered — currently not indexed” because it is thin, because it duplicates another page, or because Googlebot is rationing its crawl budget on a large site. The sitemap got the URL on the radar; it cannot force a pass.

What sitemaps genuinely help with: large sites where the internal link graph is too shallow to guarantee discovery of all valuable pages, new content that has no inbound links yet, and pages buried deep in pagination or faceted navigation that a crawl starting from the homepage would take days to reach.

The Three URL Sets That Should Agree

Every XML sitemap audit bottoms out in comparing three sets of URLs against each other.

Set 1: Sitemap-declared URLs. Every URL across all sitemap files submitted to Search Console, resolved and normalized. These are the URLs you are asserting deserve crawl attention.

Set 2: Internally-linked crawlable URLs. Every URL discoverable by following links from your homepage, respecting robots.txt and status codes. This is what Googlebot’s link-following behavior would find independently of your sitemap.

Set 3: GSC-indexed and GSC-known URLs. URLs Google has confirmed as indexed (the “Valid” bucket in GSC coverage) or at minimum encountered (any coverage bucket). This is ground truth about what Google knows.

Each pairwise gap between these sets is a different class of problem.

Sitemap-Only URLs (Not in the Internal Link Graph)

A URL in the sitemap but not reachable by following internal links is functionally an orphan. It may be accessible at its direct URL, but no page on your site points to it. Googlebot finds it from the sitemap, but it earns near-zero internal PageRank and has no contextual signal from anchor text or surrounding topical content.

These pages tend to get crawled infrequently after the initial discovery pass. If they subsequently fall out of the sitemap update cycle, they may drift into “Crawled — currently not indexed” for extended periods. The fix is almost always to add relevant internal links from hubs or related content, not to remove the URL from the sitemap. The sitemap is papering over a structural problem. See orphan pages: when good content has no way in for the full diagnostic.

Crawled-but-Not-in-Sitemap URLs

These are pages Googlebot finds through your link graph but that do not appear in any submitted sitemap. Often this is intentional: category pages, author pages, thin filter combinations. But frequently it is accidental — a redesign dropped a section from the sitemap template, a CMS migration regenerated slugs and the sitemap wasn’t updated, or a content type was excluded by misconfigured sitemap generation logic.

The decision for each is: does this URL deserve indexation? If yes, it belongs in the sitemap. If no, it should either have a noindex directive or redirect to a page that should be indexed. Silence — crawlable but undeclared — leaves it in an ambiguous state where Google must decide on its own whether the page merits attention.

Indexed-but-Not-in-Sitemap URLs

When Google has indexed a URL that does not appear in your submitted sitemaps, you have lost the lastmod recrawl signal for that URL. It will still get recrawled based on its internal link strength and historical crawl frequency, but you cannot nudge a recrawl by updating the sitemap. For actively updated pages — product listings, news articles, event pages — this is a meaningful gap. For stable evergreen content it matters less.

Sitemap URLs That Are Broken, Redirecting, or Noindexed

This is the most immediately damaging category. A sitemap that declares URLs returning 404, 301/302, or carrying noindex directives wastes crawl attention and signals poor sitemap hygiene.

404 URLs in the sitemap are a clear signal to remove or fix them. If the content moved, update the URL in the sitemap to the destination and add a redirect from the old path. If the content is gone permanently, remove from the sitemap.

Redirecting URLs in the sitemap should be replaced with the canonical destination. Declaring redirect chains in a sitemap is particularly bad — you are spending crawl budget on intermediate hops that should be resolved internally.

Noindexed URLs in the sitemap are contradictory declarations. You are simultaneously telling Google “here is a URL worth crawling” and “but don’t index it.” Google will generally honor the noindex and de-index the page, but the conflicting signal wastes crawl quota and clutters the coverage report with “Submitted URL has ‘noindex’ tag” errors. Keep the sitemap scoped to indexable content only.

The lastmod Honesty Problem

The lastmod field is one of the few sitemap fields Google actually uses. When it trusts it, a changed lastmod timestamp can trigger an expedited recrawl for high-frequency updated content — news, product pages, events. When it does not trust it, it ignores it entirely.

The most common way sites destroy lastmod credibility: a deployment pipeline or CMS that touches the lastmod for every URL every time any URL changes. This produces a sitemap where the homepage, a ten-year-old blog post, and a freshly published article all carry today’s date. Google’s crawlers notice within a few weeks that lastmod changes do not predict meaningful content changes, and they stop treating it as a recrawl signal.

Trustworthy lastmod means: use the actual Last-Modified HTTP header from your server or the CMS-recorded modified timestamp for the specific page’s content, not a deployment date, not a sitemap generation date, and not the same timestamp for every URL. If you cannot generate accurate per-URL timestamps, omit lastmod entirely. An absent lastmod is neutral; a dishonest one is actively counterproductive.

Sitemap Index Files and Extension Sitemaps

A single sitemap file is capped at 50,000 URLs and 50 MB uncompressed. Past that threshold you need a sitemap index file — an XML file that lists the individual shard sitemaps — submitted to Search Console. The index file itself does not contain page URLs; it contains <sitemap> entries pointing to the shards.

A common failure mode: a site grows past the 50k threshold, the sitemap generation script silently truncates rather than sharding, and the last several thousand URLs in alphabetical or ID order simply disappear from the declared set. If your organic URL count is significantly higher than your sitemap URL count and you have not deliberately excluded content types, count the URLs in your sitemap files and verify the total against your CMS.

Extension sitemaps — news, image, video — add domain-specific signals alongside the standard URL entry. A news sitemap includes publication dates and titles that the News Search index uses independently of Google Web Search. An image sitemap declares image URLs that may not be in src attributes (lazy-loaded images, CSS backgrounds). These are additive; they do not replace the standard <url> sitemap entry for the page they extend.

Reconciling Against GSC Coverage

The GSC Sitemaps report shows, per submitted sitemap, how many URLs were discovered, how many are indexed, and which ones fall into error buckets. The coverage report disaggregates by status: Indexed, Excluded (with sub-reasons), and Errors. Cross-referencing these is the manual version of what a systematic reconciliation requires.

Specific GSC coverage statuses to resolve by priority:

“Submitted URL returns fetch error” — the crawler could not retrieve the URL. Could be a server error, a timeout, or an access restriction. Urgent.

“Submitted URL has crawl issue” — broad category covering connection timeouts, redirect loops, or redirects that eventually 404.

“Submitted URL not selected as canonical” — Google has chosen a different canonical than the one the URL declares. The page may or may not be indexed under the chosen canonical. This is a canonicalization problem, not a sitemap problem, but the sitemap is where you first see the symptom.

“Submitted URL seems to be a Soft 404” — Google judges the page as returning thin or error-like content despite a 200 status. Common with out-of-stock product pages, empty filtered results pages, or pages that render a “no results found” state in their body.

VisibilityIQ performs this reconciliation automatically — crawling the site to map the internal link graph, fetching every declared sitemap URL, checking status codes and canonical declarations, and cross-referencing against GSC coverage data. Each discrepancy class surfaces as a distinct finding with structured evidence rather than a raw URL list. The distinction matters: knowing that you have 400 sitemap-declared redirects is an alert; knowing which ones are redirect chains versus single-hop canonicalizable patterns tells you what the fix actually looks like.

A Practical Audit Sequence

Run these steps against your largest or most strategically important site section first, then extend.

Fetch all URLs from every submitted sitemap, resolve to their final status codes, and segment by 200 / 3xx / 4xx / 5xx. Any non-200 URL is a removal or fix candidate.

For the 200 URLs, check canonical declarations — do the <link rel="canonical"> tags in the response match the sitemap URL? Mismatches indicate either a canonicalization problem or a sitemap that has not been updated to reflect a slug change.

For the 200 canonical-consistent URLs, check noindex directives in both the <meta robots> tag and the X-Robots-Tag response header. A noindex URL should not be in the sitemap regardless of whether it is the canonical.

Diff the resulting clean set against what your internal link graph can discover from the homepage. Pages in the clean sitemap set but not reachable internally need internal link authority attention. Pages reachable internally but absent from the sitemap need sitemap declaration or a deliberate exclusion decision.

Finally, overlay the GSC indexed set. The gap between sitemap-declared-and-crawlable versus GSC-indexed reveals where Google is choosing not to index despite having found the content. That gap warrants separate investigation — crawl budget pressure on large sites, duplicate content signals, or quality assessment failures — but the reconciliation itself is what makes the gap visible.

For sites managing thousands or tens of thousands of URLs, bulk URL analysis at scale covers the tooling patterns needed to run this without doing it row by row. And if the reconciliation surfaces a significant number of accidental noindex conditions — noindexed pages in the sitemap, or pages that picked up noindex in a template change — the discovery matters because those conditions are invisible without systematically fetching and reading the HTTP response for every declared URL.

The sitemap is a statement about what your site considers indexable. When that statement disagrees with your link structure or your server’s responses, something is wrong with one of the three — and each type of disagreement points to a different fix.