Large sitemap troubleshooting

Sitemap too large or contains too many URLs

A single sitemap or sitemap index may contain no more than 50,000 entries and may be no larger than 50 MB uncompressed. Split the inventory before either ceiling, then publish a sitemap index that lists the child files.

Count entries and bytes independently

The entry limit and file-size limit are separate. A sitemap with 49,000 very long URLs and rich image or alternate-language extensions can exceed 50 MB. A compact file can cross 50,000 entries while remaining far below the byte ceiling. Measure the final UTF-8 document after XML serialization and measure its uncompressed size even when the deployed response uses gzip.

For a URL sitemap, count url entries. For a sitemap index, count sitemap entries. Do not use the number of lines: minified XML may occupy one physical line, while formatted entries may span many lines.

Split before the hard maximum

Leave operational headroom instead of filling every child file to exactly 50,000 entries. A generator that appends new URLs can otherwise cross the limit between scheduled checks. A lower internal target also makes files quicker to generate, transfer, validate, and retry.

Useful stable partition boundaries include:

• content family, such as products, categories, articles, and documentation;
• locale or regional site section;
• bounded numeric or hash ranges for large inventories;
• calendar archives when publishing patterns naturally follow time;
• separate image, video, or news sitemap workflows where the search engine supports them.

Avoid repartitioning the entire inventory whenever one URL is added. Stable child membership produces clearer diffs, cache behavior, logs, and Search Console reporting.

Create a sitemap index

The index gives crawlers one submission URL while the content remains distributed across child files:

The index itself has the same 50,000-entry and 50 MB uncompressed limits. Its optional lastmod refers to the child sitemap file. Update that value when the child content changes rather than rewriting every entry on every run.

Compression helps transfer size, not protocol size

Gzip can reduce bandwidth and storage, and child locations may point to .xml.gz files. Search engines enforce the 50 MB ceiling after decompression. Record both compressed and uncompressed byte counts in generator logs so a small archive does not conceal an oversized XML document.

Detect truncation and partial output

An oversized generation job may time out or exhaust memory before it writes a closing urlset tag. That failure appears as malformed XML rather than a clean size-limit error. Compare the source inventory count, serialized entry count, and published entry count. A file ending mid-entry, a missing final partition, or an index that lists only the files completed before a timeout all indicate a pipeline failure.

Write each child to a temporary or versioned object, close and validate it, then make it public. Publish the new index last. This sequence keeps crawlers on the previous complete generation until every new child is ready.

Filter the inventory before splitting it

Large sitemaps often contain URLs that do not belong in any partition: tracking variants, internal search results, redirects, duplicates, noncanonical pages, staging hosts, or pages intentionally excluded from indexing. Removing those entries improves the sitemap signal and may reduce the inventory enough to simplify partitioning.

Submitting a URL in a sitemap does not guarantee crawling or indexing. Search engines treat the file as a discovery and canonicalization signal. Use Search Console's page indexing reports to evaluate outcomes after the structural size problem is resolved.

Test the generator at its boundaries

1. Generate fixtures just below and just above your internal entry target.
2. Measure serialized UTF-8 bytes before compression.
3. Confirm every eligible URL appears exactly once across child files.
4. Confirm the index lists every expected child exactly once.
5. Validate each document family and sample the first and last partition.
6. Deploy, fetch, and decompress representative child files.

When a job stops midway, publish atomically so the public index never points at a partial generation. Upload complete child files first and replace the index only after every referenced child is available.

Keep useful generation metrics

A small set of aggregate measurements makes future failures much easier to diagnose:

• eligible source records and excluded records by reason;
• child file count, entry count, UTF-8 bytes, and compressed bytes;
• duplicate canonical URLs removed before serialization;
• generation duration and the identifier of the published version;
• expected child files compared with the final index entries.

Do not log entire customer or private URL inventories merely to obtain these counts. Aggregate metrics and a bounded sample of internal record identifiers are usually enough to trace a generator defect.

Choose the submitted entry point

Keep the public index URL stable while its children evolve. Search Console can then retain one submission history, and robots.txt can reference one durable location. Child filenames may be stable and overwritten atomically, or versioned and replaced through a newly generated index. Whichever model you choose, remove references to retired children only after the replacement files are publicly available.

Protocol references

The Sitemaps protocol defines the 50,000-entry and 50 MB limits for sitemap and index files. Google's large-sitemap guidance explains splitting files and submitting an index.