The Ultimate Guide to XML Sitemaps

XML Sitemaps: What They Do and What They Do Not Do

An XML sitemap is a machine-readable list of the pages and files you want search engines to know about. It can help them discover preferred URLs and understand useful details such as when a page was last meaningfully updated. It is especially helpful on large, new, frequently changing, media-heavy, or structurally complex websites.

A sitemap is not an instruction that forces a search engine to crawl or index every listed URL. It also does not make a page rank simply because the URL appears in the file. Google describes sitemap submission as a hint, so the practical goal is to publish a clean, automatically maintained list of canonical URLs and then investigate any gap between what you submitted and what search engines actually crawl or index. See Google Search Central’s overview of sitemaps for the underlying guidance.

A sitemap helps explain which URLs you want discovered; it does not confirm indexing or an AI citation. Keep the sitemap, internal links and intended public pages consistent. The AI search readiness guide puts this technical work in context.

XML Sitemaps in 60 Seconds

• Use one when it reduces discovery risk: Google says a small, comprehensively linked site of about 500 pages or fewer may not strictly need a sitemap, although many content management systems create one automatically.
• List preferred URLs: Include fully qualified, canonical URLs that you genuinely want to appear in search.
• Exclude conflicting URLs: Redirects, errors, noindex pages, robots-blocked pages, duplicate URL variants, internal search results, and private utility pages normally do not belong.
• Use <lastmod> honestly: Update it only after a meaningful page change. Google ignores <priority> and <changefreq>.
• Respect the limits: One sitemap can contain up to 50,000 URLs and be up to 50 MB uncompressed. Use a sitemap index when you exceed either limit or when grouping helps analysis.
• Submit and monitor: Add the sitemap to Search Console, reference it in robots.txt, and submit it to Bing Webmaster Tools where relevant.
• Interpret “Success” correctly: It means Google fetched and read the file; it does not mean every submitted URL is indexed.

When an XML Sitemap Is Worth Prioritising

A sitemap is most valuable when relying on links alone creates a realistic chance that important URLs will be discovered late or monitored poorly.

• Large or complex sites: Ecommerce catalogues, marketplaces, publishers, and sites with deep sections can have important pages several clicks from the homepage.
• New sites with few external links: Search engines have fewer established paths through which to discover new URLs.
• Frequently changing content: Accurate <lastmod> values can help Google schedule recrawls for URLs it already knows.
• Images, video, news, or international content: Sitemap extensions can provide additional information for specialised content and alternate-language pages.
• Segmented diagnosis: Separate product, category, article, or regional sitemaps make it easier to compare submitted groups with indexing outcomes.

A sitemap should supplement, not replace, a clear internal site structure. It cannot repair inaccessible navigation, remove a noindex directive, override a conflicting canonical tag, fix server errors, or guarantee that weak pages will be indexed.

Which URLs Belong in an XML Sitemap?

Use this five-part inclusion test. A normal page URL belongs in your sitemap when all five statements are true:

1. You want the page to be eligible to appear in search results.
2. The URL is the preferred canonical version of that content.
3. The page returns an HTTP 200 (successful) response to search-engine crawlers rather than redirecting or failing.
4. The page is crawlable and does not carry a noindex directive.
5. The URL is public, stable, and generated from a reliable source of truth.

Usually include

• The homepage and indexable service, product, category, location, article, and other landing pages.
• Indexable paginated URLs when each page is deliberately canonical to itself and useful in search.
• Canonical alternate-language pages, with hreflang annotations added through the supported sitemap extension when that is your chosen implementation.
• Image, video, or news metadata only when the content qualifies and the relevant extension is maintained correctly.

Usually exclude

• Redirecting URLs, 404s, soft 404s, and server-error URLs.
• Pages marked noindex or blocked from crawling by robots.txt.
• Duplicate, parameterised, filtered, sorted, session, print, or tracking variants unless a variant is intentionally canonical and indexable.
• Internal search results, account areas, carts, checkout steps, staging URLs, and other private or low-value utility pages.
• Expired or deleted content unless the URL still serves a deliberate, indexable replacement page.

Leaving a URL out of the sitemap does not prevent indexing. A sitemap is a discovery and reporting input, not an exclusion control.

How to Create, Submit, and Verify an XML Sitemap

1. Find out what already exists

   Check common locations such as /sitemap.xml and /sitemap_index.xml, inspect the Sitemap: lines in robots.txt, and review your CMS or ecommerce platform settings. Avoid adding a second generator before you understand the current one.

2. Define the eligible URL set

   Start from your CMS database, product catalogue, route registry, or another reliable source. Compare that list with a crawl to find redirects, errors, canonical mismatches, blocked pages, and accidental parameter URLs. Correct the generator’s rules rather than repeatedly hand-editing the output.

3. Generate the sitemap automatically where practical

   For more than a few dozen URLs, automatic generation is usually safer than maintaining XML by hand. Configure the generator to add new eligible pages, remove deleted or redirected pages, and update modification dates only when the underlying page materially changes.

4. Use valid XML and absolute URLs

   The file should be UTF-8 encoded, use the sitemap namespace, and contain complete URLs including the protocol and hostname. XML-sensitive characters must be escaped. A minimal sitemap looks like this:

   <?xml version="1.0" encoding="UTF-8"?>
   <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
     <url>
       <loc>https://www.example.com/services/technical-seo-audit</loc>
       <lastmod>2026-08-18</lastmod>
     </url>
   </urlset>

   Google’s guide to building and submitting a sitemap covers supported formats, placement, canonical URL selection, and submission methods.

5. Use <lastmod> only when it reflects reality

   The value should represent the last significant page update, such as a meaningful content, structured-data, or link change. Do not rewrite every timestamp daily or after a copyright-year change. Google can disregard unreliable dates. You can normally omit <priority> and <changefreq> because Google ignores both values.

6. Split large or operationally useful groups

   A single sitemap is limited to 50,000 URLs or 50 MB uncompressed. Split larger sets into child sitemaps and reference them from a sitemap index. You may gzip a sitemap to reduce transfer size, but the 50 MB limit still applies after decompression. Even below the hard limit, grouping by content type can make diagnosis easier, provided each group remains meaningful and maintainable.

   <?xml version="1.0" encoding="UTF-8"?>
   <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
     <sitemap>
       <loc>https://www.example.com/sitemaps/products.xml.gz</loc>
       <lastmod>2026-08-18</lastmod>
     </sitemap>
     <sitemap>
       <loc>https://www.example.com/sitemaps/articles.xml.gz</loc>
       <lastmod>2026-08-17</lastmod>
     </sitemap>
   </sitemapindex>

7. Publish the file where crawlers can fetch it

   For a standard single-site setup, the site root is the simplest location. The sitemap URL should be public, stable, and fetchable without a login. Check that it does not return a 404, server error, or unexpected HTML page, and make sure robots.txt does not block the sitemap itself.

8. Tell search engines where it is

   Submit the sitemap or sitemap index through Google Search Console and, where relevant, Bing Webmaster Tools. Add an absolute sitemap line to robots.txt:

   Sitemap: https://www.example.com/sitemap.xml

   Do not rely on Google’s old unauthenticated sitemap ping URL. Google retired the sitemap ping endpoint, and requests to it return a 404 response.

9. Verify the file and sample the URLs

   Use the Search Console Sitemaps report to confirm that Google fetched and parsed the file, then inspect representative URLs from each sitemap group. Use the Page indexing report and URL Inspection to investigate exclusions rather than assuming the sitemap caused them.

10. Maintain the generator, not just the file

    Recheck the sitemap after migrations, URL changes, product retirement, CMS updates, and indexability changes. A healthy process removes obsolete URLs automatically and keeps the submitted inventory aligned with the pages your site actually wants indexed.

How to Troubleshoot Common Sitemap Problems

Search Console says “Couldn’t fetch”

Open the sitemap URL directly and confirm that the address is correct, publicly accessible, and returning the expected XML or supported format. Check for robots rules that block the sitemap, authentication, DNS or server failures, and temporary hosting problems.

The sitemap says “Success,” but URLs are not indexed

“Success” confirms that Google fetched and read the sitemap. It does not approve the pages. Inspect a sample of affected URLs for noindex, robots restrictions, redirects, canonicalisation to another URL, server errors, soft 404s, duplication, or other indexing reasons. A sitemap cannot override those signals.

Discovered URL counts look wrong

Open the submitted file and any child sitemaps rather than trusting the generator’s dashboard. Confirm that the file contains URL entries, that child sitemap locations are correct, and that you submitted it under the appropriate Search Console property. Processing and crawling are not instant, so judge the setup using the file itself, fetch status, and URL-level inspection together.

Old or unwanted URLs keep returning

Find the source feeding the generator. Common causes include CMS archives, faceted navigation, retired products, staging records, alternate URL fields, and plugins that generate overlapping sitemaps. Fixing that source is more reliable than deleting entries after every rebuild.

The sitemap and robots.txt disagree

Do not use the sitemap to advertise pages that your robots rules prevent crawlers from accessing. Review the rules carefully; the guide to robots.txt mistakes that block SEO growth explains the most common conflicts.

<lastmod> changes every day

Connect the value to the page’s real modification event, not to sitemap generation time. A site-wide daily timestamp removes useful distinction between genuinely updated and unchanged pages and can make the signal less trustworthy.

Worked Example: Cleaning an Ecommerce Sitemap

Consider a hypothetical ecommerce site with 3,200 crawlable URLs. The business actually wants 650 live products, 40 categories, and 120 advice articles in search. The remaining URLs are filter combinations, internal search pages, retired products that redirect, and tracking variants.

Putting all 3,200 URLs into one sitemap would hide the real inventory and mix valid pages with conflicting signals. A stronger setup would generate three child sitemaps—products, categories, and articles—from the store’s canonical records. Each entry would return a successful response, be indexable, and use an accurate modification date. The sitemap index would then let the team compare indexing patterns by page type instead of treating the whole site as one unexplained total.

This does not guarantee that all 810 preferred URLs will be indexed. It does make the submission cleaner and the diagnosis faster: a product-specific gap points the team toward product templates, availability rules, canonicals, or content quality rather than toward the article section.

What to Do Next

Open your current sitemap and inspect a representative sample rather than assuming the generator is correct. Check that the entries are canonical, indexable, successful URLs; verify that <lastmod> is meaningful; and remove recurring errors at the generator or CMS level. Then submit the sitemap, confirm that it can be fetched, and use URL-level indexing evidence to investigate any remaining gap.

The best sitemap is not the longest one. It is the smallest accurate representation of the pages you genuinely want search engines to discover and evaluate.

XML Sitemap Checklist and Authoritative Resources

Pre-submission checklist

• The sitemap or sitemap index is publicly accessible and returns the intended file.
• Every listed page URL is absolute, canonical, and intended for search.
• Redirects, errors, noindex pages, robots-blocked pages, duplicate variants, and private utilities are excluded.
• The XML is UTF-8 encoded, correctly namespaced, and entity escaped.
• <lastmod> reflects significant page changes rather than the time the sitemap was regenerated.
• <priority> and <changefreq> have not been added merely to influence Google.
• Each file stays within 50,000 URLs and 50 MB uncompressed.
• Large or diagnostically useful groups are organised through a valid sitemap index.
• The sitemap is referenced in robots.txt and submitted through the relevant webmaster tools.
• Search Console status and URL-level indexing evidence are reviewed separately.

Authoritative resources

• Google Search Central: sitemap overview and when you may need one
• Google Search Central: build and submit a sitemap
• Google Search Console: Sitemaps report and error meanings
• Sitemaps.org: XML sitemap protocol
• Bing Webmaster Tools: sitemap submission guidance