Ecommerce Crawl Budget: Myths, Diagnosis & Fix Matrix

Crawl budget is not a default explanation for ecommerce indexing problems. It becomes worth investigating when a large or fast-changing catalogue is not being crawled quickly enough, Google is spending substantial activity on low-value URL variants, or the site is hitting serving-capacity limits. This guide shows how to prove which problem you actually have before changing robots.txt, canonicals, filters, infrastructure or internal linking.

Written by Hansel McKoy Founder of ScanMySEO

Published10 August 2026 Updated30 September 2026 Reading time13 min read

For many ecommerce sites, “crawl budget” is the wrong first diagnosis. Google’s current guidance is aimed primarily at very large or rapidly changing sites, and at sites with a significant number of URLs stuck as Discovered – currently not indexed. Google gives rough examples of one million or more moderately changing pages, or 10,000 or more rapidly changing pages; it also stresses that these are estimates, not hard thresholds. If new products are being discovered and crawled promptly, a crawl-budget project may not be necessary at all. See Google’s current crawl-budget guidance.

Ecommerce can still create genuine crawl-efficiency problems because filters, sorts, pagination, search states, session parameters and product variants can multiply one catalogue into a much larger URL inventory. The right response is not “make Google crawl more.” It is to determine whether Google is short of crawl capacity, spending time on URLs you do not want processed, failing to discover important URLs, or crawling pages and then deciding not to index them. Those are different problems and need different fixes.

Start here: does your ecommerce site have a crawl-budget problem?

Use the symptoms below before making a crawl-control change. The priority labels are ScanMySEO triage judgements for investigation, not Google severity classifications.

What you observe	What it usually suggests	Crawl-budget priority	First evidence to check
New or changed products are crawled quickly and important catalogue URLs are indexed normally.	You probably do not have a crawl-budget problem.	Low	Page Indexing trends and a sample of recently launched products.
Large numbers of important URLs remain Discovered – currently not indexed for long periods.	Crawl scheduling or serving capacity may be part of the problem, especially on large or fast-changing sites.	High enough to investigate	Page Indexing report, Crawl Stats and server logs.
Googlebot spends substantial requests on filter, sort, tracking or session-style URL patterns that you do not want indexed.	URL inventory is larger than it needs to be.	High when volume is material	Server-log URL patterns and Google’s faceted-navigation guidance.
Host status problems, rising response times, repeated 5xx responses or 429 rate limiting appear during crawling.	Crawl capacity may be constrained by the site’s ability to serve requests.	High	Crawl Stats host status, response breakdown and infrastructure logs.
Google crawls the product pages, but they remain Crawled – currently not indexed.	This is not automatically a crawl-budget problem; canonicalisation, duplication, content usefulness or other indexing decisions may be more relevant.	Usually secondary	URL Inspection, canonical signals, duplication and page quality.
A modest site has some 404s but no discovery delay or host-capacity symptoms.	Fix broken internal links for users and discovery, but do not treat ordinary 404 responses as proof of crawl-budget waste.	Low	Internal-link source and whether the removed URLs should have replacements.

Google’s Page Indexing report is useful for understanding whether important URLs are being discovered, crawled and indexed. The Crawl Stats report then shows Google’s request history, response codes, download volume, average response time and host availability. For a large ecommerce site, server logs add something Search Console cannot: you can group actual crawler requests by URL pattern and see which sections consume activity.

Crawl-budget myths that cause bad ecommerce fixes

The most expensive crawl-budget mistake is acting on the wrong mechanism. Google describes crawl budget as the combination of crawl capacity limit and crawl demand: what Google can crawl without overloading the host, and what its systems want to crawl. That is very different from a fixed allowance that can simply be “spent” on the wrong page.

Myth	What current Google guidance says	Practical ecommerce implication
“Every ecommerce site needs crawl-budget optimisation.”	Google positions crawl-budget management as an advanced topic mainly for very large, very fast-changing or persistently undiscovered sites.	Do not redesign filtering or infrastructure just because the store has many products. Prove a crawl problem first.
“404 pages waste crawl budget.”	Google’s crawling myths documentation says ordinary 4xx responses, except 429, do not waste crawl budget. Google also recommends 404 or 410 for permanently removed URLs so they can drop out of active crawling over time.	Fix broken internal links and bad product-removal logic, but do not prioritise a 404 cleanup solely because each 404 is supposedly draining crawl budget. Soft 404s are a different problem and should be corrected.
“A canonical tag stops Google crawling filtered URLs.”	For faceted navigation, Google says canonicals may reduce crawling of non-canonical variants over time, but are generally less effective for crawl control than preventing crawl of unwanted facet URLs or using fragments where appropriate.	Use canonicalisation to express preferred duplicate URLs, not as your only defence against a near-infinite filter space.
“noindex is a crawl-budget control.”	Google has to crawl a URL to see a `noindex` rule, so it is not an immediate way to stop crawling.	Use `noindex` when the indexing outcome is what you need. Use long-term crawl controls for URLs that you genuinely do not want crawled.
“Faster front-end performance is the whole crawl-budget fix.”	Stable or improving response times can raise crawl capacity, while latency, 5xx errors and 429 responses can reduce it. But crawl demand still matters, and faster low-value pages do not automatically become more important to crawl.	Separate server/response problems from Core Web Vitals work. A performance project can be valuable without being the main crawl-budget fix.
“More crawling means better rankings.”	Crawling is required before a page can be indexed, but Google explicitly says crawl rate is not a ranking signal.	The goal is timely discovery and processing of important URLs, not maximising raw Googlebot request counts.

Google’s own myths and facts about crawling is a useful reference when a proposed fix depends on a claim such as “404s drain budget,” “clean URLs are required,” or “more crawling improves rankings.”

What crawl budget can affect — and what it cannot

A genuine crawl-efficiency problem can delay Google’s discovery or recrawling of important products and categories. On a catalogue that changes every day, that can mean new inventory, stock changes or revised category pages are processed later than you want. Faceted navigation can worsen the problem because URL parameters may create an effectively unbounded combination of filter states. Google warns that this can cause overcrawling and slower discovery of useful URLs; its faceted-navigation documentation is especially relevant to ecommerce implementations.

What crawl budget does not do is guarantee indexing or ranking. A page can be crawled and still not be indexed, and an indexed page can rank poorly for reasons unrelated to crawling. If Google is already fetching the product page, repeatedly asking “how do we get more crawl budget?” can distract from the actual issue.

There is also a user-side distinction. Broken category links, empty filter states, long redirect chains and sluggish infrastructure can all make a store harder to use, but their SEO mechanisms differ. Treating every technical defect as the same “crawl budget” problem makes prioritisation worse, not better.

Diagnostic workflow: prove the bottleneck before you fix it

Establish whether the site is in crawl-budget territory. Record approximate canonical/indexable URL count, how frequently important catalogue pages change, and whether important new URLs are being crawled promptly. Google’s rough scale examples are a screening tool, not an eligibility test.
Separate “discovered” from “crawled.” In the Page Indexing report, a persistent population of important URLs in Discovered – currently not indexed deserves different investigation from Crawled – currently not indexed. The second group has already crossed the crawl step.
Check Crawl Stats for capacity symptoms. Look at host status, average response time and response-code patterns. A material rise in 5xx errors, 429 responses or serving instability is more useful evidence than a vague belief that “Googlebot is crawling less.”
Measure URL-pattern demand. From server logs where available, group Googlebot requests into patterns such as canonical products, categories, pagination, sort parameters, filter parameters, internal search, carts, wishlists and tracking/session URLs. The question is not whether parameter URLs exist; it is whether large numbers of low-value combinations are being requested.
Check discovery paths for important products. A product that exists only in a sitemap but is several layers away from crawlable category navigation can have a discovery problem rather than a pure capacity problem. Google recommends crawlable ecommerce navigation from categories to subcategories and products. ScanMySEO’s site-structure guide covers the broader architecture side.
Check what the site is asking Google to process. Sitemaps should contain the canonical URLs you want in Search, not every filter or tracking variant. Accurate lastmod values can help communicate meaningful updates, but a sitemap is still a hint rather than a guarantee of crawling or indexing.

One extra nuance matters for multi-host ecommerce stacks: Google’s crawl-budget guidance treats a site as a unique hostname for this purpose. For example, www.example.com and shop.example.com are treated separately. Do not combine their crawl activity into one diagnosis without checking the host-level picture.

Prioritisation matrix: what to fix first

Do not score issues with invented 1–5 numbers unless you have your own measured business model behind them. A more defensible matrix is to prioritise by the observed failure mode, the affected URL set and the evidence that the fix addresses that failure.

Evidence	Likely problem	Recommended priority	Action	How to verify
Host-status problems, sustained latency, 5xx responses or 429 rate limiting coincide with reduced crawling.	Serving capacity is constraining crawl capacity.	Fix first when important URLs are affected.	Stabilise the origin/CDN path, remove avoidable bottlenecks and add server resources where justified.	Host status recovers, error/rate-limit responses fall and important URLs are crawled without serving failures.
Large request volume goes to filter/sort/session combinations that should not appear in Search.	Unwanted URL-space expansion.	High when it materially dominates crawling.	Reduce unnecessary URL generation and linking; for facet URLs that should not be crawled, design durable crawl controls based on the actual parameter patterns.	The unwanted patterns decline in logs/Crawl Stats while canonical product/category discovery remains healthy.
Important products are absent from navigational links or buried behind non-crawlable interactions.	Discovery/site architecture.	High when affected products matter commercially.	Create crawlable category-to-product paths and keep important canonical URLs in sitemaps.	Products become discoverable through crawlable links and move through discovery/crawling normally.
Long redirect chains, many soft 404s or repeated duplicate URLs are being requested.	Avoidable crawl work and URL inconsistency.	Medium to high depending on scale.	Link directly to final URLs, return genuine 404/410 responses for removed content, and consolidate duplicate URL variants.	Chains shorten, soft-404 reporting falls and crawler requests converge on intended URLs.
There are many legitimate 404s for permanently removed products, but no host or discovery problem.	Normal catalogue churn, possibly with broken internal references.	Not a crawl-budget emergency.	Keep correct 404/410 responses; repair internal links or redirect only where a genuine replacement exists.	Users and crawlers stop being sent to removed URLs from live navigation.
Product URLs are crawled but remain not indexed.	Indexing/canonical/content issue rather than lack of crawling.	Investigate separately.	Review canonical signals, duplication, rendering, content distinctiveness and whether the URL deserves to be indexed.	URL Inspection and Page Indexing show the intended canonical/indexing outcome after reprocessing.

Facets are where ecommerce crawl advice most often goes wrong. Start with one question: Should this filtered URL be eligible to appear in search results?

If no: reduce internal links that manufacture the URL space and, where appropriate, prevent long-term crawling of those patterns. Google specifically recommends robots.txt as one option when facet URLs do not need to be indexed, and says URL fragments can be another design choice because fragments generally are not used for crawling and indexing in Google Search.
If yes: make the URLs stable and useful. Google recommends consistent parameter separators and filter ordering, preventing duplicate filter combinations, and returning a genuine 404 when a filter combination has no results.
If you need an already indexed URL removed: remember that Google must crawl a page to see a noindex rule. Do not hide the page behind robots.txt and simultaneously expect Google to read a page-level noindex. Decide whether your immediate goal is de-indexing, long-term crawl prevention, or both, then sequence the controls accordingly.

Canonical tags still matter for duplicate URL consolidation, but they are not a guaranteed “do not crawl” instruction. For unwanted filter spaces, Google says canonicalisation may reduce crawling over time and is generally less effective than stronger crawl-prevention approaches described in its faceted-navigation guidance.

Robots rules can be powerful enough to hide valuable catalogue sections when a wildcard is too broad. Test the exact parameter patterns before deployment and keep the indexing/crawling distinction clear. For implementation pitfalls, see ScanMySEO’s robots.txt mistakes guide.

When performance really is a crawl-budget issue

Page speed is neither irrelevant nor the whole answer. Google says its crawl capacity limit can increase when the site responds consistently and response times remain stable or improve. It can decrease when latency grows, responses take longer, or the site returns server errors or rate-limiting signals such as HTTP 429.

That makes server response and availability the relevant evidence. A poor Largest Contentful Paint score on its own does not prove that Google cannot crawl enough catalogue pages. Conversely, a store with acceptable front-end metrics can still restrict crawl capacity if the origin struggles under crawler traffic or returns intermittent 5xx responses.

Google also recommends avoiding long redirect chains and using HTTP caching where appropriate; a valid 304 Not Modified response lets a crawler reuse a cached version when the resource has not changed. These are efficiency improvements, not ranking guarantees.

How to verify a crawl-budget fix

Do not judge success by a single total such as “Googlebot requests increased 20%.” Google’s demand changes, and blocking unwanted URLs does not guarantee that every freed request is reallocated elsewhere. The better test is whether the crawl pattern now matches your intended catalogue.

Record a before state. Capture Crawl Stats, host status, response-code distribution and the proportion of log requests going to key URL patterns.
Deploy one controlled change where possible. For example, fix one family of facet URLs or one redirect pattern before changing three unrelated systems at once.
Check that valuable crawling survived. Sample canonical category and product pages, especially newly launched or frequently changing inventory.
Check that unwanted crawling declined. Compare the same filter/sort/session patterns in logs or Crawl Stats examples.
Check indexing separately. Use Page Indexing and URL Inspection to confirm whether important pages are moving from discovery to crawl and then to the intended canonical/indexed state.
Watch for regressions after catalogue or platform releases. New merchandising filters, tracking parameters, client-side navigation and pagination changes can recreate URL-space problems even after a successful cleanup.

What not to do

Do not chase raw crawl volume. More requests are not a ranking signal.
Do not block valuable URLs to make a chart look cleaner. Crawl efficiency is useful only if important products and categories remain discoverable.
Do not use robots.txt as an indexing-removal tool. A disallowed URL can still be known to Google, and blocking crawl prevents Google from seeing page-level noindex.
Do not assume every parameter is bad. Query parameters are crawlable; the problem is uncontrolled or low-value URL multiplication, not the mere presence of ? in a URL.
Do not treat ordinary 404s and soft 404s as the same thing. Correct 404/410 responses are useful signals for removed pages; soft 404s should be fixed.
Do not rely on canonicals alone to control an infinite facet space. Canonicals are part of duplicate consolidation, while unwanted crawling needs a crawl-control strategy.

The decision rule

If Google is already crawling important ecommerce pages promptly, spend your effort on the problem you can actually prove rather than manufacturing a crawl-budget project. If important pages are delayed, identify whether the constraint is capacity, discovery or URL-inventory waste. Then change the smallest system that addresses that failure, verify that valuable URLs remain crawlable, and measure the before-and-after request pattern instead of relying on SEO folklore.

That is the practical purpose of crawl-budget work: not to maximise crawling, but to make it easier for Google to spend its limited crawling resources on the URLs your ecommerce site genuinely needs processed.