Here is a pattern we see in most large store audits. The client has around 2,000 products. Google has discovered 180,000 URLs. It has indexed 900 of them, and a third of the products that make money are not among them.
The cause is almost always faceted navigation.
How the numbers explode
Take a collection with five filters: size, colour, brand, price band and material. Say each has six options. Every combination of those creates a URL, and combinations multiply.
Add sort order and pagination on top of each combination and the number climbs again. Six filters with six options each is more than forty thousand URLs from one collection, and every one of them is a slight variation of the same product list.
Google does not have infinite patience for that. It allocates a crawl budget, spends it on the variations, and never gets to the pages you care about.
How to tell if it is happening to you
Four checks, in order:
- Search Console, Pages report. Compare "Discovered, currently not indexed" and "Crawled, currently not indexed" against your real product count. If discovered URLs run to many times your catalogue size, you have this.
- Site search. A rough
site:search in Google gives you a ballpark on indexed pages. Compare it against what you expect. - Crawl the site. Screaming Frog with parameters included will show you how fast the URL count climbs.
- Server logs, if you can get them. This is the definitive test, because it shows what Googlebot actually requested. If most requests are parameter URLs, your budget is going to filters.
The three ways to control it
1. Decide which facets deserve a real page
Some filter combinations have genuine search demand. "Black leather sofas" is a real search. "Black leather sofas, 3 seater, under 1000, in stock, sorted by newest" is not.
Where demand exists and you have enough products, build a real, indexable, crawlable collection with its own copy and its own H1. Everything else stays a filter.
Be strict. Ten good faceted landing pages beat two hundred thin ones, and thin near-duplicates get filtered out anyway.
2. Stop the rest being crawled
For everything that is not a real page, the cleanest option is to make the filter links uncrawlable in the first place. If Googlebot never sees the link, it never requests the URL and the budget is never spent.
The usual approaches are rendering filters through JavaScript that does not produce a plain anchor with an href, or using a # fragment rather than a query parameter.
If your filters already produce crawlable links and you cannot change that, robots.txt disallow rules on the parameter patterns will stop the crawling. Be careful: this stops crawling, not indexing, and a blocked URL that already has links can still appear in results without a description.
3. Canonicals for what does get crawled
Any filtered URL that stays crawlable should carry a canonical pointing at the unfiltered collection. noindex also works, but it means Google spends budget crawling a page in order to be told to ignore it. Canonical is cheaper for everyone.
Never combine noindex with a robots.txt block on the same URL. Google cannot read the noindex if it is not allowed to fetch the page.
Sort order and pagination
Sort parameters should never be indexable. Same products, different order, no additional value.
Pagination should be crawlable so Google can reach deeper products, but page two onward should not repeat the collection intro copy. If your paginated pages are thin, consider whether the collection should be split instead.
What fixing it actually does
Crawl budget work rarely produces a chart that goes straight up. What it does is let Google reach and refresh the pages that make you money. On stores where important products were simply never indexed, that shows up within weeks as those pages start appearing at all.
It is unglamorous, it is technical, and on a large catalogue it is usually the highest value thing in the audit. We handle it as part of technical SEO and e-commerce SEO. The technical SEO checklist has the full list of what we check.