Index Product Pages: E-commerce Indexing at Scale
A store with forty pages of content and four thousand product pages has, in practice, two indexing problems. The content indexes the way most sites index. The catalogue does not. To index product pages at scale you are fighting three things at once: pages that look like each other, a URL structure that multiplies them, and a crawler that has no obligation to visit all of them. None of these is solved by submitting harder. Each is solved by making the catalogue smaller, cleaner, and easier to verify than it currently is.
This post works through the three, in the order that usually pays.
Why product pages are the hardest pages to index
Product pages are structurally unlucky. They are numerous, they resemble one another, and the differences between them are often a size, a colour, or a price. From Google’s side of the fence, a catalogue looks like a large set of near-duplicates with thin descriptions, and Google is selective about near-duplicates with thin descriptions.
Three patterns do most of the damage.
Variants multiply URLs. One shirt in five sizes and four colours can exist at twenty URLs if each variant gets its own address with a query string. Google’s guidance on consolidating duplicate URLs exists for exactly this case: pick one canonical URL per product and point every variant at it. Without that, your twenty URLs compete with each other for one indexing decision.
Faceted navigation multiplies them again. Filters for brand, price band, and availability generate crawlable URLs by the thousand. Most of them are permutations nobody searches for. A crawler that spends its visit walking ?colour=blue&sort=price_asc pages is not visiting the product pages you actually wanted indexed.
Platform paths add a layer. Shopify is the well-known example: a product reachable through a collection lives at /collections/summer/products/linen-shirt while its canonical form is /products/linen-shirt. Shopify sets the canonical tag correctly by default, but the collection-path URLs still get crawled, and any tool or sitemap that harvests the collection paths is submitting URLs Google intends to fold into another address. If Google seems to be picking odd URLs for your products, how canonicalization decides which URL gets indexed is the mechanism to understand first.
Decide what deserves to be indexed, then say so
The counterintuitive move on a large store is to shrink the indexable surface before trying to grow the indexed count. Google’s own advice for managing crawl budget on large sites is written for exactly this shape of site, and the theme running through it is elimination: fewer duplicate paths, fewer soft errors, fewer URLs that lead nowhere useful.
For a catalogue, that means a short list of decisions made deliberately rather than by default. Canonical tags on every variant, pointing at the one URL you want to rank. Filter permutations kept out of the crawl path rather than left to be discovered and discarded. Out-of-stock pages given a policy (keep them live if the product returns, redirect them if it does not) instead of being allowed to drift into soft 404 territory. Discontinued products returning a real 404 rather than a thin “no longer available” page.
Every URL you remove from the pile improves the odds for the URLs that remain. A crawler that stops wasting requests on permutations has more of them left for new products.
Sitemaps carry the catalogue
Once the surface is clean, the sitemap is how you present it. The Sitemaps protocol allows up to 50,000 URLs per file, with an index file tying multiple sitemaps together, so even a very large catalogue fits in a handful of files.
Two habits make sitemaps genuinely useful rather than ceremonial. First, list only canonical URLs. A sitemap full of collection-path or parameterised URLs actively argues against your own canonical tags. Second, split by section: products, categories, and content in separate files. When indexing stalls, a segmented sitemap tells you which part of the site stalled, which a single 40,000-line file never will.
Sitemaps are discovery, though, not a queue. Google reads them and then makes its own decisions on its own schedule. New product launches, price-sensitive seasonal pages, and republished ranges often need a more direct route.
Submitting at scale, and checking what happened
This is the point where scale becomes a logistics problem. Requesting indexing one URL at a time through Search Console is fine for a blog and hopeless for a catalogue refresh of three thousand SKUs. The mechanics of doing it in one pass are covered in how to submit 10,000 URLs at once; the short version is that you want a pipeline that accepts a CSV, not an afternoon of pasting.
This is also where a tool like AIndexify fits the e-commerce case specifically. You export your product URLs, upload the CSV (up to 10,000 URLs per submission), and each URL is then checked against live Google results on a schedule, so submitted and indexed are reported as two separate events rather than one optimistic tick. For a catalogue, the preflight matters as much as the submission: a URL that carries a noindex tag, returns a 404, or is blocked by robots comes back as Couldn’t check, with the credit refunded. On a platform-generated store, that status list is often the first honest inventory of how many of your “product pages” were never eligible in the first place.
Whatever you use, keep the two events separate in your own reporting. Submission asks Google to look. Indexing is Google deciding the page belongs. A catalogue dashboard that only records the first event will tell you everything shipped and nothing about what arrived.
A working order for a real store
The sequence that tends to hold up: fix canonicals and variant handling first, prune the faceted and parameterised URLs second, rebuild sitemaps from canonical URLs third, then submit in bulk and let scheduled checks report what Google actually kept. If large sections stay unindexed after that, the problem is usually eligibility rather than discovery, and the diagnosis moves to content and site structure, which the full guide to Google indexing walks through end to end.
None of this makes Google index a catalogue it considers thin. What it does is remove every reason you control, and then measure the result instead of assuming it. On four thousand SKUs, the measuring is the difference between an indexing strategy and a hope.
If you run a catalogue, the useful first step is evidence: submit your URLs and see, product by product, where your pages actually stand.