Google on Large Product Aggregator Websites and Programmatic SEO

A large product aggregator site can rack up more than a million crawls yet show only a handful of URLs in a search index. That gap between crawling and indexing is a signal worth studying, because it points to two fixable issues, redirect chains and programmatic SEO quality, that any large aggregator can audit for.

What happened with the 1.17 million crawl site?

A large product aggregator, ponito.com, received 1.17 million crawls over a few weeks but had only four URLs indexed out of an estimated 180,000 pages. The site’s home page was not indexed at all. The crawl stats in Google Search Console spiked sharply during that window. The site’s own URL examination shows that every indexed URL redirected to the homepage, which means Google had no real content to keep in its search results.

Why did Google refuse to index the home page?

Two issues stood out:

• A technical issue with redirects. A batch of the URLs that were indexed from the site all redirected to the homepage, which means the indexed set was effectively a redirect loop with no destination content. Until those redirect loops are cleaned up, Google has nothing to keep in the index.
• The harder problem is convincing search engines and users that programmatically generated pages on a large product aggregator are worth crawling.

What makes large product aggregators hard to index?

A few structural factors line up:

• URL volume outpaces content value. A site can publish tens or hundreds of thousands of URLs through templates and feeds, but if each page adds little beyond what neighboring pages already say, the value drops quickly.
• Content that offers no unique data or original descriptions gets treated like other aggregators by Google.
• Technical debt scales with URL count. Redirects, canonicalization, and internal linking all have to hold up at scale. A small misconfiguration on a 200-page site is a quiet issue; the same misconfiguration on a 180,000-page site can drag the whole site out of an indexable state.

How can a large aggregator improve their indexation?

The fix list maps to the two issues flagged:

• Audit the redirect chains. Find every indexed URL that redirects to the homepage and decide whether it should 301 to a real category or product page, 404 if the content is gone, or self-canonicalize on a unique destination. Crawl logs and Search Console’s URL Inspection tool both help here.
• Reduce thin or duplicated programmatic pages. Consolidate near-duplicate templates, add unique data per page (specs, prices, availability), and block indexing of low-value combinations through robots rules or noindex directives.
• Strengthen the homepage and category hubs. Internal linking from high-authority templates to the pages that should rank gives Google a clearer map of which URLs matter.
• Submit a refreshed sitemap. Once the redirect mess is cleaned, a clean sitemap helps Google rediscover the URLs worth keeping.

For a large aggregator, technical SEO at scale is the price of admission. NewSunSEO has been working on SEO in one form or another since 2001, and the pattern is the same: sites that keep their technical foundation clean at every URL get the indexing budget; sites that let it rot at scale lose it.

FAQ

What is a large product aggregator website?

A large product aggregator is a site that pulls product listings or data from many sources into one large catalog, often producing hundreds of thousands of programmatically generated URLs. Search engines treat them as a distinct category because the technical and content-quality challenges scale with the URL count.

Why would Google crawl a site 1.17 million times but index only 4 pages?

High crawl volume does not guarantee indexing. In the case covered here, every indexed URL redirected to the homepage, and the rest of the site’s URLs appeared to be programmatic pages that did not meet Google’s quality bar. Crawling and indexing are separate decisions, and both have to pass for a page to appear in search results.

Does Google penalize programmatic SEO?

Google has stated that it is generally not a fan of programmatic SEO. The site-level effect is not a manual penalty but a quality filter: programmatic pages that lack unique value are harder to get indexed and harder to rank. Programmatic pages with real, differentiated content and clean technical signals can still perform.

Related coverage

• What 1.8 million Google Business Profiles reveal about local SEO success – LocalSEOBot
• Google’s top ranking factors, according to 131 SEO professionals – LocalSEOBot

SEOScanPro

SEOScanPro’s site audit tool runs a full technical audit of a site and shows the measured result behind every check. Open the site audit tool.