Deleting 2.1M pages adds $638K a month in organic revenue
Faceted navigation generated millions of near-identical filter combinations. Crawl budget drains there and never reaches new inventory. Removing them models organic revenue from $412K to $1.05M a month.
| Measure | Before | After |
|---|---|---|
| Monthly organic revenue | $412K | $1.05M |
| Added cash flow per month | — | +$638K |
| Crawl requests hitting valuable URLs | 13% | 79% |
| Time to index a new listing | 19 days | 1.6 days |
| Indexed pages that earn a click | 9% | 44% |
The problem
Log analysis told the story bluntly: 87% of crawl requests hit filter permutations — colour plus size plus sort plus page — that no human had ever landed on. Fresh listings took 19 days to be indexed, by which point roughly a third had already sold. Category pages that mattered were being crawled once every six weeks.
What the model found
A model classifies six weeks of raw log lines and 2.4M URL patterns into keep, canonicalise and block buckets, cross-referenced against actual entrance and revenue data. By hand that was quoted at three months; it takes four days, and it is what makes deleting 2.1M URLs a decision rather than a gamble.
The plays
Read the server logs first
Six weeks of raw logs segmented by URL pattern, response code and bot. Every decision after this is made against observed crawl behaviour rather than a crawl simulation.
Collapse the facet space
A rules matrix decides which facet combinations are indexable, which are canonicalised, and which are blocked outright. 2.1M URLs removed from the crawlable set; 40,000 valuable combinations kept and strengthened.
Make freshness discoverable
Segmented sitemaps by inventory age with accurate lastmod, plus a new-listings feed. Indexation of new inventory drops from 19 days to under 2.
Rebuild internal linking around demand
Hub pages for the facet combinations with real search demand, linked from category headers, replacing a flat pagination chain that buried everything past page three.
Localise properly, per market
Corrected reciprocal hreflang across 11 countries and split duplicated English variants that had been competing with each other for years.
The lever
Deleting, not adding. Removing 2.1M worthless URLs from the crawlable set does more for revenue than any content produced that year.
How the model was calibrated
Grounded in the 2025 shift in AI Overview query intent — informational share fell from 91.3% to 57.1% as commercial and transactional queries moved in, across 10M+ keywords. Semrush’s separate study of 200,000 keywords found AI Overview links overlap the organic top 10 only 20–26% of the time, with an average of 11 links per answer. For a marketplace, that means inventory pages have to be indexed and fresh to be eligible at all — which is a crawl problem before it is a content problem.
Sources
- Semrush — AI Overviews Study — 200,000 keywords. AI Overview links overlap the organic top 10 only 20–26% of the time; an average AI Overview carries 11 links and includes the #1 organic result just 46% of the time on desktop, 34% on mobile.
- Semrush — AI Overview prevalence and intent, 2025 — AI Overviews appeared on 6.49% of searches in January 2025, 24.61% in July and 15.69% in November. Purely informational share of those queries fell from 91.3% to 57.1% as commercial intent moved in.
- Semrush — AI Search & SEO Traffic Study — ~90% of ChatGPT search citations point to pages ranking 21+; AI search visitors convert at 4.4× ordinary organic; 50% of ChatGPT links go to business and service sites.
Want the same read on your category?
Send a brief and we will run twenty prompts before we talk.