TECHNICAL · SEO GLOSSARY
Crawl Trap
A URL pattern that generates a near-infinite number of crawlable URLs — such as infinite scroll, recursive calendar links, or session-ID parameters — trapping Googlebot in an endless crawl loop and wasting crawl budget.
Definition
A crawl trap (also called a spider trap) is any URL structure that causes a crawler to follow an unbounded number of links without ever crawling all useful content. Common crawl traps: (1) **Infinite scroll or calendar navigation** — a site that generates links to `/events/2020/01/`, `/events/2020/02/` ... `/events/2099/12/`, extending indefinitely into the future; (2) **Session IDs or tracking parameters in URLs** — a URL like `/page?sessionid=abc123` that generates a new crawlable URL for every visitor session, causing the crawler to index millions of near-identical pages; (3) **Recursive URL patterns** — a faceted navigation that appends filters iteratively (`/shoes/red/`, `/shoes/red/size-9/`, `/shoes/red/size-9/brand-nike/`...) producing exponentially many URL variants; (4) **CMS auto-generated archive intersections** — tag + category + date archive combinations that multiply into thousands of nearly empty listing pages; (5) **Search result pages without noindex** — site search pages indexed with query parameters, each generating a unique URL. Crawl traps waste Googlebot's crawl budget on low-value or duplicate pages instead of crawling and indexing important content.
Why it matters for SEO
Crawl traps directly damage SEO in two ways: (1) **Crawl budget waste** — Googlebot has a limited amount of crawling it will do per site per day; time spent on trap URLs is time not spent on your important pages; (2) **Index bloat** — if traps aren't blocked, Google may index millions of thin or duplicate pages, diluting site quality signals and potentially triggering Helpful Content or thin-content algorithmic demotion. The fix is a combination of: blocking trap URL patterns in `robots.txt` (for parameter-based traps), adding `rel="canonical"` to consolidate parameter variants to the canonical URL, removing links to trap pages from the HTML, and using `noindex` on low-value archive pages that can't be removed.
How DeepSEOAnalysis checks this
The audit detects crawl trap indicators during the crawl: URL patterns where the same content is accessible via large numbers of different URLs (parameter variants), calendar or paginated navigation links that extend more than 3 levels deep without diminishing content, and site search pages accessible to crawlers via URL parameters. It flags these patterns with the specific parameter names or path patterns generating the trap, and recommends the appropriate robots.txt Disallow rule or canonical tag to close the trap.
Useful tools and resources
GLOSSARY
Related terms
technical
Crawl Budget
The number of pages Googlebot will crawl on a site within a given timeframe — determined by crawl rate limit and crawl demand.
Read definition →technical
Crawl Frequency
How often Googlebot revisits pages on a site — influenced by crawl budget, update frequency, PageRank, and server response time — determining how quickly new or updated content enters Google's index.
Read definition →technical
Index Bloat
Having too many low-quality, thin, or duplicate pages indexed by Google — wasting crawl budget, diluting site authority, and potentially triggering Helpful Content penalties.
Read definition →onpage
Duplicate Content
Identical or substantially similar content appearing at multiple URLs — which forces Google to choose one version to index and can dilute ranking signals across copies.
Read definition →technical
Faceted Navigation
Filter-and-sort UI on category pages (color, size, price, brand) that generates a combinatorial explosion of parameter URLs — the most common source of crawl budget waste on e-commerce sites.
Read definition →See how your site scores on Crawl Trap.
The free DeepSEOAnalysis audit checks crawl trap and 100+ other signals. Full report, no signup.
Run a free audit →