TECHNICAL · SEO GLOSSARY
Crawl Budget Optimisation
The practice of helping Google allocate its crawl capacity to your highest-value pages — by eliminating low-value URLs, consolidating duplicate pages, and improving crawl efficiency through internal link structure.
Definition
Crawl budget refers to the number of URLs Googlebot will crawl on a site within a given timeframe. Google's crawl allocation is determined by two factors: crawl rate limit (how fast Googlebot can crawl without overloading the server — based on server response speed and capacity signals) and crawl demand (how much Google wants to crawl based on popularity and freshness signals). Crawl budget is most relevant for large sites — those with hundreds of thousands of pages, frequently updated content, or known indexing delays. For smaller sites (under 10,000 pages), crawl budget is rarely a constraint. Crawl budget optimisation means steering Googlebot toward your most valuable pages and away from low-value URL sprawl. The primary levers: block low-value URLs via robots.txt `Disallow` (removes them from crawl), use `noindex` on thin pages (still crawled but not indexed), implement canonical tags on duplicates (redirects crawl equity to the canonical), and improve internal link architecture so important pages receive more internal PageRank (attracting more frequent crawling).
Why it matters for SEO
On large ecommerce or publishing sites, crawl budget misallocation is a direct cause of slow indexing: new content added daily may not be crawled for days or weeks if Googlebot is spending its allocation on paginated archive pages, session ID URL variations, and internal search result pages. The crawl log (server logs filtered for Googlebot user-agent) is the authoritative source for understanding how Googlebot is spending its allocation. Common crawl budget leaks: infinite-scroll or faceted navigation generating thousands of unique parameter URLs, session IDs in URLs, duplicate content at www/non-www or HTTP/HTTPS variants, aggressively paginated archives, and URL parameters from analytics or A/B testing tools (?utm_*, ?ab_test_*) that generate unique URLs.
How DeepSEOAnalysis checks this
The audit checks the primary crawl budget efficiency signals: parameterised URL proliferation (sorting, filtering, session, UTM parameters generating unique crawlable URLs), low-value page types accessible to Googlebot (internal search results, thin paginated archives, printer-friendly page variants), robots.txt coverage of known low-value paths (/tag/, /page/, /search?), and canonical tag consistency across duplicate URL variants. It reports the estimated scale of URL waste as a crawl budget impact score.
Useful tools and resources
GLOSSARY
Related terms
technical
Crawl Budget
The number of pages Googlebot will crawl on a site within a given timeframe — determined by crawl rate limit and crawl demand.
Read definition →technical
Robots.txt
A text file at the root of a domain that tells crawlers which pages or sections to access or avoid.
Read definition →technical
Canonical URL
The preferred URL for a page, declared via <link rel="canonical"> to prevent duplicate content from splitting ranking signals.
Read definition →technical
Faceted Navigation
Filter-and-sort UI on category pages (color, size, price, brand) that generates a combinatorial explosion of parameter URLs — the most common source of crawl budget waste on e-commerce sites.
Read definition →technical
Index Bloat
Having too many low-quality, thin, or duplicate pages indexed by Google — wasting crawl budget, diluting site authority, and potentially triggering Helpful Content penalties.
Read definition →See how your site scores on Crawl Budget Optimisation.
The free DeepSEOAnalysis audit checks crawl budget optimisation and 100+ other signals. Full report, no signup.
Run a free audit →