TECHNICAL · SEO GLOSSARY

Crawl Stats

Crawl Stats is a report in Google Search Console (Settings → Crawl Stats) that shows Googlebot\'s crawling activity on your site over the past 90 days — total crawl requests, average response time, and status code distribution. It is the primary tool for diagnosing crawl budget problems and monitoring Googlebot\'s interaction with your site.

Definition

The Crawl Stats report (GSC → Settings → Crawl Stats) provides three views: **Total crawl requests**: the number of URLs Googlebot fetched per day over 90 days. Spikes may indicate Googlebot discovering new content (positive) or following error chains (negative). Sharp drops can indicate robots.txt changes blocking crawl or site availability problems. **By response**: breakdown of crawl requests by HTTP response code — 2xx (successfully crawled), 3xx (redirects), 4xx (not found/forbidden), 5xx (server errors). High 4xx rates indicate broken internal links or removed content without redirects. High 3xx rates indicate redirect chain overhead. High 5xx rates indicate server capacity problems. **By file type**: HTML, JavaScript, CSS, image, and other file type crawl distribution. High JavaScript and CSS crawl volume indicates Googlebot is rendering pages (expected for modern sites). **Host status**: TTFB and availability from Googlebot\'s perspective — if Googlebot sees high TTFB or errors, this signals crawl rate reduction. **How to use Crawl Stats**: Monitor total crawl requests trend (is Google crawling more or fewer pages over time?), watch for 4xx and 5xx spikes, compare crawl rate before/after robots.txt changes, and assess whether Googlebot\'s TTFB aligns with user-facing performance. The crawl request count can also help estimate crawl budget sufficiency: if a 100,000-page site gets 5,000 crawls/day, it takes 20 days for a full site crawl — new content and updates may take weeks to be discovered and indexed.

Why it matters for SEO

Crawl Stats is one of the few windows into what Googlebot actually does on your site, rather than what you\'ve configured for it to do. It reveals if robots.txt changes have accidentally blocked important content, if redirect chains are consuming crawl budget, if server errors are reducing crawl rate, and if new content is being discovered at an appropriate rate. For large sites, Crawl Stats is essential for crawl budget management.

How DeepSEOAnalysis checks this

The audit doesn\'t access GSC Crawl Stats data directly, but it identifies the factors that would show up as problems in Crawl Stats: robots.txt configuration that might accidentally block important pages, redirect chain depth (3xx overhead in Crawl Stats), sitemap quality (helping Googlebot prioritise the right URLs), and server response time (TTFB issues that Crawl Stats would reflect as poor Host Status).

Useful tools and resources

GLOSSARY

technical

Crawl Budget

The number of pages Googlebot will crawl on a site within a given timeframe — determined by crawl rate limit and crawl demand.

Read definition →

technical

Googlebot

Googlebot is Google\'s web crawler — the automated bot that discovers, fetches, and indexes web pages for inclusion in Google Search. Googlebot crawls the web continuously, following links and respecting robots.txt directives, then passes pages to Google\'s indexing pipeline for analysis and ranking.

Read definition →

technical

Robots.txt

A text file at the root of a domain that tells crawlers which pages or sections to access or avoid.

Read definition →

technical

Crawl Budget Optimisation

The practice of helping Google allocate its crawl capacity to your highest-value pages — by eliminating low-value URLs, consolidating duplicate pages, and improving crawl efficiency through internal link structure.

Read definition →

technical