TECHNICAL · SEO GLOSSARY
Crawl-Delay
A robots.txt directive that instructs crawlers to wait a specified number of seconds between requests — used to protect server resources, but not respected by Googlebot, which manages crawl rate through Search Console instead.
Definition
The `Crawl-delay` directive in `robots.txt` asks crawlers to pause a specified number of seconds between successive requests to the server. Syntax: `Crawl-delay: 10` under a `User-agent:` block tells that crawler to wait at least 10 seconds between each request. This was originally designed to prevent crawlers from overwhelming shared hosting servers with rapid-fire requests. Crucially: **Googlebot does not honour `Crawl-delay`**. Google manages its crawl rate through its own adaptive system, and the official way to reduce Googlebot's crawl rate on a live site is via the Crawl Rate settings in Google Search Console. Other major crawlers — Bing's BingBot and Apple's Applebot — do respect the Crawl-delay directive, as do most SEO tool crawlers (Ahrefs, Semrush). Setting a high `Crawl-delay` in robots.txt may slow down third-party SEO tool crawls (affecting how quickly data updates in rank trackers) but will not slow Googlebot.
Why it matters for SEO
Using `Crawl-delay` to manage Googlebot is a common misconception — it simply doesn't work. If your server is being overwhelmed by Googlebot requests, the correct fix is Google Search Console's Crawl Rate Limit setting (under Settings → Crawling). Conversely, if you have crawl budget issues (important pages not being crawled frequently enough), the answer is improving server response time and ensuring a clean sitemap — not a Crawl-delay directive. For other crawlers that do respect it: an excessively high Crawl-delay (e.g. 60 seconds) will cause SEO tools to take very long to crawl your site, which isn't a meaningful SEO benefit but can frustrate audits and competitor research.
How DeepSEOAnalysis checks this
The audit reads and parses the robots.txt file, including any `Crawl-delay` directives. It flags unusually high Crawl-delay values (e.g. >10 seconds) as a potential issue for third-party SEO tool crawls, notes that Googlebot won't be affected, and recommends using GSC's Crawl Rate setting for Googlebot-specific rate management. It also checks for `Crawl-delay` set on all user-agents (`User-agent: *`) vs specific user-agents.
Useful tools and resources
GLOSSARY
Related terms
technical
Robots.txt
A text file at the root of a domain that tells crawlers which pages or sections to access or avoid.
Read definition →technical
Crawl Budget
The number of pages Googlebot will crawl on a site within a given timeframe — determined by crawl rate limit and crawl demand.
Read definition →technical
Crawl Frequency
How often Googlebot revisits pages on a site — influenced by crawl budget, update frequency, PageRank, and server response time — determining how quickly new or updated content enters Google's index.
Read definition →technical
Crawl Depth
The number of clicks (links) a crawler must follow from the homepage to reach a given page — pages deeper than 4 clicks receive less frequent crawling and less internal PageRank.
Read definition →technical
Google Search Console
Google's free tool for monitoring how your site appears in Google Search — showing impressions, clicks, CTR, average position, index coverage, and crawl errors.
Read definition →See how your site scores on Crawl-Delay.
The free DeepSEOAnalysis audit checks crawl-delay and 100+ other signals. Full report, no signup.
Run a free audit →