TECHNICAL · SEO GLOSSARY
Robots.txt
A text file at the root of a domain that tells crawlers which pages or sections to access or avoid.
Definition
Robots.txt is a plain-text file at `yourdomain.com/robots.txt` that specifies crawl access rules for bots. It uses `User-agent` directives to target specific crawlers and `Allow`/`Disallow` rules to grant or restrict access to paths. Crawlers that respect the Robots Exclusion Protocol (REP) will follow these rules — but the file is advisory, not enforced.
Why it matters for SEO
A misconfigured robots.txt can de-index an entire site. Accidentally disallowing `User-agent: *` (all bots) or disallowing `/` blocks Googlebot from crawling anything. This is one of the most catastrophic and most common SEO mistakes — it often happens during site migrations or when a staging robots.txt carries over to production. In 2024, robots.txt gained additional importance for AI-visibility: GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers all check robots.txt for access rules.
How DeepSEOAnalysis checks this
The audit fetches robots.txt and checks whether it disallows Googlebot or all user-agents from key paths. It also checks for AI crawler rules: whether GPTBot, ClaudeBot, PerplexityBot, and Gemini-Google-Bot are explicitly allowed or blocked. Accidental AI crawler blocks are flagged as a warning because they reduce the site's AI-visibility score.
Useful tools and resources
GLOSSARY
Related terms
ai visibility
llms.txt
A plain-text file at the root of a domain that guides AI systems to a site's most useful and citeable pages.
Read definition →ai visibility
Generative Engine Optimization (GEO)
The practice of structuring content so that AI-powered answer engines can retrieve, understand, and cite it in generated responses.
Read definition →See how your site scores on Robots.txt.
The free DeepSEOAnalysis audit checks robots.txt and 100+ other signals. Full report, no signup.
Run a free audit →