How to Test Your robots.txt File (and What to Fix)

Your robots.txt file is small — usually under 30 lines. But it sits between every crawler and every page on your site, and one misconfigured line can silently de-index your entire domain. This guide explains how to test it correctly, what each common mistake looks like, and how to fix it.

What does a robots.txt file actually do?

A robots.txt file at yourdomain.com/robots.txt tells crawlers which parts of your site they're allowed to access. It uses User-agent directives to target specific bots and Allow or Disallow rules to specify paths.

User-agent: *
Disallow: /admin/
Allow: /admin/public-report/

User-agent: Googlebot
Disallow: /staging/

Sitemap: https://example.com/sitemap.xml

Every crawler that follows the Robots Exclusion Protocol (REP) will read this file before crawling. That includes Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, and hundreds of others.

Important distinction: robots.txt controls access (whether a bot can fetch the URL). It does not control indexing. A page blocked by robots.txt won't be crawled, but it can still appear in search results if other pages link to it — Google will just show the URL with no snippet. To prevent indexing, use <meta name="robots" content="noindex"> on the page itself.

How do I test my robots.txt file?

There are three ways to test a robots.txt file effectively.

1. Use a live robots.txt tester

The fastest method is a live tester that fetches your actual robots.txt and evaluates it against a specific user-agent and URL. The DeepSEOAnalysis robots.txt tester does this:

1. Enter your domain
2. The tester fetches your current live robots.txt
3. Test any URL against any user-agent to see if it's allowed or blocked
4. Check AI crawler access specifically (GPTBot, ClaudeBot, PerplexityBot, and more)

This catches errors that static analysis misses — robots.txt files sometimes serve differently from behind a CDN, return wrong content-types, or are cached stale.

2. Check Google Search Console

Google Search Console includes a robots.txt tester under Settings. It shows the last robots.txt Google fetched, when it fetched it, and lets you test URLs against the Google crawler. The key advantage here is it shows what Google actually saw, not what you think your robots.txt says — these can differ if a deploy broke your robots.txt route.

3. Fetch it directly and read it

curl -s https://yourdomain.com/robots.txt

For a quick sanity check, just fetch it and read it. Look for lines that disallow more than intended. The most dangerous line is:

Disallow: /

Under any User-agent: * directive, this blocks all crawlers from the entire site.

What are the most common robots.txt mistakes?

Mistake 1: Staging noindex carried to production

The single most common catastrophic robots.txt mistake is shipping with:

User-agent: *
Disallow: /

This line is standard in staging and development environments to prevent search engines from indexing a work-in-progress site. It becomes a disaster when it carries over to production — usually during a platform migration, a server clone, or when a developer pushes a robots.txt template without realizing it overwrites the live one.

How to detect it: The full audit checks whether your robots.txt disallows all user-agents from the root path and flags it as a critical error. The fix is to remove the Disallow: / line or replace it with specific path restrictions.

Mistake 2: Disallowing CSS and JavaScript

Old SEO advice said to block crawlers from CSS and JavaScript to save crawl budget. Modern Googlebot renders JavaScript and needs CSS to understand your page layout. Blocking these with:

Disallow: /wp-content/themes/
Disallow: /assets/js/

...causes Google to see your pages without their rendered content, which often dramatically misrepresents what's on the page.

Mistake 3: Blocking AI crawlers unintentionally

In 2024–2025, many site owners started explicitly blocking AI crawlers. The problem is that broad User-agent: * Disallow rules often block AI bots along with Googlebot. If you want to block AI crawlers specifically, be explicit:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

If you want to allow Google while restricting AI crawlers, separate the rules. A blanket Disallow: / under User-agent: * blocks everyone.

Mistake 4: Syntax errors that make rules unparseable

Robots.txt is whitespace-sensitive. A common error is missing the colon after User-agent or adding a space before it:

# Broken:
Useragent: *
User-agent :*

# Correct:
User-agent: *

Another common error is using the wrong case: Disallow must be capitalized. disallow: / won't work in many crawlers.

Mistake 5: Forgetting the Sitemap declaration

The robots.txt Sitemap directive (Sitemap: https://yourdomain.com/sitemap.xml) is optional but important — it tells crawlers directly where your sitemap is, independent of any meta tag or Search Console submission. A missing Sitemap directive in robots.txt means crawlers have one fewer signal pointing them to your full URL list.

How do I check AI crawler access in robots.txt?

In 2025, an increasing share of traffic and brand discovery runs through AI answer engines. GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, and Gemini-Google-Bot all check robots.txt before crawling.

If you haven't explicitly addressed these bots, your default behavior is whatever the User-agent: * rules say. To see exactly which AI crawlers can access your site right now, use the AI crawler checker tool — it fetches your live robots.txt and evaluates access for 10 AI bots.

A typical allow-all-AI configuration looks like this:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Gemini-Google-Bot
Allow: /

A typical block-all-AI configuration:

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Google-Extended
Disallow: /

How do I interpret a robots.txt test result?

A robots.txt test returns one of three outcomes for a URL+user-agent pair:

• Allowed — no rule restricts this bot from this URL
• Blocked — a Disallow rule applies; the bot won't crawl this URL
• Allowed (overrides Disallow) — an Allow rule is more specific than a Disallow and takes precedence

The specificity rule matters: Allow: /admin/public/ beats Disallow: /admin/ for the path /admin/public/report.html. The more specific rule always wins, regardless of which appears first in the file.

What should be in a well-configured robots.txt?

A solid robots.txt for most sites:

User-agent: *
Disallow: /api/
Disallow: /admin/
Disallow: /dashboard/
Disallow: /?*          # Block parameterized variants if duplicate content is a concern

Sitemap: https://yourdomain.com/sitemap.xml

Add explicit AI crawler rules based on your position: block them to protect content you don't want scraped for training; allow them to maximize AI visibility and citation opportunities.

For a complete automated check of your robots.txt alongside 100+ other SEO signals, the full site audit} covers technical, on-page, performance, and AI visibility in one report.

---

FAQ

Does robots.txt affect Google rankings? Indirectly. Blocking important pages from Googlebot means they won't be crawled or indexed, so they can't rank. Blocking CSS and JavaScript degrades how Google renders and understands pages. Correct robots.txt configuration is necessary for any page to rank, but having a well-configured file alone doesn't improve rankings.

Can I use wildcards in robots.txt? Yes. Disallow: /*/preview blocks any path that contains /preview after any segment. Disallow: /*.pdf$ blocks all PDFs. Not all crawlers support wildcards — Googlebot does; some older bots don't.

What happens if robots.txt returns a 404 or 500 error? A 404 means there's no robots.txt and crawlers treat the whole site as unrestricted. A 5xx server error causes Googlebot to temporarily restrict crawling until the file is accessible again — it treats a 5xx as "maybe the site is having trouble, don't crawl aggressively" and backs off.

How often does Googlebot re-fetch robots.txt? Google typically re-fetches robots.txt every 24 hours but may cache it for up to 24 hours. If you make a change to robots.txt that you need to take effect immediately, you can request a re-fetch via Google Search Console.