ROBOTS.TXT TESTER

robots.txt tester

Paste robots.txt, choose a crawler, and see whether a path is allowed or blocked — including the exact matching rule.

Browse all toolsRun the full audit →

Tester

Paste robots.txt and test a path

Rules are evaluated by the most specific matching user-agent group and longest Allow/Disallow pattern.

How it works

How does this tool work?

  • Paste the robots.txt content you want to test.
  • Choose a user-agent and path. The tester selects the best matching group and longest rule.
  • Review the AI crawler summary to catch accidental blocks on GPTBot, ClaudeBot, PerplexityBot, and related bots.

MORE FREE TOOLS

Other tools you might need

LL

llms.txt Generator

Draft a clean llms.txt file with key pages, summaries, and an optional llms-full.txt reference.

{}

Schema Markup Generator

Create copy-paste JSON-LD for Organization, WebSite, FAQPage, BreadcrumbList, and Article schema.

SERP

Meta Tag & SERP Preview

Preview title tags and meta descriptions on desktop and mobile before they get truncated.

FROM THE BLOG

Want to go deeper?

→How to Test Your robots.txt File (and What to Fix)

FAQ

Questions people ask before using this tool.

What does robots.txt control?

robots.txt tells crawlers which paths they are allowed or disallowed to fetch. It is crawl guidance, not a privacy or security control.

Why test AI crawlers separately?

AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, and Google-Extended may match different user-agent groups than Googlebot.

What happens when no rule matches?

If no Allow or Disallow rule matches the selected user-agent and path, the default robots.txt behavior is allowed.

Does allowing a crawler guarantee indexing or AI citations?

No. It only means robots.txt is not blocking access. Page quality, schema, internal links, and answer-ready content still matter.