robots.txt: How to Write It, What to Block, and What to Leave Open

robots.txt is a plain-text file at the root of your domain that tells crawlers which URLs they're allowed to access. It's the first thing Googlebot reads when it visits a site — and a single misconfiguration can block your entire site from crawling.

This guide covers the format, what to block vs leave open, how to handle AI crawlers, common mistakes, and how to test it.

What robots.txt does (and doesn't do)

What it does:

• Tells compliant crawlers (Googlebot, ClaudeBot, GPTBot, Bingbot) which URLs to skip
• Points crawlers to your sitemap via the Sitemap: directive
• Prevents crawlers from wasting time on admin panels, staging areas, and duplicate-content patterns

What it doesn't do:

• Does not prevent indexing — a page blocked in robots.txt can still appear in Google's index if other sites link to it. To prevent indexing, use <meta name="robots" content="noindex"> on the page itself
• Does not affect malicious bots — scrapers and malicious crawlers ignore robots.txt; it's only for compliant bots
• Does not pass PageRank — links on a disallowed page still count for PageRank calculations; robots.txt only affects crawling

The robots.txt format

A robots.txt file consists of groups, each with a User-agent: line (which crawler this applies to) and one or more Disallow: or Allow: lines:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

User-agent: * matches all crawlers. You can also target specific crawlers:

User-agent: Googlebot
Disallow: /no-google/

User-agent: *
Disallow: /admin/

Key syntax rules:

• The file must be UTF-8 encoded and served at exactly /robots.txt (lowercase)
• HTTP status must be 200; a 404 is treated as "no restrictions" — not an error, but you lose the Sitemap directive
• A 500 error causes Googlebot to temporarily stop crawling the site
• Blank lines separate groups; everything after # is a comment
• Rules are case-sensitive on case-sensitive servers: /Admin/ and /admin/ are different paths
• An empty Disallow: value (i.e., Disallow: with nothing after it) means "allow everything" — not "block everything"

What to block

Block these with confidence:

User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /login/
Disallow: /dashboard/
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/
Disallow: /thank-you/
Disallow: /search/
Disallow: /internal-preview/
Disallow: /staging/

Internal-only and authentication pages — no SEO value and they waste crawl budget.

Search results pages — your /search/?q=keyword pages are thin content and URL parameter explosions. Block the search endpoint (e.g., /search/) to prevent Googlebot from crawling millions of query combinations.

Cart, checkout, account, thank-you pages — not indexable content. Block them to save crawl budget.

Duplicate parameter patterns — if your e-commerce site generates thousands of URLs from sorting and filtering, you can block the parameter patterns:

Disallow: /*?sort=
Disallow: /*?color=
Disallow: /*&session_id=

However: canonical tags on the parameter pages are a more precise solution (they let Googlebot discover the pages but signal the canonical). robots.txt blocks are a blunt instrument.

What to leave open

Never block these:

CSS and JavaScript files — Google needs to render your pages to see JavaScript-rendered content and apply mobile-first indexing. Blocking /assets/, /static/, or *.js prevents Google from understanding your page's rendered state. A common WordPress mistake: blocking /wp-content/ which contains all theme CSS.

Core content pages — your blog posts, product pages, landing pages, service pages. These should all be crawlable.

Images — unless they're purely functional (icons, UI sprites). Block Googlebot-Image from specific private image directories if needed, not from images site-wide.

Your sitemap URL — don't Disallow: /sitemap.xml. Reference it in the Sitemap: directive instead.

The robots.txt file itself — you can't block robots.txt; the spec requires it to always be accessible.

AI crawlers and robots.txt

Since 2023, AI companies have deployed crawlers to train their models and power AI answer engines. These crawlers follow robots.txt. Whether to allow or block them is now a significant strategic decision:

If you block AI crawlers, you reduce:

• Training data contributions (a concern if you want to opt out of LLM training)
• AI citation visibility — ChatGPT, Perplexity, Claude, and Google AI Overviews can't cite pages their crawlers couldn't read

If you allow AI crawlers, you gain:

• Potential citations in AI answers (a traffic driver as users act on AI recommendations)
• Presence in AI-generated "best of" lists and comparisons

The major AI crawler user-agents:

| User-agent | Powers | |---|---| | GPTBot | OpenAI (ChatGPT, DALL-E) | | ClaudeBot | Anthropic (Claude) | | PerplexityBot | Perplexity AI | | Googlebot-Extended | Google AI products, Bard/Gemini | | anthropic-ai | Anthropic alternative agent | | cohere-ai | Cohere | | CCBot | Common Crawl (training datasets) |

Recommended approach for most sites: allow AI crawlers. If your goal is organic visibility — including visibility in AI answers — blocking GPTBot, ClaudeBot, and PerplexityBot directly undermines that goal. The training data concern is separate: if you're concerned about LLM training, consult the individual companies' opt-out mechanisms rather than blocking via robots.txt alone.

A permissive robots.txt for AI visibility:

User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search/

# AI crawlers — allow for AI citation visibility
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml

The explicit Allow: / rules override any wildcard Disallow directives that might otherwise affect AI crawlers.

The Sitemap directive

Always include your sitemap URL at the bottom of robots.txt:

Sitemap: https://example.com/sitemap.xml

This tells every crawler that discovers your robots.txt where to find your sitemap — not just Googlebot. You can include multiple Sitemap directives if you have a sitemap index:

Sitemap: https://example.com/sitemap-index.xml
Sitemap: https://example.com/sitemap-blog.xml

The Sitemap directive is independent of User-agent groups — it applies to all crawlers.

Generating robots.txt by framework

Next.js (App Router)

Create app/robots.ts — auto-serves at /robots.txt:

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: ['/admin/', '/api/'] },
      { userAgent: 'GPTBot', allow: '/' },
      { userAgent: 'ClaudeBot', allow: '/' },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}

WordPress

Use Yoast SEO or Rank Math — both generate robots.txt automatically and provide UI to add custom rules. The generated file lives in the database (not the filesystem) and is served dynamically at /robots.txt.

Remix

Create a resource route at app/routes/robots[.]txt.tsx:

export function loader() {
  const content = `User-agent: *\nDisallow: /admin/\n\nSitemap: https://example.com/sitemap.xml`
  return new Response(content, { headers: { 'Content-Type': 'text/plain' } })
}

Astro

Place a robots.txt file in public/robots.txt — Astro serves everything in /public as static files at the root.

Common robots.txt mistakes

Blocking the entire site

User-agent: *
Disallow: /

This happens during development (blocking staging from indexing) and accidentally ships to production. The result: Googlebot stops crawling, and rankings collapse over days to weeks as pages age out of the index. Always check robots.txt on every deploy.

Blocking CSS and JavaScript

User-agent: *
Disallow: /wp-content/
Disallow: /assets/
Disallow: /*.js$

Prevents Google from rendering your pages. Pages appear broken in Google's mobile-first index: missing layout, missing content that loads via JavaScript, wrong viewport rendering. Fix: allow all CSS and JS files.

No Sitemap directive

The most common omission. Fix: add Sitemap: https://yourdomain.com/sitemap.xml to the last line.

Using wildcards incorrectly

Disallow: /products/*?sort= only blocks URLs matching that exact pattern. A URL like /products/shoes?sort=price is blocked; /shop?sort=price is not. Wildcards (*) match any sequence of characters, and $ anchors to the end of the URL. Test your patterns carefully.

Accidentally blocking via wildcard Disallow during development

A development Disallow: / that ships to production is a common catastrophe. If you use robots.txt to block staging, keep a separate version for production — or use environment variables to conditionally block only in non-production environments.

How to test your robots.txt

1. Fetch it directly: visit https://yourdomain.com/robots.txt in your browser. Confirm the content is correct and it returns 200.

2. Google's robots.txt Tester: Google Search Console → Settings → robots.txt Tester. Enter your rules and test specific URLs against them to see if they'd be blocked or allowed.

3. DeepSEOAnalysis robots.txt tester: paste your robots.txt and test specific URLs or user-agents against it — including AI crawlers.

4. AI crawler access checker: check whether GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers can access your site.

5. Full SEO audit: the technical category checks robots.txt accessibility, sitemap directive presence, AI crawler access, and any common syntax errors.

---

Frequently asked questions

Does robots.txt prevent pages from appearing in search results?

No. robots.txt prevents crawling — it doesn't prevent indexing. Google can still index a URL that it can't crawl if that URL has inbound links from other sites. To prevent a page from appearing in search results, use <meta name="robots" content="noindex"> on the page itself. The noindex tag is respected even if robots.txt allows crawling.

What happens if my robots.txt returns a 404?

Google treats a 404 robots.txt as "no restrictions" — equivalent to an empty robots.txt with User-agent: * / Allow: /. It's not a crawl error, but you lose the Sitemap directive, meaning crawlers must find the sitemap independently. Fix: ensure /robots.txt returns a 200 with valid content.

Can I have multiple User-agent groups?

Yes. Each group applies to the specified user-agents. More specific rules (for a specific bot) override the wildcard User-agent: * rules for that bot. Order matters within a group but not between groups.

Should I block GPTBot and other AI crawlers?

That depends on your goals. Blocking AI crawlers reduces training data contributions, but it also reduces your chance of being cited in AI answers (ChatGPT, Claude, Perplexity, Google AI Overviews). For most sites seeking organic visibility — including AI-search visibility — allowing AI crawlers is the better choice. The AI visibility audit checks whether your site is accessible to the major AI crawlers and scores your AI citation readiness across five signals.