llms.txt: What It Is, Who Respects It, and How to Create One

If you have been tracking AI search visibility, you have probably seen llms.txt mentioned alongside robots.txt and sitemap.xml. It is not an official standard — no RFC, no W3C spec — but it is spreading quickly because it solves a real problem: giving AI crawlers a fast, citable summary of what your site contains and what it is good for.

This post explains what llms.txt is, what should go in it, which AI systems actually read it, and how to create one in about five minutes.

What is llms.txt, and why does it exist?

llms.txt is a plain-text Markdown-style file placed at the root of a domain (e.g., https://example.com/llms.txt). The convention was proposed by Jeremy Howard of fast.ai in September 2024. The core idea: give large language models and AI crawlers a single document they can read to understand a site's purpose, key pages, and content quality before committing crawl budget.

The problem it addresses is real. A modern website can have hundreds of pages — product pages, blog posts, API docs, login flows, error pages — and an AI crawler with limited context has no clear way to know which pages are authoritative and worth citing. robots.txt tells crawlers what they cannot access. sitemap.xml lists URLs but has no natural language descriptions. llms.txt fills that gap with a human-readable (and LLM-readable) digest.

At its simplest, it looks like this:

# DeepSEOAnalysis

> Free SEO and AI-visibility audits for any public URL. No signup required.

## Key pages

- [Homepage](https://deepseoanalysis.com/): Run a free SEO + GEO audit
- [Methodology](https://deepseoanalysis.com/methodology/): How the 100+ checks are scored
- [AI Visibility methodology](https://deepseoanalysis.com/methodology/ai-visibility/): GEO check details
- [Blog](https://deepseoanalysis.com/blog/): SEO and AI-search guides

That's enough for an AI system to understand what the site offers without crawling it.

What should a llms.txt file contain?

The spec (such as it is) recommends four sections, in order:

1. H1 title — the site or product name.
2. Blockquote summary — one to two sentences describing what the site is and who it serves.
3. H2 sections — groups of links to key pages, each with a short description of what the page covers.
4. (Optional) llms-full.txt reference — a link to a longer version for AI systems that want more detail.

A few principles that matter in practice:

• Descriptions should be factual and citable, not marketing copy. Write "DeepSEOAnalysis scores 100+ checks across technical SEO, Core Web Vitals, structured data, and AI-readiness" not "The most powerful SEO platform in the world."
• Link to your most authoritative and unique content first. Tool pages, research pages, and methodology documents are higher-value than generic product pages.
• Keep the file short. The goal is a fast read for a system with limited context, not a content dump. If you need more, link to an /llms-full.txt.
• Update it when important pages change. A llms.txt that links to a 404 or a stale page hurts more than it helps.

Which AI systems actually read llms.txt?

This is the most common question — and the honest answer is: some do, some don't, and it is evolving.

As of mid-2026, the following AI crawlers have published documentation or confirmed behavior indicating they use or read /llms.txt:

• Perplexity — their crawler (PerplexityBot) is documented to look for llms.txt as a site context signal.
• Anthropic — ClaudeBot has documented support for the convention.
• OpenAI — GPTBot follows the file when present, though the exact weighting is not publicly confirmed.

Google's crawlers (Googlebot, Google-Extended) have not announced support. Google AI Overviews are driven by traditional crawl signals, schema, and E-E-A-T — not by a separate AI-specific convention. That said, llms.txt does not conflict with Google's signals, so there is no downside to having one.

The critical point: AI citation readiness is not just about llms.txt. It is about the combination: llms.txt signals your intent, robots.txt must allow AI bots access, FAQPage and HowTo schema help AI systems extract citable answers, and your content structure (answer-shaped paragraphs, short sections, question-led headings) determines how often a page actually gets cited. Our AI visibility methodology covers all five dimensions.

In our own data across hundreds of audits, roughly 30% of sites have AI bots explicitly blocked in robots.txt — usually because of catch-all Disallow: / rules that predate the AI-crawler era. Blocking GPTBot or ClaudeBot while having a llms.txt is contradictory: you are inviting AI systems to understand your content while also preventing them from accessing it.

How does llms.txt fit alongside robots.txt and sitemap.xml?

The three files are complementary, not redundant:

| File | Who reads it | What it controls | |---|---|---| | robots.txt | All crawlers | Access permissions — what can and cannot be fetched | | sitemap.xml | Search engines | URL inventory for indexing | | llms.txt | AI crawlers and LLMs | Content identity, intent, and key pages for context |

Think of robots.txt as the door policy, sitemap.xml as the room-by-room floor plan, and llms.txt as the concierge card that says "the thing you should know about this place is X, and these are the rooms worth visiting."

You can check whether your robots.txt is blocking AI crawlers with our robots.txt tester. If GPTBot, ClaudeBot, or PerplexityBot are blocked, fix that before optimizing your llms.txt — a well-written context file is useless if the crawler cannot access the content it points to.

What does a well-structured llms.txt look like?

Here is a template for a typical SaaS or content site:

# [Site Name]

> [One-sentence factual description of what the site does and who it serves.]

## Tools

- [Tool name](URL): [What it does in one sentence]
- [Tool name](URL): [What it does in one sentence]

## Documentation

- [Key doc page](URL): [What it covers]
- [Key doc page](URL): [What it covers]

## Blog

- [Article title](URL): [What the article covers]

## About

- [About/Methodology page](URL): [What makes the site authoritative on its topic]

## Optional: Extended content

- [llms-full.txt](URL): Full site index for AI systems that want more detail

For tool-heavy sites, the Tools section matters most — it tells AI systems what unique capabilities exist that would be worth recommending to users who ask for them. For content sites (newsletters, blogs, research), the Blog and About sections are higher priority.

How do you create and publish an llms.txt file?

Step 1: Write or generate the file. Use our free llms.txt generator to produce a correctly formatted file in under five minutes. Add your site name, a one-sentence description, and your key page URLs with short descriptions. The generator outputs a file you can download or copy.

Step 2: Publish it at /llms.txt. The file must live at the root of your domain: https://yourdomain.com/llms.txt. In most static site frameworks, this means adding llms.txt to your public/static directory. In Next.js, you can add a app/llms.txt/route.ts that returns the file content with Content-Type: text/plain.

Step 3: Verify AI crawlers can access it. Run our robots.txt tester and confirm GPTBot, ClaudeBot, and PerplexityBot are not blocked. Then manually fetch https://yourdomain.com/llms.txt to confirm it returns a 200 and the correct content.

Step 4: Include it in your full AI visibility audit. llms.txt is one of five dimensions we check in the AI visibility score at DeepSEOAnalysis. The audit also checks AI crawler access, FAQPage/HowTo schema, question-led headings, and content chunkability. A high GEO score means all five are in order — not just the file.

Step 5: Keep it updated. Whenever you launch a major new page, add a research study, or publish a tool, update the llms.txt to include it. Set a monthly reminder or automate generation from your sitemap.

---

FAQ

Does llms.txt improve Google rankings?

No. Google does not use llms.txt as a ranking signal. It is specifically for AI crawlers and large language models. Google AI Overviews are driven by standard crawl signals, structured data, and page quality — not this convention.

What is the difference between llms.txt and llms-full.txt?

llms.txt is the short version — a quick-read context file. llms-full.txt is an optional longer version with more detailed page descriptions, full content excerpts, or a complete URL inventory. The short file references the full file if it exists. Not every site needs a full version; start with the short one.

Can llms.txt block AI systems from training on my content?

No. llms.txt signals intent but does not enforce access control. To restrict AI training access, you need to use robots.txt with the appropriate user-agent directives (e.g., User-agent: GPTBot / Disallow: / if you want to block OpenAI's crawler entirely). The llms.txt convention is designed to help AI systems, not restrict them.

How often should I update my llms.txt?

Whenever significant content changes: new tools, new research, major new articles, or changes to your core product description. A stale llms.txt that links to outdated pages sends bad signals. Monthly reviews are a reasonable minimum; automated generation from a sitemap is better.

Is llms.txt an official standard?

Not yet. It is a community proposal from Jeremy Howard (fast.ai) that has gained significant adoption. There is no IETF RFC or W3C spec. That said, the convention is widely adopted and actively supported by Anthropic, Perplexity, and OpenAI's crawler infrastructure. Think of it like robots.txt before it became an official standard: practically universal, not formally ratified.