TECHNICAL · SEO GLOSSARY
X-Robots-Tag
An HTTP response header that provides the same crawling and indexing directives as the robots meta tag, but without requiring HTML access — making it the only noindex method that works for non-HTML files like PDFs, images, and XML documents.
Definition
The `X-Robots-Tag` is an HTTP response header that provides crawling and indexing directives equivalent to the `<meta name="robots">` HTML tag, but delivered at the HTTP header level rather than in the page\'s HTML. The header format: `X-Robots-Tag: [directive]`. It supports all the same directives as the robots meta tag: `noindex`, `nofollow`, `nosnippet`, `max-snippet: N`, `noimageindex`, etc. Multiple directives can be combined: `X-Robots-Tag: noindex, nofollow`. Bot-specific targeting is also supported by including the bot name: `X-Robots-Tag: googlebot: noindex`. **Key use cases where X-Robots-Tag is preferred over the meta tag**: (1) **Non-HTML resources** — PDFs, Word documents, images, and other non-HTML files cannot contain an HTML `<head>` section and therefore cannot use the meta robots tag. The X-Robots-Tag is the only way to apply noindex or nosnippet to a PDF or image served by the web server. (2) **Server-level bulk application** — applying noindex to an entire directory or subdomain without modifying each page\'s template. Server configuration (nginx, Apache) can apply X-Robots-Tag headers to all responses matching a path pattern: `location /search/ { add_header X-Robots-Tag "noindex"; }` noindexes all search result pages at the server level without HTML changes. (3) **CDN/proxy-level application** — Cloudflare Workers or CDN edge configurations can inject X-Robots-Tag headers based on URL patterns or response content, allowing noindex/nosnippet application without origin server changes. (4) **Headless/API architectures** — for headless CMS setups where page-level meta tag injection requires changes to the frontend framework, X-Robots-Tag headers can be applied via server or CDN configuration without touching the rendering layer.
Why it matters for SEO
The X-Robots-Tag is particularly important for two scenarios that the meta robots tag cannot handle. First, non-HTML files: if a site has PDFs accessible via direct URLs (e.g., /reports/annual-report-2026.pdf) that shouldn\'t appear in search results, the X-Robots-Tag is the only valid noindex mechanism — there is no HTML head to add a meta tag to. Without an X-Robots-Tag noindex, these PDFs will be indexed and may appear in search results. Second, bulk application without HTML access: for sites where the frontend template is managed by a third party or CMS platform that doesn\'t allow custom meta tag injection, server-level X-Robots-Tag headers provide a bypass. The header is processed by Googlebot before the HTML is parsed — it takes effect immediately when Google crawls the page, with no dependency on HTML parsing or JavaScript execution.
How DeepSEOAnalysis checks this
The audit inspects HTTP response headers for every crawled URL, not just the HTML body. X-Robots-Tag headers are extracted and evaluated alongside meta robots tags — the audit uses whichever signal is more restrictive (an X-Robots-Tag noindex combined with a meta tag `index` results in the page being treated as noindexed). Specific checks: X-Robots-Tag on HTML pages conflicting with meta robots (e.g., X-Robots-Tag noindex on a page whose meta tag says index — the header takes precedence); X-Robots-Tag on PDF files linked from the site (flagged if the PDF has been submitted to the sitemap but has an X-Robots-Tag noindex — sitemap inclusion and noindex are contradictory); and X-Robots-Tag applied to the sitemap XML itself (a sitemap at /sitemap.xml with `X-Robots-Tag: noindex` would prevent Google from processing the sitemap — a critical configuration error).
Useful tools and resources
GLOSSARY
Related terms
technical
Noindex Tag
A directive (`<meta name="robots" content="noindex">` or `X-Robots-Tag: noindex` HTTP header) that tells search engines not to include a page in their index — preventing it from appearing in search results regardless of its link equity or content quality.
Read definition →technical
Robots Meta Tag
An HTML `<meta name="robots">` tag that tells crawlers whether to index a page, follow its links, or display snippets — more granular than robots.txt.
Read definition →technical
Robots.txt
A text file at the root of a domain that tells crawlers which pages or sections to access or avoid.
Read definition →technical
PDF SEO
Optimising PDF files so Google can crawl, index, and rank them in search results — including descriptive file names, searchable (not scanned) text content, metadata (Title, Author, Description), internal links within PDFs, and ensuring PDFs are served at crawlable URLs without unnecessarily consuming crawl budget.
Read definition →See how your site scores on X-Robots-Tag.
The free DeepSEOAnalysis audit checks x-robots-tag and 100+ other signals. Full report, no signup.
Run a free audit →