TECHNICAL SEO · SEO GLOSSARY
HTTP Headers for SEO
SEO-relevant directives sent in HTTP response headers rather than in HTML: X-Robots-Tag for noindex/nofollow on non-HTML resources, Link header for canonical tags, and cache-control for crawl efficiency.
Definition
HTTP response headers can carry SEO-relevant directives that apply before or instead of HTML meta tags. The most important SEO HTTP headers: **X-Robots-Tag** — the header equivalent of the meta robots tag. It accepts the same directives (noindex, nofollow, noarchive, etc.) but can be applied to any resource type, including PDFs, images, and other non-HTML files that can\'t contain HTML meta tags. For example, a response header `X-Robots-Tag: noindex` prevents a PDF from being indexed. **Link: rel="canonical"** — canonical tag as HTTP header, useful for non-HTML resources. Format: `Link: <https://example.com/canonical-url>; rel="canonical"`. Google accepts canonical tags both in HTML `<head>` and in the HTTP Link header. **Last-Modified** — tells Googlebot when the page was last changed, improving crawl schedule efficiency (Googlebot can send conditional GET requests and skip unchanged pages). **Cache-Control** — affects how long Googlebot caches crawled content; extremely long cache times may delay recrawl of updated pages. **Content-Type** — specifies the character encoding (UTF-8) and content type; a missing or incorrect Content-Type can cause Googlebot to misparse page content.
Why it matters for SEO
HTTP headers provide SEO control for resources that have no HTML and therefore can\'t use meta tags. A PDF white paper, a dynamically generated sitemap, or a JavaScript file can\'t include a meta robots tag — but they can have an X-Robots-Tag header. Additionally, for sites with conditional HTTP headers (set by CDN rules or server configuration), header-level directives may conflict with HTML meta tags, creating ambiguity — Google uses the most restrictive interpretation when both sources give conflicting signals. X-Robots-Tag set to noindex by a CDN rule, combined with an index meta robots tag in the HTML, results in noindex taking effect.
How DeepSEOAnalysis checks this
DeepSEOAnalysis checks HTTP response headers for SEO-relevant signals during page crawl: X-Robots-Tag values that may conflict with or override HTML meta robots tags, Content-Type headers for correct charset specification, and HTTP status codes. The audit surfaces X-Robots-Tag noindex directives alongside HTML meta robots findings to identify conflicts.
GLOSSARY
Related terms
crawling & indexing
Meta Robots Tag
An HTML meta tag in the page <head> that instructs search engine crawlers whether to index a page and follow its links: index/noindex and follow/nofollow directives.
Read definition →technical
X-Robots-Tag
An HTTP response header that provides the same crawling and indexing directives as the robots meta tag, but without requiring HTML access — making it the only noindex method that works for non-HTML files like PDFs, images, and XML documents.
Read definition →technical
Canonical Tag
A canonical tag (rel="canonical") is an HTML link element that tells search engines which URL is the authoritative version of a page when duplicate or near-duplicate content exists at multiple URLs. It consolidates ranking signals from all duplicate variants to the canonical URL.
Read definition →technical
Crawl Budget
The number of pages Googlebot will crawl on a site within a given timeframe — determined by crawl rate limit and crawl demand.
Read definition →technical
Robots.txt
A text file at the root of a domain that tells crawlers which pages or sections to access or avoid.
Read definition →See how your site scores on HTTP Headers for SEO.
The free DeepSEOAnalysis audit checks http headers for seo and 100+ other signals. Full report, no signup.
Run a free audit →