TECHNICAL · SEO GLOSSARY

Robots Meta Tag

An HTML `<meta name="robots">` tag that tells crawlers whether to index a page, follow its links, or display snippets — more granular than robots.txt.

Definition

The robots meta tag is an HTML element placed in the `<head>` of a page that gives page-level crawling and indexing instructions to search engines. Common directives: `noindex` (don't index this page), `nofollow` (don't follow outbound links), `nosnippet` (don't show a description snippet in the SERP), `noimageindex` (don't index images on this page), `noarchive` (don't show a cached copy), and `max-snippet: N` (limit description snippet to N characters). Directives can be combined: `<meta name="robots" content="noindex, nofollow">`. The `X-Robots-Tag` HTTP header achieves the same effect for non-HTML files (PDFs, images). When robots.txt blocks a page but the meta tag says `noindex`, the `noindex` may never be seen because the crawler was blocked before reaching the page — a common misconfiguration.

Why it matters for SEO

Incorrect robots meta tags are a common source of accidental de-indexation. A `noindex` tag left on a staging environment that gets deployed to production, or a CMS that adds `noindex` to paginated pages, can silently remove pages from Google's index. The converse is also a problem: pages that should be noindexed (thin content, duplicate parameter URLs, internal search results) often lack the tag, leading to index bloat that wastes crawl budget.

How DeepSEOAnalysis checks this

The audit reads the `<meta name="robots">` tag and `X-Robots-Tag` HTTP header on every crawled page. It flags: (1) important pages with a `noindex` directive that are likely intended to be indexed; (2) pages that appear in the sitemap but have `noindex`; (3) pages where robots.txt blocks access but a `noindex` is assumed (the block prevents the `noindex` from being seen); and (4) pages that return `noindex` on some crawls but not others (a caching or dynamic rendering inconsistency).