Meta Robots Tag

Definition

The meta robots tag (formally `<meta name="robots" content="...">`) is a page-level directive that controls search engine crawler behavior for a specific page. The two primary directives: **index / noindex** — controls whether the page should be included in the search engine\'s index; **follow / nofollow** — controls whether the crawler should follow the links on the page and pass link equity. These directives can be combined: `<meta name="robots" content="noindex, nofollow">` tells crawlers not to index the page and not to follow its links. The most common practical uses: `noindex` to exclude thin pages, staging environments, search result pages, and private user content from the index; `nofollow` (rarely needed at page level) to prevent PageRank distribution from pages with many external links. The meta robots tag is distinct from the robots.txt file — robots.txt controls crawler access (whether Googlebot can fetch the URL at all), while the meta robots tag controls what Googlebot does with the page after fetching it. A noindex directive in a meta robots tag must be visible in the initial server-rendered HTML for Googlebot to see it — a noindex injected only by client-side JavaScript may be missed if Googlebot processes the page before JavaScript execution completes.

Why it matters for SEO

Incorrect meta robots configuration is one of the most common technical SEO errors. A noindex tag on a production page immediately prevents that page from appearing in search results, regardless of its content quality or inbound links. Common accidents: staging environments with noindex deployed to production; CMS default settings that add noindex to category/tag archives; A/B test variants with noindex that persist after the test ends; plugin or theme updates that reset robots settings. Every page you want to rank must have no noindex directive (either absence of the meta tag, or an explicit index directive). Equally, pages you want excluded (thin pagination, parameter-based duplicates, internal search results) must be consistently noindexed.

How DeepSEOAnalysis checks this

DeepSEOAnalysis checks every crawled page for meta robots tag presence and content, flagging noindex directives that may be unintentional (on content pages), missing noindex directives on pages that should be excluded (thin pagination, duplicate parameter pages), and conflicts between robots.txt allow directives and meta robots noindex (both can coexist — Googlebot can fetch the page but will not index it).