Robots Meta Tag

\ro-bots met-uh tag\n.

Filed underSEO

In brief · quick answer

A robots meta tag is an HTML directive that tells search engine crawlers how to index and interact with a specific page, such as noindex, nofollow, or nosnippet.

§ 1 Definition

The robots meta tag is a piece of HTML code in a page's `<head>` section that gives page-level instructions to search engine crawlers. Common directives include `noindex` (do not include this page in search results), `nofollow` (do not follow links on this page), `nosnippet` (do not show a snippet in SERPs), and `noarchive` (do not cache this page). Unlike robots.txt, robots meta tags are respected by crawlers that reach the page itself.

§ 2 robots.txt vs Meta Robots

This is the most common confusion. robots.txt blocks crawling (the bot never reaches the page). Meta robots blocks indexing (the bot crawls the page but is told not to index it or follow its links). If you use noindex in meta robots without blocking the page in robots.txt, Googlebot will still crawl it, see the directive, and remove it from the index.

§ 3 The X-Robots-Tag Alternative

For non-HTML resources like PDFs, images, or videos, use the X-Robots-Tag HTTP header instead of a meta tag. It supports the same directives and is set server-side. This is especially useful for preventing PDF files from appearing in search results.

§ 4 Note

The `nofollow` in meta robots applies to ALL links on the page, not individual links. For per-link control, use the `rel="nofollow"` attribute on individual anchor tags.

§ 5 In code

<!-- Prevent indexing and link following -->
<meta name="robots" content="noindex, nofollow">

<!-- Allow indexing, do not show snippet -->
<meta name="robots" content="nosnippet">

<!-- X-Robots-Tag for PDFs (set in .htaccess or server config) -->
<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

Key takeaways

Meta robots controls indexing at the page level
robots.txt controls crawling, not indexing
Use X-Robots-Tag for PDFs and non-HTML files

How Atomic Glue helps

Atomic Glue builds SEO strategies that turn technical fundamentals into measurable rankings. Get in touch to discuss your SEO & GEO services needs.

Get in touch

# Robots Meta Tag

A robots meta tag is an HTML directive that tells search engine crawlers how to index and interact with a specific page, such as noindex, nofollow, or nosnippet.

Category: SEO

Author: Atomic Glue SEO & GEO Team

## Definition

## robots.txt vs Meta Robots

This is the most common confusion. **robots.txt** blocks crawling (the bot never reaches the page). **Meta robots** blocks indexing (the bot crawls the page but is told not to index it or follow its links). If you use noindex in meta robots without blocking the page in robots.txt, Googlebot will still crawl it, see the directive, and remove it from the index.

## The X-Robots-Tag Alternative

## Note

The `nofollow` in meta robots applies to ALL links on the page, not individual links. For per-link control, use the `rel="nofollow"` attribute on individual anchor tags.

## In code

<!-- Prevent indexing and link following -->
<meta name="robots" content="noindex, nofollow">

<!-- Allow indexing, do not show snippet -->
<meta name="robots" content="nosnippet">

<!-- X-Robots-Tag for PDFs (set in .htaccess or server config) -->
<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

## Key takeaways

Meta robots controls indexing at the page level
robots.txt controls crawling, not indexing
Use X-Robots-Tag for PDFs and non-HTML files

## Related entries

Last updated June 2026. Permalink: atomicglue.co/glossary/robots-meta-tag