Robots.txt
Robots.txt is a text file placed at the root of a website that instructs search engine crawlers which URLs they may or may not access.
§ 1 Definition
Robots.txt is the first thing Googlebot checks when it arrives at a site. It is a plain text file at `https://example.com/robots.txt` that uses a simple syntax to allow or disallow crawling of specific paths. It is a crawl control mechanism, not a security tool. Blocked URLs are still accessible to anyone who knows the URL; robots.txt only stops bots that choose to obey it.
§ 2 Common Robots.txt Directives
Use `User-agent` to specify which bot the rule targets (`*` means all bots). Use `Disallow` to block paths. Use `Allow` to override a broader Disallow (for specific files within a blocked directory). Use `Sitemap` to point to your XML sitemap location. Use `Crawl-delay` to suggest a crawl rate (though Googlebot ignores this in favor of actual server response times).
§ 3 The Robots.txt Trap
The most dangerous robots.txt mistake is accidentally blocking Googlebot from crawling your entire site. A `Disallow: /` rule for all user-agents means Googlebot cannot crawl anything. A `Disallow:` rule with no path means nothing is blocked. This distinction has tripped up many site migrations.
§ 4 Note
§ 5 In code
User-agent: *
Disallow: /wp-admin/
Disallow: /private/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml- Robots.txt controls crawling, not indexing
- A single misconfigured rule can block entire site indexing
- Use both robots.txt (crawl control) and meta robots (index control)
Atomic Glue builds SEO strategies that turn technical fundamentals into measurable rankings. Get in touch to discuss your SEO & GEO services needs.
Get in touchRobots.txt is a text file placed at the root of a website that instructs search engine crawlers which URLs they may or may not access.
Category: SEO
Author: Atomic Glue SEO & GEO Team
## Definition
Robots.txt is the first thing Googlebot checks when it arrives at a site. It is a plain text file at `https://example.com/robots.txt` that uses a simple syntax to allow or disallow crawling of specific paths. It is a crawl control mechanism, not a security tool. Blocked URLs are still accessible to anyone who knows the URL; robots.txt only stops bots that choose to obey it.
## Common Robots.txt Directives
Use `User-agent` to specify which bot the rule targets (`*` means all bots). Use `Disallow` to block paths. Use `Allow` to override a broader Disallow (for specific files within a blocked directory). Use `Sitemap` to point to your XML sitemap location. Use `Crawl-delay` to suggest a crawl rate (though Googlebot ignores this in favor of actual server response times).
## The Robots.txt Trap
The most dangerous robots.txt mistake is accidentally blocking Googlebot from crawling your entire site. A `Disallow: /` rule for all user-agents means Googlebot cannot crawl anything. A `Disallow:` rule with no path means nothing is blocked. This distinction has tripped up many site migrations.
## Note
Robots.txt blocks crawling but does not prevent indexing. If another site links to a blocked URL, Google may still index it (just without crawling the content). For actual index blocking, use a `noindex` meta tag or X-Robots-Tag header. This is one of the most misunderstood distinctions in technical SEO.
## In code
User-agent: * Disallow: /wp-admin/ Disallow: /private/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap.xml
## Key takeaways
- Robots.txt controls crawling, not indexing
- A single misconfigured rule can block entire site indexing
- Use both robots.txt (crawl control) and meta robots (index control)
## Related entries
- [Robots Meta Tag](atomicglue.co/glossary/robots-meta-tag)
- [Crawling](atomicglue.co/glossary/crawling)
- [XML Sitemap](atomicglue.co/glossary/xml-sitemap)
- [Indexing](atomicglue.co/glossary/indexing)
Last updated June 2026. Permalink: atomicglue.co/glossary/robots-txt