The Group Leak

Three weeks ago we published our three-tier bot access model: open, gated, baited. The response was consistent. People implemented it. They wrote the robots.txt groups, applied the noai tags, and moved on.

Then we started scanning their sites. A tier you set is not a tier that holds.

There is one rule in the robots exclusion protocol that breaks more bot strategies than every bad actor combined. It is not obscure. It is written into the standard. Almost nobody accounts for it.

Groups do not inherit.

The moment you name a crawler in your robots.txt, that crawler stops reading the rest of your file. Not just the * block. Everything outside its own group. Every rule you wrote for everyone, every path you meant to protect site-wide, becomes invisible to the exact crawlers you cared enough about to name.

A group leak is a path your robots.txt protects from one set of crawlers and hands to another, because the rule that protects it lives in a group the second set never reads. It is not a misconfiguration. The file is valid, the crawlers are compliant, and the policy inverts anyway.

Here is one we found in production.

Do robots.txt Groups Inherit From the Wildcard Group?

No. They do not. This is the entire mechanism, so it is worth being precise about it.

RFC 9309, the 2022 standard that formalized robots.txt, is specific about how a crawler picks its instructions. A crawler finds the group whose user-agent line most specifically matches its own name. It obeys that group. It ignores every other group in the file.

Not merges. Ignores.

So a file with a * group and seven named groups is not one policy with seven exceptions. It is eight separate, unrelated policies that happen to live in the same text file. A rule in one group has no bearing on any other.

This is the opposite of how every operator we have talked to assumes it works. The mental model is CSS: a base rule, then overrides. The reality is a switch statement with no fallthrough.

The consequence is mechanical and predictable. Every path you disallowed in * and forgot to repeat in a named group is wide open to that named crawler. Every path you disallowed in a named group and left out of * is wide open to everyone else.

Both failures usually happen in the same file. In opposite directions.

What the Operator Wrote

We scanned a B2B security software site whose team had done the work. They had read the guidance. They had a real tiered policy, not a default file. Their robots.txt had eight groups: a * block plus seven named crawlers.

The relevant parts, with the site details removed:

User-agent: *
Allow: /
Disallow: /admin
Disallow: /api/auth
Disallow: /lp
Disallow: /demo-custom
Disallow: /*/thank-you

Sitemap: https://example.com/sitemap.xml

# AI training & answer engine crawlers: allow content, block conversion pages
User-agent: GPTBot
Allow: /
Disallow: /admin
Disallow: /api/auth
Disallow: /demo
Disallow: /demo-custom
Disallow: /*/thank-you

That named block repeats for ChatGPT-User, ClaudeBot, PerplexityBot, and CCBot. Applebot-Extended gets a shorter version. Bytespider gets Disallow: /.

Read the comment. The intent is unambiguous and it is a good intent: allow content, block conversion pages. This is the gated tier from our framework, implemented by someone who understood the assignment.

Now read the two files against each other. The * group disallows /lp, the landing page directory. The named AI groups do not. The named AI groups disallow /demo, the conversion page. The * group does not.

Neither omission is visible when you read the file top to bottom. Both are catastrophic.

What the Crawlers Actually Saw

We tested 22 named crawlers against every path. Two paths inverted completely.

/lp, the landing pages. The operator disallowed this once, in *, expecting it to apply to everyone.

Sixteen crawlers blocked. Six allowed. The six allowed are the six the operator named. The landing pages are hidden from Google and open to OpenAI, Anthropic, Perplexity, Common Crawl, and Apple's training crawler.

The effort of writing named groups is what created the hole.

/demo, the conversion page. The operator disallowed this in the five named AI groups, which is where the comment says the policy lives.

Exactly inverted. Six blocked, sixteen allowed.

The Sibling Problem

Look closer at the /demo row, because this is where the group leak stops being a technicality and starts costing money.

Blocked from /demo: GPTBot, ClaudeBot, PerplexityBot.

Allowed on /demo: OAI-SearchBot, Claude-User, Claude-SearchBot, Perplexity-User, anthropic-ai.

Those are siblings. Same vendors. Different jobs.

GPTBot and ClaudeBot and PerplexityBot are the training and indexing AI crawlers. They fill corpora. They are the ones every article tells you to block, including ours.

OAI-SearchBot, Claude-User, Claude-SearchBot, and Perplexity-User are the live retrieval agents. They fetch a page because a human is sitting in front of a chat window right now, waiting for an answer about your product.

The operator blocked the crawlers that read the page months from now and left open the crawlers that read the page during a live buying conversation. The blocking was backwards on the axis that matters, and it was backwards because of a naming convention, not a decision. Nobody chose this. The file chose it.

There is no version of a bot tier strategy that survives this. Your gated tier is not gated if the sibling walks in. Your baited page is not baited if the live agent reads the solution the training crawler was blocked from.

Vendors ship new agent names continuously. Every new sibling name is a new group your file does not have, defaulting to whatever * says, which is usually the opposite of what you intended for that vendor.

Why a Group Leak Never Shows Up in Your Logs

Every tool we know of reports robots.txt as a document. It fetches the file, confirms it parses, confirms it returns 200, and tells you it is valid.

It is valid. That is the problem. A group leak is a perfectly valid file that says different things to different readers.

The three standard ways operators check their work all miss it:

Reading the file. You read it as a policy with sections. The crawler reads one section and discards the rest. Eight groups by six paths is 48 decisions, and your eye follows the paths while the crawler follows the groups.

Testing with one user agent. Almost every robots tester defaults to Googlebot. Googlebot matches * on this site. Test with Googlebot and /lp comes back blocked, which is the answer the operator expected, from the only crawler that was never the concern.

Watching server logs. A group leak produces no errors. Nothing is misconfigured. The allowed crawler requests the page and gets a clean 200, exactly as instructed. The signal you would need is the absence of a request from a crawler you forgot you had blocked, and absence does not appear in logs.

This is why "my robots.txt is not working" is almost always the wrong diagnosis. The file works. It is doing exactly what it says to exactly the crawler that read it. What is not working is the assumption that it said the same thing to everyone.

The only way to see a group leak is to resolve every crawler against every path independently and compare the answers across the grid. Twenty-two crawlers against the paths that matter is several hundred separate evaluations, each one requiring correct group-specificity matching. That is not eyeball work.

Two More Ways a Tier Fails Quietly

The group leak was not the only silent failure on that site, and in our scans so far it is not even the most common one.

The sitemap contradicts the file. That same site had 106 URLs sitting in its XML sitemap while robots.txt disallowed them. The sitemap says index this. The file says do not read it. Both are published, both are authoritative, and the crawler resolves the conflict on its own terms rather than yours. Included in the 106: the homepage, the blog index, the contact page, and the entire glossary. The team had shipped a sitemap generator and a robots policy in different quarters and never diffed one against the other.

The file itself is unreachable. On two of the sites we have scanned, robots.txt returned HTTP 403. The document that governs crawler access was forbidden to crawlers. A crawler that cannot read your rules does not pause to ask. It proceeds under its own default, which for most vendors means proceed.

There is a third pattern worth naming even though it is a different article: pages that return a clean 200 to a crawler and contain nothing a crawler can parse. Empty or missing <noscript> fallbacks showed up on three of the eight sites we have scanned. Access granted, content invisible. Blocked and unreadable produce identical outcomes in an AI answer, and only one of them shows up in your access controls.

The Fix Is Boring

The remediation for a group leak is not clever. It is bookkeeping.

Repeat the full ruleset in every named group. Every path in your * block belongs in every named block, plus that group's specific additions. Redundancy is the correct design here. The protocol gives you no inheritance, so you write it out.

Or delete the named groups. If your policy is the same for all AI crawlers, express it once in * and name nobody. One group cannot leak. Most sites we scan would be better served by this than by the named groups they have.

Enforce sibling parity. Decide policy per vendor, then apply it to every agent that vendor ships. If ClaudeBot is blocked from a path, Claude-User and Claude-SearchBot and anthropic-ai are blocked from that path. If you actually want the split, and there are good reasons to want it, make it a decision you wrote down rather than a side effect of the groups you happened to name.

Diff the sitemap against the file. Every quarter, and after any CMS or sitemap plugin update. These two artifacts are usually owned by different people and never compared.

Remember that robots.txt is not your only crawler-facing surface. The robots meta tag and the X-Robots-Tag header govern indexing and training separately from access, and llms.txt is a third file with its own conventions. A group leak is specific to robots.txt group matching, but a tier expressed in one surface and contradicted in another fails the same quiet way.

None of that is hard. All of it depends on knowing which of your paths are leaking to which crawlers, and that is the part you cannot do by reading.

Common Questions

Why is my robots.txt not working for some crawlers?

In most cases it is working, and the rule you are looking for is in a group that crawler never read. A crawler obeys the single group that best matches its name and ignores the rest, so a Disallow written only under User-agent: * has no effect on any crawler you named elsewhere in the file. Check which group each crawler matched before assuming the file is broken.

Do I need to name AI crawlers in robots.txt at all?

Usually not, and naming them is what creates the exposure. If your policy is the same for every AI crawler, write it once under * and name nobody. A single group cannot leak. Name a crawler only when you genuinely want a different policy for it, and then repeat every wildcard rule inside that group.

How do I check which crawlers can actually reach a path?

You need the answer resolved per crawler, per path, group by group, because a single-agent test tells you only about the group that one crawler matched. Most testers default to Googlebot, which usually falls through to * and returns the answer you expected. Our free AI visibility scan does this across 22 named crawlers and shows you where they disagree.

What We Cannot Tell You From Here

We can tell you the rule. We can show you a real file where the rule inverted a well-intentioned policy twice in opposite directions. We can tell you that the sibling split is the failure mode with a direct revenue consequence, because live retrieval agents fetch pages during active buying conversations.

We cannot tell you whether it is happening on your site. That answer is specific to your file, your paths, and the current roster of agent names, and it is 22 crawlers wide.

That is what the scan is for. It resolves every crawler against every path independently, group by group, without assuming inheritance, and shows you the grid: where your crawlers agree, where siblings diverge, and which paths are open to readers you meant to exclude. No score. No login. No email.

Run the Scan

Point it at any public URL and read the matrix. If every row agrees, you have no group leak and you can stop thinking about this. If two rows from the same vendor disagree, you have found the thing your logs will never show you.

Run a free AI visibility scan. Then, if the grid raises more questions than it answers, talk to us about GEO and what generative engine optimization actually involves once your access layer is telling the truth.

The bots are following your rules. The rules are just not the ones you think you wrote.