Group Leak

§ 1 Definition

A group leak is the gap created when a robots.txt rule applies to some crawlers and not others as a side effect of file structure rather than intent. RFC 9309, the standard that formalized the robots exclusion protocol in 2022, requires a crawler to find the user-agent group that most specifically matches its own name, obey that group, and ignore every other group in the file. Groups do not inherit and they do not merge. A file with a wildcard group and seven named groups is not one policy with seven exceptions. It is eight independent policies that happen to share a text file.

§ 2 Why It Happens

Operators read robots.txt the way they read a stylesheet: a base rule, then overrides. Under that mental model, naming GPTBot to add one restriction leaves the wildcard rules intact. Under the actual specification, naming GPTBot removes every wildcard rule from GPTBot’s view. The result is bidirectional. Any path disallowed only in the wildcard group is open to every crawler you named. Any path disallowed only in a named group is open to every crawler you did not name.

§ 3 The Sibling Case

The costliest form of a group leak splits agents from the same vendor. ClaudeBot, Claude-User, and Claude-SearchBot are separate user-agent tokens, as are GPTBot and OAI-SearchBot, and PerplexityBot and Perplexity-User. Naming only the training crawler leaves the live retrieval agent matching the wildcard group, so a page blocked from a training corpus stays fully readable to the agent fetching it mid-conversation for a user who is asking about your product right now. The block lands on the crawler with the longest time horizon and misses the one with immediate commercial consequence.

§ 4 How to Detect It

Detection requires resolving every crawler independently against every path and comparing the answers across the grid. Reading the file will not surface a leak, because the eye follows paths while the crawler follows groups. Single-agent testers will not surface it either, since most default to Googlebot, which usually matches the wildcard group and returns the answer the operator expected. Server logs never surface it: the allowed crawler requests the page and receives a clean 200, exactly as instructed, and the missing signal is the absence of a request that absence does not record.

A group leak is not a misconfiguration and not a compliance failure. The file is valid, the crawlers are obeying it correctly, and nothing errors. That is precisely why it survives audits.

§ 5 Note

§ 6 Common Questions

Q. Do robots.txt groups inherit from the wildcard group?

A. No. RFC 9309 requires a crawler to obey the single user-agent group that most specifically matches its own name and to ignore every other group in the file. Groups do not inherit and they do not merge, so a rule written only in the wildcard group does not apply to any crawler you named.

Q. Why can GPTBot read a page I disallowed for everyone?

A. Because naming GPTBot in its own group removed the wildcard group from GPTBot’s view entirely. If the disallow for that path was written only under User-agent: *, GPTBot never reads it. The fix is to repeat every wildcard rule inside every named group.

Q. Does blocking ClaudeBot also block Claude-User?

A. No. ClaudeBot, Claude-User, and Claude-SearchBot are separate user-agent tokens. Blocking ClaudeBot in its own group leaves the others matching whatever the wildcard group says. The same applies to GPTBot versus OAI-SearchBot, and PerplexityBot versus Perplexity-User.