The finding
22 of 46sites block ClaudeBot, against 16 for GPTBot
ClaudeBot was blocked by 22 of 46 sites and GPTBot by 16. All 16 GPTBot blockers also block ClaudeBot; of the six that block ClaudeBot alone, four never name GPTBot.
On this page 13 sections
- Method
- What the 46 sites say
- Finding 1: training crawlers are blocked more than search and user agents
- Finding 2: every site that blocks GPTBot also blocks ClaudeBot
- Finding 3: two thirds of these sites have an AI policy at all
- Finding 4: fifteen sites block GPTBot, ClaudeBot and CCBot together
- The 31 sites that name an AI agent
- What this does not tell you
- If you run a site, what this implies
- Related reading
- Reproducing this
- Download the data
- Limits
ClaudeBot is the AI crawler these sites block most often. Of 46 well-known sites whose robots.txt we could read, 22 block ClaudeBot and 16 block GPTBot, and every one of those 16 also blocks ClaudeBot. The six that block ClaudeBot alone mostly say nothing about GPTBot: four never name it, one restricts it from some paths, and one names it and allows it.
This is a snapshot of stated policy, read once. It says nothing about which crawlers obey it.
Method
On 1 September 2026, at about 20:24 UTC, we requested https://<host>/robots.txt once from each of 50 well-known domains across news, reference, developer, commerce and health. The client was Python’s urllib, following redirects, with a 12-second timeout and the user agent proxy.wiki-research/1.0 (+https://proxy.wiki/research/). 46 domains returned a readable file, and each of those files was scored for the same 18 AI user agents.
Each file was parsed into user-agent groups, with consecutive User-agent: lines treated as one group sharing the rules that follow. That is what RFC 9309 specifies, and large sites rely on it heavily. For each agent we recorded one of four verdicts:
- blocked: a group naming the agent contains
Disallow: /and noAllow: / - allowed: a group names the agent and either disallows nothing or pairs
Disallow: /withAllow: / - partial: a group names the agent and disallows some paths, but not the whole site
- unmentioned: no group names the agent, so only the wildcard group applies to it
The parser was checked against raw files before the numbers were used. On nytimes.com it reported GPTBot blocked, and the file contains User-agent: GPTBot followed by Disallow: /. On theguardian.com it reported ClaudeBot blocked from a group of at least twenty agents, and that group ends in Disallow: /. GPTBot does not appear anywhere in the Guardian’s file.
What the 46 sites say
| Agent | Operator | Blocked | Partial | Allowed | Unmentioned |
|---|---|---|---|---|---|
| ClaudeBot | Anthropic | 22 | 1 | 1 | 22 |
| CCBot | Common Crawl | 21 | 1 | 2 | 22 |
| Applebot-Extended | Apple | 19 | 1 | 1 | 25 |
| Bytespider | ByteDance | 18 | 1 | 1 | 26 |
| meta-externalagent | Meta | 18 | 1 | 0 | 27 |
| Google-Extended | 17 | 1 | 2 | 26 | |
| GPTBot | OpenAI | 16 | 3 | 3 | 24 |
| PerplexityBot | Perplexity | 16 | 2 | 2 | 26 |
| Amazonbot | Amazon | 16 | 3 | 1 | 26 |
| anthropic-ai | Anthropic (older name) | 15 | 1 | 2 | 28 |
| cohere-ai | Cohere | 14 | 0 | 1 | 31 |
| Diffbot | Diffbot | 13 | 1 | 0 | 32 |
| omgili | Webz.io | 13 | 0 | 0 | 33 |
| Claude-User | Anthropic | 12 | 2 | 0 | 32 |
| ChatGPT-User | OpenAI | 11 | 6 | 1 | 28 |
| Timpibot | Timpi | 9 | 1 | 0 | 36 |
| ImagesiftBot | ImageSift | 8 | 0 | 0 | 38 |
| OAI-SearchBot | OpenAI | 7 | 6 | 1 | 32 |
Every row sums to 46. The operator column is our attribution of each user agent to the company that runs it.
Finding 1: training crawlers are blocked more than search and user agents
The clearest pattern in the table is not about which company is least popular. The same operator gets very different treatment depending on what the agent is for, and the purposes below are the operators’ own descriptions from their crawler documentation.
| Agent | Operator | Stated purpose | Sites blocking it |
|---|---|---|---|
| ClaudeBot | Anthropic | Collects content that may be used for model training | 22 |
| Claude-User | Anthropic | Fetches a page when a Claude user’s question needs it | 12 |
| GPTBot | OpenAI | Makes its foundation models more useful and safe | 16 |
| ChatGPT-User | OpenAI | Certain user actions in ChatGPT and Custom GPTs | 11 |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT search results | 7 |
For both operators, the agents that fetch for a search or for a person are blocked less often than the training crawler. OAI-SearchBot is blocked by 7 sites and ChatGPT-User by 11, against 16 for GPTBot. Claude-User is blocked by 12, against 22 for ClaudeBot. Anthropic’s search agent, Claude-SearchBot, was not one of the 18 agents scored.
That fits publishers who refuse to be training data but still want to be found and cited. The partial column points the same way. OAI-SearchBot and ChatGPT-User each have six sites that restrict some paths rather than the whole site, the highest partial counts in the table. Those sites wrote a considered rule instead of a blanket one.
Finding 2: every site that blocks GPTBot also blocks ClaudeBot
Sixteen sites block GPTBot, and all sixteen block ClaudeBot as well. Six more block ClaudeBot and not GPTBot, and what they say about GPTBot is not one thing:
| Site | ClaudeBot | GPTBot |
|---|---|---|
theverge.com |
blocked | allowed: named and not disallowed |
investopedia.com |
blocked | partial: some paths disallowed |
theguardian.com |
blocked | not named |
washingtonpost.com |
blocked | not named |
arstechnica.com |
blocked | not named |
wired.com |
blocked | not named |
Only one of the six states a decision to let GPTBot in. On the other four, GPTBot falls under the wildcard group, and we did not keep the wildcard verdict for those four sites, so we cannot say how much of each site that leaves open to it.
The Guardian’s file shows why the gap is hard to read as an oversight. Its ClaudeBot rule sits in a group of at least twenty agents that ends in Disallow: /, and GPTBot is not among them or anywhere else in the file. A list that long without GPTBot looks like a choice. The file does not say why.
The comparison is per agent name, so it does not grow with the number of names an operator publishes. Anthropic’s crawler page lists three agents and OpenAI’s lists four. Operator totals would be a different measure, and we have not computed them, because the mapping from agent to operator is our attribution rather than a published fact.
Finding 3: two thirds of these sites have an AI policy at all
31 of 46 sites (67%) name at least one AI agent in robots.txt. Fifteen name none, and are covered only by whatever their wildcard group says.
The wildcard groups themselves are not the story people expect:
Wildcard User-agent: * posture |
Sites |
|---|---|
| Partial: some paths disallowed | 37 |
Blocked: Disallow: / for everyone |
6 |
| No wildcard group at all | 2 |
| Allowed: nothing disallowed | 1 |
Only six of 46 close the door to every crawler. reddit.com is one of them: its file is a wildcard Disallow: / and names no AI agent at all. Elsewhere the AI-specific rules are additions to files that leave most of the site open, aimed at named agents rather than a general retreat from being crawled.
Finding 4: fifteen sites block GPTBot, ClaudeBot and CCBot together
Of the 16 sites that block both GPTBot and ClaudeBot, 15 also block CCBot. The exception is medium.com, which does not name CCBot. That combination, two model developers plus the Common Crawl archive that many other datasets are built from, is the closest thing to a standard posture in the sample. CCBot is blocked by 21 sites in all, more than GPTBot and one fewer than ClaudeBot, which suggests publishers treat it as an upstream source of training data rather than as a neutral archive.
The 31 sites that name an AI agent
Every site in the sample whose robots.txt names at least one AI crawler, ordered by how many of the eighteen agents it blocks outright. ✗ blocked, ✓ allowed, ~ partial, ○ not named. Per-site verdicts were kept for the seven agents shown; for the other eleven, only each site’s total in the last column survives.
| Site | GPTBot | OAI-Search | ClaudeBot | Google-Ext | CCBot | Perplexity | Bytespider | Blocked of 18 |
|---|---|---|---|---|---|---|---|---|
nytimes.com |
✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 17 |
cnn.com |
✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 17 |
sciencedirect.com |
✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | 17 |
bbc.co.uk |
✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 15 |
bloomberg.com |
✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 15 |
amazon.com |
✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 15 |
theverge.com |
✓ | ○ | ✗ | ✗ | ✗ | ✗ | ✗ | 15 |
nature.com |
✗ | ○ | ✗ | ✗ | ✗ | ✗ | ✗ | 15 |
investopedia.com |
~ | ~ | ✗ | ✗ | ✗ | ✗ | ✗ | 14 |
arstechnica.com |
○ | ○ | ✗ | ✗ | ✗ | ✗ | ✗ | 13 |
wired.com |
○ | ○ | ✗ | ✗ | ✗ | ✗ | ✗ | 12 |
washingtonpost.com |
○ | ○ | ✗ | ○ | ✗ | ✗ | ✗ | 11 |
techcrunch.com |
✗ | ○ | ✗ | ✗ | ✗ | ○ | ✗ | 11 |
healthline.com |
✗ | ○ | ✗ | ○ | ✗ | ○ | ✗ | 11 |
yelp.com |
✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ○ | 10 |
figma.com |
✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ○ | 10 |
theguardian.com |
○ | ○ | ✗ | ○ | ✗ | ✗ | ✗ | 9 |
ebay.com |
✗ | ~ | ✗ | ○ | ✗ | ✗ | ✗ | 9 |
tripadvisor.com |
✗ | ~ | ✗ | ✗ | ✗ | ~ | ✗ | 9 |
medium.com |
✗ | ○ | ✗ | ○ | ○ | ○ | ✗ | 6 |
jstor.org |
✗ | ○ | ✗ | ✗ | ✗ | ○ | ○ | 4 |
webmd.com |
✗ | ○ | ✗ | ○ | ✗ | ○ | ○ | 3 |
zillow.com |
○ | ○ | ○ | ○ | ○ | ○ | ○ | 3 |
reuters.com |
○ | ~ | ○ | ○ | ○ | ○ | ○ | 1 |
indeed.com |
~ | ~ | ~ | ~ | ~ | ~ | ~ | 1 |
notion.so |
○ | ○ | ○ | ○ | ○ | ○ | ○ | 1 |
nerdwallet.com |
○ | ○ | ○ | ✗ | ○ | ○ | ○ | 1 |
wsj.com |
~ | ~ | ○ | ○ | ○ | ○ | ○ | 0 |
imdb.com |
○ | ○ | ○ | ○ | ○ | ○ | ○ | 0 |
wordpress.org |
✓ | ○ | ✓ | ✓ | ✓ | ✓ | ✓ | 0 |
cloudflare.com |
✓ | ○ | ○ | ✓ | ✓ | ✓ | ○ | 0 |
Three patterns are visible in the rows rather than the totals.
News sites cluster at the top. Most of the heaviest blockers are publishers whose product is text: nytimes.com, cnn.com, bbc.co.uk, bloomberg.com. That is the group with the clearest commercial exposure to a model answering questions from their reporting, and the legal departments to act on it.
Developer and reference sites cluster at the bottom. Several name only one or two agents, or none, and are governed by a wildcard group alone. Whether that reflects a decision or the absence of one is not something a robots file can tell you.
The search agent is often spared inside an otherwise total block. Of the 16 sites that block GPTBot, nine do not block OAI-SearchBot: one allows it, two restrict it to some paths and six do not name it. On sciencedirect.com the search agent is explicitly allowed while GPTBot is blocked, which is the training-versus-answering split stated as plainly as a robots file can state it.
What this does not tell you
robots.txt records a stated wish. It is not enforcement, and RFC 9309 is explicit that compliance is voluntary. A file that names an agent tells you what the operator asked for, and nothing about what that agent did.
Four of the 50 domains, stackoverflow.com, quora.com, pinterest.com and mayoclinic.org, answered our request for the file with an HTTP error. That is its own finding: on those sites the stated policy could not be read by our client, so “check robots.txt before you crawl” was advice we could not follow there.
The counts are per agent, as Finding 2 sets out. A per-operator total would be a different and arguably fairer metric, and we have not computed it because the mapping from agent to operator is our judgement rather than a published fact.
If you run a site, what this implies
Three things follow from the data, whichever side of the crawl you are on.
Naming agents individually is now the norm, and it ages badly. Thirty-one of these files name specific agents, and operators add new ones: OpenAI’s crawler page lists OAI-AdsBot, an agent this study did not score. A file listing anthropic-ai but not ClaudeBot, or GPTBot but not OAI-SearchBot, expresses an intention that the other agent never reads. Eighteen sites in the sample name anthropic-ai and fifteen block it, although Anthropic’s crawler page lists only ClaudeBot, Claude-User and Claude-SearchBot. That is a fair sign of how many of these lists are maintained by hand and revisited rarely.
The training-versus-answering distinction is the decision worth making deliberately. Blocking a bulk crawler keeps your text out of a training set. Blocking the agent that fetches a page to answer a live question removes you from the answer instead, along with any citation it would have carried. Those are opposite outcomes, and a single Disallow: / across every AI agent chooses both without distinguishing them. We did not record which paths the partial rules cover, so this study cannot show how many of them draw that line.
A file nobody can fetch states nothing. Four of the fifty domains returned an error instead of the file. If your edge rules block unfamiliar clients before they reach /robots.txt, a compliant crawler cannot discover your policy and a non-compliant one is unaffected. The check falls only on the crawlers that were going to obey it.
Our own file takes the opposite position and says so in its first line: full-text crawling is welcome, including for AI training and answer engines. That is a choice a reference site can make and a newspaper often cannot, which is most of what the table above shows.
Related reading
The mechanics of the file itself, including the longest-match rule this study does not evaluate, are in the robots.txt glossary entry. What the file can and cannot enforce sits alongside the wider legal picture in web scraping. If you are on the crawling side, collecting training data for a language model covers provenance and rate limits for bulk collection, and proxies for AI agents that browse covers the user-triggered kind of fetch that Finding 1 shows sites treat more leniently.
For the layers below stated policy, what a server can tell about a client before it reads a single header, see our companion measurements on what HTTP clients send by default and on TLS handshake fingerprints. robots.txt records what a site asks; those two record what it can detect regardless.
Reproducing this
The whole study is a fetch and a parse. The parsing rule that matters is the grouping one: consecutive User-agent: lines share the rules that follow them, and a parser that misses this will under-count every site that maintains a long list. These are the two functions the study used, with a small file to run them on.
def groups(txt):
"""user-agent (lower) -> [(directive, path)]"""
out, current, prev_ua = {}, [], False
for raw in txt.splitlines():
line = raw.split('#', 1)[0].strip()
if not line or ':' not in line:
continue
k, v = [x.strip() for x in line.split(':', 1)]
k = k.lower()
if k == 'user-agent':
if not prev_ua: # a new block starts here
current = []
current.append(v.lower())
for ua in current:
out.setdefault(ua, [])
prev_ua = True
elif k in ('allow', 'disallow'):
prev_ua = False
for ua in current:
out.setdefault(ua, []).append((k, v))
return out
def verdict(g, agent):
rules = g.get(agent.lower())
if rules is None:
return 'unmentioned'
disallows = [v for k, v in rules if k == 'disallow']
allows = [v for k, v in rules if k == 'allow']
if any(v == '/' for v in disallows):
return 'allowed' if any(v == '/' for v in allows) else 'blocked'
if not disallows or all(v == '' for v in disallows):
return 'allowed'
return 'partial'
sample = """User-agent: ClaudeBot
User-agent: Claude-User
User-agent: CCBot
Disallow: /
User-agent: *
Disallow: /search
"""
g = groups(sample)
for a in ('ClaudeBot', 'Claude-User', 'CCBot', 'GPTBot', '*'):
print(f'{a:<12} {verdict(g, a)}')
Run with Python 3.9.6, it prints:
ClaudeBot blocked
Claude-User blocked
CCBot blocked
GPTBot unmentioned
* partial
Swap in a parser that starts a new group on every User-agent: line and the same file scores ClaudeBot and Claude-User as allowed, because only CCBot’s line is followed directly by a rule. Verify any parser against a raw file before trusting its totals. Ours was checked against two sites with different structures, one naming a single agent and one grouping at least twenty, and both agreed.
Download the data
The per-site results are published as files under CC BY 4.0. Reuse them with a link to this page.
- ai-crawler-robots-policies.csv: one row per domain, 50 rows, with the fetch result, whether the file names any AI agent, a verdict column for each of the 18 agents and the count of agents blocked.
- ai-crawler-robots-policies.json: the same rows, plus the per-agent totals, the wildcard totals, the method and the list of known gaps.
An empty cell means the verdict was not kept, which is different from unmentioned. Per-site verdicts survive for the seven agents in the table above. For the other eleven agents, the 31 sites that name an AI agent have empty cells, and only the per-agent totals and each site’s count of blocks remain. The four domains that returned an error are empty throughout.
Limits
- One fetch per domain, on one day. These files change often, and a second reading would not necessarily match.
- A hand-picked sample, not a ranking. The 50 were chosen to span news, reference, developer, commerce and health. They are not the top 50 by traffic, and the percentages should not be read as representative of the web.
- Site-wide posture only. We record whether an agent is disallowed from everything, not which paths a partial rule covers, and the parser does not apply longest-match precedence between paths.
- An incomplete record. Per-site verdicts were kept for 7 of the 18 agents, the wildcard verdict per site for only the four sites we spot-checked, and only the exception class, not the status code, for the four failed fetches.
- Stated policy, not behaviour. Nothing here measures whether any crawler complied.
Download the data
The raw results behind this study, as files. Reuse them under CC BY 4.0 with a link to this page.