robots.txt Allowed 87 AI Requests. The Servers Blocked 25 of Them Anyway.

On October 17 The Verge reported that websites are closing their doors to AI agents.¹ The story names Delta, which stops agents from booking tickets, and JPMorgan Chase and Charles Schwab, which block them from accounts. It also names Yelp, which lets in only agents that pay for a data license, and says users report that Cars.com and Amtrak stopped working too. It is a good article, and it has no numbers in it. Every claim rests on a company statement or a user report.

A site has an official place to say what it allows, a file called robots.txt. So I asked a narrow question: does that file predict what the server actually does? I sent three AI user-agent strings to 66 sites and compared the two. On the 39 sites where the comparison was clean, robots.txt allowed 87 requests and the servers refused 25 of them. It denied 19, and the servers answered 12.

What is being compared

A user agent is the name a client announces in an HTTP header, such as a Chrome version string, or GPTBot. robots.txt is a plain text file at the root of a site that lists which user agents may fetch which paths. Its standard says that these rules “are not a form of access authorization” and that crawlers are “requested to honor” them.² Nothing in the protocol enforces it. A server that wants to refuse a client has to do it on its own, usually by returning 403 (forbidden) or 429 (too many requests), or by sending a tiny page or no reply at all.

I used three user agents, all documented by the companies that run them:

• GPTBot is OpenAI’s training crawler.³
• ChatGPT-User is OpenAI’s fetcher for actions a person triggers in ChatGPT. OpenAI says that because a user starts these, robots.txt rules “may not apply.”³
• ClaudeBot is Anthropic’s crawler for collecting training content, and Anthropic says its bots honor robots.txt.⁴

I did not test the agents in The Verge story (Meta’s Muse, Grok, OpenAI’s Dots). I could not find their user-agent strings, so what follows is how sites treat these three names, not how they treat those products.

How I measured

For each site I fetched robots.txt once, then requested the home page with five user agents: a Chrome string, a second copy of that Chrome string as a control, and the three AI names. I repeated that in three rounds, shuffled the order every time, and waited two seconds between requests. Eleven sensitive sites (banks, airlines, PayPal) got one round only. In total that was 880 home page requests.⁵

My first pass was wrong, and the way it was wrong is worth knowing. I called a response “blocked” when the status code was 400 or higher. That missed Schwab, which answers all three AI names with a 200 and a 503-byte body, while the Chrome string gets a page of at least 300 kB. It also called Amazon “not blocking” when Amazon sends everyone, Chrome included, a 2 kB stub. Fixing this I found a second bug in my own comparison: I read at most 300 kB per response, so a full 257 kB page from Gap looked “smaller” than a truncated 300 kB one. The final rule treats a 2xx response as a stub only if it is far smaller than the Chrome page, or under 10 kB when the Chrome page hit my read cap.

Not every site could be judged:

Result 1: allowed on paper, refused in practice

Across the 39 usable sites there were 87 pairs of a site and an AI name where robots.txt allowed the request and the server answered at all. In 25 of them the server refused. Another 19 pairs were denied by robots.txt, and 12 of those were answered with a normal page.

So 29% of requests that robots.txt allows (25 of 87) were refused, and 63% of those it denies (12 of 19) went through. Two different things are happening here, and they should not be lumped together.

The first is simple. Nine sites say yes and act no: AA, Home Depot, Lowe’s, Nike, Schwab, Sephora, Wayfair and Zillow refuse all three names, and Wikipedia refuses ClaudeBot only. Sephora is the cleanest example. Its robots.txt allows all three, its Chrome string gets a 200 with at least 300 kB, and each AI name gets a 403 with a 369-byte body, nine times out of nine.

The second is less of a surprise. Six sites say no and serve the page: BBC and LinkedIn for all three names, Instacart for two, TechCrunch for two, and The Verge and Wired for one each. This is what “not a form of access authorization” means in practice. Part of it is also by design, since OpenAI describes ChatGPT-User as a fetch a person asked for. The BBC and LinkedIn, though, answered GPTBot too, a crawler that OpenAI says follows robots.txt.

Result 2: the three names are not treated alike

ClaudeBot was refused by six more sites than ChatGPT-User, and four of those refuse it alone: TechCrunch, The Guardian, The Verge and Wikipedia serve GPTBot and ChatGPT-User the same page the Chrome string gets (122,197 bytes on Wikipedia) and answer ClaudeBot with a 403. Wikipedia’s is 82 bytes. I do not know why. The obvious guess is a site-specific rule about one crawler’s past behavior, but I have no evidence for it.

The refusals also come in more than one flavor. Forbes returned 402, whose official name is Payment Required, to GPTBot and ClaudeBot and served ChatGPT-User normally. CNN returned 451, Unavailable For Legal Reasons,⁶ to all three. Wayfair and Expedia returned 429, and Expedia did it to the Chrome string too, so that one is plain rate limiting.

Three sites never said no. Amtrak, United and Best Buy answered the Chrome string every time and did not answer the AI names at all: timeouts and dropped connections, with no status code. For a person that looks like an outage. For an agent it is indistinguishable from one.

What The Verge named

Delta and Chase are the interesting rows. Their home pages treat an AI name like a person, and The Verge says both stop agents from doing real work. That fits a block that happens further in (a login, a search form, a checkout), where this test never went.

Who is saying no

The response headers give a hint. On refused responses the Server header most often said cloudflare (72 responses), then AkamaiGHost (54), and 30 had no Server header at all, with Varnish, DataDome and CloudFront further down.⁵ Those names are content delivery networks and bot-filtering services, companies that sit in front of a site and decide which requests reach it. If that is the layer doing the refusing, then robots.txt, which the site’s own team writes, and the filter, which is often a product setting or a managed rule, are two separate systems that need not agree.

flowchart TB
R["robots.txt: advice read by the client"] -.-> A["AI request"]
A --> B{"Edge filter: CDN or bot service"}
B -->|"refuses"| C["403, 429, or no reply"]
B -->|"passes"| D{"Site application"}
D -->|"rule or paywall"| E["402, 451, or a thin page"]
D -->|"allows"| F["Normal page"]

Trade-offs and what this does not show

• I am not who I claim to be. My requests carry an AI company’s name but come from a home connection. OpenAI and Anthropic both publish the IP ranges their bots use,³⁴ so a site can check the claim. Some of these refusals may mean “you say you are GPTBot and you are not,” which is a different thing from “we block GPTBot.” I could not separate the two.
• Versions. I sent GPTBot/1.1, and OpenAI lists 1.4 today. My ClaudeBot string follows the format I remembered and is not copied from Anthropic’s page.
• Home page only. Nothing here touches login, search or checkout, which is where The Verge’s examples live.
• Small and picked by hand. 39 usable sites, chosen from the article plus popular ones. This is not a sample of the web.
• Status and size, not content. I did not store response bodies, so a “same size” page could still be a different page, and I do not know what Schwab’s 503 bytes said.
• One IP, one afternoon. Sites that rate limit or change rules would look different on another day.

Takeaways

• robots.txt is a statement of intent. Of the AI requests it allowed on 39 sites, 29% were refused, and most of the refusals came from nine sites.
• Of the requests it denied, 63% were served. If you want to keep AI fetchers out, the file alone does not do it.
• The three names get different treatment. ClaudeBot was refused most often and alone on four media or reference sites. ChatGPT-User, which OpenAI describes as user-initiated, was refused by 9 of 36.
• Refusals are not always a 403. I saw 402, 451, 429, a 503-byte page returned with a 200, and silence. An agent that only checks for a 403 will miss most of them.
• To know what a site does to your agent, test your agent. A claim in a policy file or a news story does not tell you that, and this experiment did not fully either.

References