robots.txt Explained: How Crawlers Read It, Where Python Gets It Wrong, and Its Limits

robots.txt is a plain text file that a website puts at its root to tell automated visitors where they may go. A crawler (a program that fetches pages automatically, like a search engine’s bot) is expected to read it first. The file is tiny and the idea is old, from 1994, but the details trip people up all the time. This guide covers what the file says, how a crawler is supposed to read it, and where it stops being useful.

The rules come from RFC 9309, the standard published in September 2022.¹ I will quote it where the wording matters. I also tested a popular implementation against it, and the result is below.

The one idea to keep

The standard says so in its introduction: “These rules are not a form of access authorization.”¹ The file is a request. A well-behaved crawler reads it and follows it. A badly behaved one never opens it, and nothing in the protocol stops that. Its security section goes further and warns that listing a path in the file “exposes them publicly and thus makes the paths discoverable.”¹ If a page must stay private, put a login in front of it.

The gap between what a file says and what servers actually do is measured in this post: on 39 sites, servers refused 29% of the AI requests the file allowed.

Anatomy of the file

A file is a list of groups. Each group starts with one or more User-agent lines, naming who the rules are for, and is followed by Allow and Disallow rules.

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: *
Disallow: /admin/
Allow: /admin/help
Disallow: /*.pdf$

Sitemap: https://example.com/sitemap.xml

• The first group applies to two named crawlers and shuts both out of the whole site (/ is the root path, so everything under it).
• The second group, *, applies to everyone else. It blocks /admin/, carves out one page inside it, and blocks any URL ending in .pdf.
• Sitemap is not part of the standard’s rules but is widely read, and the standard says it must not break a group.¹

Three symbols are supported everywhere: # starts a comment, * matches any run of characters, and $ means “the URL ends here.”¹

How a crawler decides

There are two decisions, in this order.

1. Which group applies? The crawler matches its own name (its product token, such as GPTBot) against the User-agent lines, ignoring case. If several groups name it, their rules are merged into one. If none does, it uses the * group. If there is no * group either, nothing is restricted.¹

2. Which rule wins? The crawler compares the URL path to every Allow and Disallow pattern in that group. Paths are case sensitive and a pattern matches from the start of the path. The most specific match wins, meaning the longest pattern. If an Allow and a Disallow are the same length, Allow wins. If nothing matches, the URL is allowed.¹

Two consequences that surprise people:

• Order does not matter. Put Allow: /admin/help before or after Disallow: /admin/ and the result is the same, because the longer pattern wins.
• Patterns are prefixes. Disallow: /admin also blocks /administrator and /admin-tools. Add the trailing slash, or end the pattern with $, if you mean one path.

When the file does not load

The crawler has to fetch https://yourhost/robots.txt first, and what it gets back changes everything.

flowchart TB
A["Crawler fetches /robots.txt"] --> B{"Result"}
B -->|"200"| C["Parse and follow the rules"]
B -->|"301 or 302"| D["Follow up to 5 redirects, then treat as unavailable"]
B -->|"4xx"| E["Unavailable: crawler MAY access everything"]
B -->|"5xx or network error"| F["Unreachable: crawler MUST assume disallow all"]

A few more rules from the same standard: crawlers should not reuse a cached copy for more than 24 hours, and must parse at least the first 500 KiB of the file.¹ The file belongs to one host, so www.example.com and example.com each need their own, as do http and https.

The 4xx rule has a sharp edge. A server that answers /robots.txt with a 403 or 404 by accident has, as far as the standard goes, published “everything allowed.” Of the 66 sites I fetched for a recent measurement, two (Cars.com and Stack Overflow) answered my request for the file with a 403 and a 418. That was a response to my script, not necessarily what a real crawler sees, but it shows how a protection layer can sit in front of the file itself.

Where Python’s parser disagrees with the standard

Python ships a robots.txt parser in its standard library, urllib.robotparser, and it is the first thing many people reach for. I ran ten cases through it and through a matcher I wrote from the RFC text (about 50 lines, linked below). Python 3.12.10 disagreed on five.²

The pattern is easy to read once you see it. Python takes the first rule that matches, in file order, where the standard takes the longest. It also treats * and $ as ordinary characters instead of wildcards, and it consults only the first group that names the bot. The other four cases I ran (a $ anchor on a longer path, a specific group beating *, an empty Disallow, and the /admin prefix) came out the same in both.

This matters because real files use these features. Of the 61 robots.txt files I downloaded, 50 contain a * wildcard in a rule, 29 use a $ anchor and 49 have Allow lines.³ If you use Python’s parser to decide whether you may fetch a page, you can be wrong in either direction. I used it myself in the post linked above, so I re-ran every allow/deny decision with the RFC matcher. All 106 allow/deny decisions behind that post’s counts came out identical, because the rules that mattered there were plain prefixes. That was luck of the data and not something to count on.

Google, whose engineers co-authored the RFC, publishes its own parser as open source, which is a reasonable choice if you need one that follows the standard.⁴ For a quick check, the script below runs the comparison yourself: robots-rfc9309-match.py.

Writing rules for AI crawlers

The companies behind AI products now run several bots each, and they do different jobs. Blocking one does not block the others, so you have to name them.

To keep out the two training crawlers while staying visible in search and to people using chat tools, name only those two:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Two honest limits. First, OpenAI says its user-triggered fetcher may ignore the file, so Disallow against ChatGPT-User is a request that may not be honored. Anthropic’s page says its bots honor the file, though it does not describe the user-triggered bots separately.⁶ Second, naming a bot in your file proves nothing about what it does next. In the files I downloaded, ClaudeBot appears in 21, GPTBot in 18 and ChatGPT-User in 19, and a name can sit in an Allow group as easily as a Disallow one.³

What robots.txt will not do

• It does not hide pages from search. Google says the file “is not a mechanism for keeping a web page out of Google,” and recommends noindex or a password instead. A blocked URL can still show up in results, just without a description.⁷
• It does not enforce itself. If you need a bot gone, refuse it where requests arrive: at the web server, a firewall rule, or a CDN’s bot settings. OpenAI and Anthropic both publish the IP ranges their crawlers use, so you can check that a request claiming to be GPTBot really comes from OpenAI.⁵⁶ Anthropic adds that IP blocking alone may not give a reliable opt-out, so it recommends the file as the main signal.⁶
• It does not keep a secret. Anything listed can be read by anyone who opens the file.

A practical habit that follows from this: after you change the file, fetch it the way a crawler would, with curl -s https://yourhost/robots.txt, check that the status is 200, and then test the specific URLs you care about with a parser that follows the standard.

References