Skip to content
Glossary

What is robots.txt?

Definition

A plain-text file at the root of a site that tells crawlers, by name, which paths to stay out of.

Covered in a guideRead robots.txt the way the standard doesA section of How to collect LLM training data from the web

The file is public, advisory and read by the crawler, not enforced by the server. A disallowed path is still served to anyone who requests it. What the file does is state the operator’s wishes in a format RFC 9309 has fixed since 2022, so a crawler can follow them and nobody can later say they were not told.

How a crawler reads it

User-agent: ExampleBot
User-agent: OtherBot
Disallow: /search
Allow: /search/about

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Rules come in groups. Consecutive User-agent lines share the rules beneath them, so ExampleBot and OtherBot obey the same pair. A crawler uses the group that names it and ignores the * group entirely: ExampleBot above may fetch /admin/. The * group applies only to a crawler that no group names. If several groups name the same crawler, their rules are merged.

Within the group, the rule with the longest matching path wins, and when an Allow and a Disallow match with the same length, Allow should win. A path that matches no rule is allowed. Sitemap lines sit outside the groups; RFC 9309 leaves them to the separate Sitemaps protocol.

The RFC also sets the edges. A crawler should cache the file for no more than 24 hours. If it caps how much of the file it parses, the cap must be at least 500 KiB. If fetching the file returns a 4xx status, the crawler may treat everything as allowed; if it returns a 5xx or the server cannot be reached, the crawler must assume everything is disallowed.

The rule parsers get wrong

Longest match is where implementations part company. This file opens one folder inside a disallowed shop, with the Allow written after the Disallow:

import sys, urllib.robotparser
ROBOTS = """\
User-agent: *
Disallow: /shop/
Allow: /shop/public/
"""
rp = urllib.robotparser.RobotFileParser()
rp.parse(ROBOTS.splitlines())
print(sys.version.split()[0])
for path in ("/shop/cart", "/shop/public/item-1", "/blog/"):
    print(path, rp.can_fetch("MyCrawler", "https://example.com" + path))

We ran it under four Python releases:

Path RFC 9309 Python 3.9.6 Python 3.12.13 Python 3.14.7 Python 3.15.0
/shop/cart Disallowed False False False False
/shop/public/item-1 Allowed False False True True
/blog/ Allowed True True True True

The 3.9 and 3.12 parsers apply the first rule that matches, in file order, so the Disallow above the Allow wins and the public folder is reported as blocked. The 3.14 and 3.15 parsers follow the RFC. The order dependence was reported to CPython as bpo-39187, now GitHub issue 83368, which is closed. The same script therefore gives different answers depending on which interpreter a crawler happens to run under. Pin your parser, or test it against a file like this one, before you trust its output.

What it is not

Not access control: the path is served regardless. Not privacy: listing /admin/ advertises it to anyone who reads the file. Not a licence: permission to fetch a page says nothing about the right to reuse what is on it. And it only reaches crawlers that publish a name and honour it, which matters for AI crawlers in particular. Which AI crawlers 46 major sites actually name, and how they treat training crawlers differently from answering agents, is measured in Which AI crawlers do major sites actually block?

For a crawler you run, read the file before the first request to a host, honour it per group, and log which version of it you obeyed. Collecting training data for a language model covers that in practice. An agent fetching a page because a user asked is a different case from a crawler, and Proxies for AI agents that browse explains how the distinction is drawn. The legal picture, where the file is one piece of evidence among several, is in web scraping.

How this was tested. 21 October 2026, macOS. The script above, unchanged, under Python 3.9.6 (the system Python), 3.12.13 and 3.15.0 from Homebrew, and 3.14.7 from uv. Python 3.13 was not tested.

Frequently asked questions

Is robots.txt legally binding?
Not by itself. It is a voluntary protocol, standardised in RFC 9309, that records what the operator asks of crawlers. It can be cited as evidence that you were told, which matters in a dispute, but it is not access control.
Which rule wins when Allow and Disallow both match?
Under RFC 9309, the rule with the longest matching path, and Allow on a tie. Some parsers use the first matching rule instead: Python's urllib.robotparser did in the 3.9.6 and 3.12.13 releases we tested.
Does a crawler named in its own group also follow the * group?
No. A crawler named in a group follows that group only and ignores the * group. Repeat every rule you want a named crawler to obey inside its own group.

Sources

  1. RFC 9309: Robots Exclusion Protocol
  2. Python documentation: urllib.robotparser
  3. CPython issue 83368 (bpo-39187): urllib.robotparser does not respect the longest match for the rule

All terms, A to Z