On this page 8 sections
Collecting LLM training data from the web comes down to five jobs. Crawl under a named agent at a pace each host can bear. Read robots.txt the way RFC 9309 defines it, which your parser may not. Keep the raw response with its provenance. Remove duplicates in layers, cheapest first. And check that every 200 is the document you asked for, because a crawl that stores challenge pages reports a healthy success rate the whole time.
Fetching is the easy half. The rest decides whether the corpus can be filtered, audited or partly deleted later, and all three get asked for. Every snippet below was run, on Python 3.15.0 unless noted, and the robots.txt section compares how three Python releases read the same files.
Read robots.txt the way the standard does
robots.txt is advisory. For a training corpus the question is less whether you can be stopped than whether you can show what you were told. RFC 9309, published in 2022, fixed the rules that matter:
- Consecutive
User-agent:lines share the rules that follow them. A parser that starts a new group on each line misreads most large files. In our survey of 46 sites, one file groups about twenty agents under a singleDisallow: /. - Several groups for the same agent are combined. The rules of every group naming you apply, not just the first.
- The most specific rule wins. Specificity is the length of the matching path in octets, and an
Allowbeats an equivalentDisallow. - Status codes have meanings. A 4xx means the file is unavailable and the crawler may proceed. A 5xx or a network error means it is unreachable, and the crawler must assume complete disallow.
# Your agent matches this group only. The * group below does not apply.
User-agent: ExampleCorpBot
Disallow: /private/
User-agent: *
Disallow: /
Your parser may not agree
Python’s standard-library parser is what many crawlers use, and until recently it did not implement RFC 9309. We fed the same cases to urllib.robotparser on three releases. “Blocked” and “allowed” are what can_fetch() returned.
| Case | RFC 9309 | Python 3.9.6 | Python 3.12.13 | Python 3.15.0 |
|---|---|---|---|---|
Disallow: /docs/ then Allow: /docs/public/, fetch /docs/public/page |
Allowed | Blocked | Blocked | Allowed |
Disallow: /*.pdf$, fetch /files/report.pdf |
Blocked | Allowed | Allowed | Blocked |
Same agent in two groups, /a/ then /b/ disallowed, fetch /b/x |
Blocked | Allowed | Allowed | Blocked |
Two consecutive User-agent: lines above Disallow: / |
Blocked | Blocked | Blocked | Blocked |
User-agent: bot with Disallow: /, agent ExampleCorpBot |
Allowed | Blocked | Blocked | Blocked |
A bot group plus an ExampleCorpBot group with Allow: /, agent passed as ExampleCorpBot |
Allowed | Blocked | Blocked | Allowed |
Same file, agent passed as the full header ExampleCorpBot/1.0 (+https://example.com/bot) |
Allowed | Blocked | Blocked | Blocked |
| robots.txt answers 401 or 403 | May crawl | Blocked | Blocked | Blocked |
| robots.txt answers 404 or 429 | May crawl | Allowed | Allowed | Allowed |
| robots.txt answers 500 or 503 | Assume disallow | Blocked | Blocked | Blocked |
| Nothing listening on the port | Assume disallow | Raises URLError |
Raises URLError |
Raises URLError |
| Server accepts the connection and never answers | Assume disallow | read() still blocked after 20 s |
Still blocked after 20 s | Still blocked after 20 s |
The change came from CPython issue gh-138907, closed in May 2026, with backport pull requests for the 3.13 and 3.14 branches. We did not test those branches, so check which patch release you run before trusting the newer column. Four rows stay wrong on every version we tried, and they are the ones a crawler should code around.
Agent matching is a substring test. User-agent: bot captured ExampleCorpBot on all three releases, where the standard asks for a case-insensitive match on the whole product token. Python 3.15.0 looks for an exact group first, but only for the string you pass, so passing the full User-Agent header instead of the bare token sends it back to the substring search. Pass the token.
401 and 403 mean “blocked”. RFC 9309 puts the whole 400 to 499 range under “unavailable”, where the crawler may proceed. Google’s documentation says its crawlers treat every 4xx except 429 as if no robots.txt existed. Python disallows everything on a 401 or 403, which is the cautious reading and a defensible one, but it is not the standard’s.
read() has no timeout. It calls urlopen() without one, so a server that accepts the connection and never replies holds the call indefinitely. Our test thread was still waiting when we stopped watching at 20 seconds. One stalled host can freeze a single-threaded crawler.
Network errors escape. A refused connection raises URLError out of read() rather than being treated as unreachable. Without a try, the crawler crashes; with a careless one, it proceeds as if allowed.
The way around all four is to fetch the file yourself, with a timeout, and give the parser only the text:
import time
import urllib.error
import urllib.request
import urllib.robotparser
AGENT = "ExampleCorpBot" # the product token, not the full User-Agent
UA = "ExampleCorpBot/1.0 (+https://example.com/bot)"
def load_robots(origin: str, timeout: float = 10.0) -> urllib.robotparser.RobotFileParser:
"""Fetch robots.txt with a timeout and apply RFC 9309's status rules.
RobotFileParser.read() has no timeout and treats 401/403 as disallow-all,
so fetch it ourselves and hand the parser the result.
"""
rp = urllib.robotparser.RobotFileParser(origin + "/robots.txt")
req = urllib.request.Request(rp.url, headers={"User-Agent": UA})
try:
with urllib.request.urlopen(req, timeout=timeout) as r:
body = r.read(500 * 1024) # RFC 9309 2.5: parse at least 500 KiB
rp.parse(body.decode("utf-8", "replace").splitlines())
rp.verdict = "parsed"
except urllib.error.HTTPError as e:
if 400 <= e.code < 500 and e.code != 429: # 2.3.1.3 unavailable: may crawl
rp.allow_all, rp.verdict = True, f"HTTP {e.code}: unavailable, allow"
else: # 5xx, and 429 (our choice: a rate limit is not an answer)
rp.disallow_all, rp.verdict = True, f"HTTP {e.code}: unreachable, disallow"
except (urllib.error.URLError, TimeoutError) as e:
rp.disallow_all, rp.verdict = True, f"{type(e).__name__}: unreachable, disallow"
rp.modified() # stamp the fetch time for caching
return rp
Run on Python 3.15.0 against local servers answering each status, a closed port, a server that never replies, and one real site. The local addresses are replaced with a description:
local server answering 401 HTTP 401: unavailable, allow can_fetch=True 0.0s
local server answering 403 HTTP 403: unavailable, allow can_fetch=True 0.0s
local server answering 404 HTTP 404: unavailable, allow can_fetch=True 0.0s
local server answering 429 HTTP 429: unreachable, disallow can_fetch=False 0.0s
local server answering 503 HTTP 503: unreachable, disallow can_fetch=False 0.0s
nothing listening URLError: unreachable, disallow can_fetch=False 0.0s
server that never answers TimeoutError: unreachable, disallow can_fetch=False 5.0s
https://www.rfc-editor.org parsed can_fetch=True 0.2s
Treating 429 as unreachable is our choice, not the RFC’s: a rate limit on the policy file is not an answer about the site. Cache the result per host and refetch at least daily, since RFC 9309 says a cached copy should not be used for more than 24 hours unless the file is unreachable. Store the verdict beside every document fetched under it.
Record provenance at fetch time
Every downstream question (may we use this, where did it come from, is it stale, was it duplicated) needs metadata that is cheap at fetch time and impossible to reconstruct later.
| Field | Why it is needed later |
|---|---|
| URL, final after redirects | The URL you requested is often not the one you got |
| Fetch timestamp, UTC | Freshness, and evidence of what the policy said at the time |
| HTTP status and content type | A 200 serving HTML where you expected JSON is a silent corruption |
| robots.txt verdict at fetch time | Policies change; what you were told then is not recoverable now |
| Content hash | Deduplication, and detecting silent edits on re-crawl |
| Response byte length | Catches truncation that parsing will not |
| Canonical URL from the page | The site’s own statement about which copy is authoritative |
| Address the response came from | Which exit or region produced a page that varies by location |
You do not need to invent a format for this. WARC, the web archive format (version 1.0 is ISO 28500:2009; the IIPC publishes 1.1), stores the raw request and response as separate records and gives each one header fields for most of the table: WARC-Target-URI, WARC-Date, WARC-IP-Address, WARC-Payload-Digest. It also defines a revisit record type, written when a re-crawl finds content identical to an earlier capture, so unchanged pages cost a header instead of a second copy.
Store the raw response, not only the extracted text. Extraction rules improve, and re-extracting from storage costs nothing, while re-fetching a million pages costs a great deal and may no longer be permitted. Partition by source and by fetch date, so “remove this domain” and “rebuild everything older than March” are cheap operations rather than a full scan.
Deduplicate in layers
The web repeats itself constantly, and a corpus that skips deduplication is largely boilerplate. Three layers, cheapest first:
- URL canonicalisation. Lowercase the host, drop default ports, tracking parameters and fragments, respect
rel="canonical". Much duplication is one document behind many addresses. - Exact hash of the normalised body. It catches mirrors, syndication and the same page under two URLs.
- Near-duplicate detection. Templated pages differ by a product name or a date. MinHash and SimHash exist because exact hashing misses them, and this is the expensive layer.
import hashlib, re, unicodedata
from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode
TRACKING = {"utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content", "gclid", "fbclid", "ref"}
def canonical_url(url: str) -> str:
s = urlsplit(url)
host = (s.hostname or "").lower().rstrip(".")
port = f":{s.port}" if s.port and s.port not in (80, 443) else ""
query = urlencode(sorted((k, v) for k, v in parse_qsl(s.query, keep_blank_values=True)
if k.lower() not in TRACKING))
path = re.sub(r"/{2,}", "/", s.path) or "/"
return urlunsplit((s.scheme.lower(), host + port, path, query, ""))
def body_hash(text: str) -> str:
text = unicodedata.normalize("NFKC", text)
text = re.sub(r"\s+", " ", text).strip().lower()
return hashlib.sha256(text.encode()).hexdigest()[:16]
Four URLs in, two out, on Python 3.15.0:
https://Docs.Example.com/guide/install?utm_source=newsletter&lang=en -> https://docs.example.com/guide/install?lang=en
https://docs.example.com:443/guide//install?lang=en#linux -> https://docs.example.com/guide/install?lang=en
https://docs.example.com./guide/install?lang=en&ref=sidebar -> https://docs.example.com/guide/install?lang=en
https://docs.example.com/guide/install?lang=de -> https://docs.example.com/guide/install?lang=de
2 distinct of 4
The hash shows where the cheap layers stop. "Install the CLI\n\n Run the installer." and "install the cli run the installer." both hashed to 5045e8c5ea3c30d1. The same sentence with a full stop added and an “Updated 3 days ago” footer hashed to 72bee549060abb8e, a different document as far as exact matching knows. That is the job of the near-duplicate layer. Run the first two at write time; they cost almost nothing and shrink what the third has to examine. Which query parameters are tracking noise is per site, so treat the list above as a starting point.
Pace each host, and why more addresses do not help
The instinct at scale is to add addresses. That works for per-address rate limits and for nothing else. Our measurements of what HTTP clients send by default and how their TLS handshakes differ point the same way: most of what identifies a bulk client is produced by the client, and a new address changes none of it. A per-host rate limit sees your concurrency against that host, not your total volume.
Pace by host, spaced by that host’s own response time, and obey Retry-After when it comes:
import email.utils
import time
from datetime import datetime, timezone
from urllib.parse import urlsplit
def retry_after_seconds(value):
"""Retry-After is delta-seconds or an HTTP-date (RFC 9110, 10.2.3)."""
if not value:
return None
if value.strip().isdigit():
return int(value)
try:
when = email.utils.parsedate_to_datetime(value)
except (TypeError, ValueError):
return None
return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
class HostClock:
"""One request per host at a time, spaced by that host's own response time."""
def __init__(self, min_gap=1.0, factor=4.0, max_gap=3600.0):
self.min_gap, self.factor, self.max_gap = min_gap, factor, max_gap
self.next_ok = {} # persist this, or a restart forgets every backoff
def wait(self, url):
host = urlsplit(url).hostname
delay = self.next_ok.get(host, 0.0) - time.monotonic()
if delay > 0:
time.sleep(delay)
return host
def record(self, host, elapsed, status, retry_after=None):
gap = max(self.min_gap, elapsed * self.factor)
if status in (429, 503):
gap = max(gap, retry_after_seconds(retry_after) or self.min_gap * 30)
self.next_ok[host] = time.monotonic() + min(gap, self.max_gap)
return gap
Against a local server, with a minimum gap of half a second, the third request drew a 429 carrying Retry-After: 30 and the clock pushed that host out by exactly that:
/ok waited 0.0s HTTP 200 next gap 0.5s
/ok waited 0.5s HTTP 200 next gap 0.5s
/429-long waited 0.5s HTTP 429 next gap 30.0s
The factor, the floor and the ceiling are choices, not findings. The comment on next_ok is the part that bites: a crawler that persists its queue but not its per-host backoff hammers every host it had backed off from the moment it restarts. Reading the status codes themselves, including which ones come from your own proxy rather than the site, is covered in 403 vs 407 vs 429.
Identify yourself, and mean it
At this scale the user agent is a claim you will be held to.
| Position | What it means in practice |
|---|---|
| A named agent with a contact URL | Operators can reach you and allow you specifically. The only position that lets a site say yes |
| An honest library default | Unremarkable and truthful. Fine for small, permitted work |
| No user agent at all | Stands out, because ordinary traffic almost always carries one |
| A copied browser string | Contradicted by everything else the client sends, and gives up any claim of good faith |
User-Agent: ExampleCorpBot/1.0 (+https://example.com/bot)
The browser string fails on its own terms. Our header capture recorded a Chromium navigation sending 11 headers where the median of nine libraries sent 4, in a different order and without the Sec-Fetch-* set, so a browser name over a library connection is a contradiction rather than a disguise. The TLS handshake sits underneath and nothing at the HTTP layer changes it. The same survey of 46 sites found publishers blocking GPTBot 16 times and OAI-SearchBot 7 times: they treat bulk training crawlers and on-demand fetchers differently, so say which one you are.
What breaks once the crawl is large
| Failure | Symptom | Guard |
|---|---|---|
| Challenge pages stored as text | Thousands of near-identical “verify you are human” documents | Assert on content, not on a 200 |
| Silent extraction drift | A parser change quietly empties a field for one site | Per-source populated-field counts, compared run to run |
| robots.txt fetch hangs | Crawler stalls on one host with no error | Fetch the file yourself with a timeout |
| Parser reads robots.txt differently from the RFC | Pages fetched that a compliant crawler would skip, or the reverse | Test your parser against the cases above |
| Redirect loops into one page | One document, thousands of URLs | Hash before store; record the final URL |
| Encoding damage | Mojibake in a fraction of documents | Trust the declared charset, verify by decoding |
| Unbounded frontier | Memory exhaustion days in | Bound the queue and persist it |
| Backoff lost on restart | A restart hammers hosts you had backed off from | Persist per-host state, not just the queue |
The first two are the expensive ones because they are invisible. A success rate measured on status codes stays healthy while the corpus fills with interstitials, since a 200 says a response arrived, not that it was the document. Keep a per-source contract of text the page must contain and an expected minimum length, and alert when either breaks. The RAG pipeline guide shows that check in code; it matters even more there, where one source can be the only answer to a question.
Decide what you will not collect
Some of this is legal and jurisdiction-specific, and none of it is advice. Three categories deserve an explicit decision before the crawl starts:
- Personal data. Obligations attach to it whether or not the page was public. Being able to find and delete every document from one source is a design requirement.
- Material behind a login or a paywall. Public and accessible are different things.
- Sites that asked you not to. Whether or not the file binds you, ignoring it after reading it is a different position from never having looked.
The provenance fields are what make any of this actionable. Without a per-document record of source and fetch time, “remove everything from that domain” is not a query you can run. If the same pipeline also feeds an agent that fetches pages live for users, its traffic and its rules differ; see AI agent proxy setup.
How the parser tests were run
On 21 October 2026, on macOS, the same script ran under Python 3.9.6 (the system interpreter), 3.12.13 and 3.15.0. Eight parsing cases were passed to RobotFileParser.parse() as text, with no network. Seven status cases were fetched with read() from a local HTTP server answering 200, 401, 403, 404, 429, 500 and 503. The refused case used a closed port, and the stalled case used a local listener that accepts and never writes, watched by a thread that gave up after 20 seconds. The RFC 9309 column is our reading of sections 2.2.1, 2.2.2, 2.3.1.3 and 2.3.1.4. The robots.txt loader and the deduplication snippet ran on Python 3.15.0. The pacing test ran on Python 3.12.13, with httpx 0.28.1 making the requests to a local server.
Frequently asked questions
How do you collect training data for an LLM from the web?
Does Python's urllib.robotparser follow RFC 9309?
Does robots.txt legally prevent me collecting training data?
What should happen when robots.txt returns an error?
Should my crawler use a browser user agent?
Why store raw HTML when I only need the text?
Sources
- RFC 9309: Robots Exclusion Protocol, grouping, longest match, status codes, caching and the 500 KiB limit
- CPython gh-138907: Support RFC 9309 in robotparser
- Python 3.15 documentation: urllib.robotparser
- Google Search Central: how Google interprets robots.txt status codes
- IIPC: The WARC Format 1.1, record types and header fields
- RFC 9110: HTTP Semantics, Retry-After (10.2.3)
- RFC 6585: status code 429, Too Many Requests