Skip to content
AI and agents

How to collect LLM training data from the web

Collecting LLM training data from the web: provenance, layered dedup, per-host pacing, and the robots.txt rules Python's parser gets wrong.

On this page 8 sections
  1. Read robots.txt the way the standard does
  2. Your parser may not agree
  3. Record provenance at fetch time
  4. Deduplicate in layers
  5. Pace each host, and why more addresses do not help
  6. Identify yourself, and mean it
  7. What breaks once the crawl is large
  8. Decide what you will not collect
  9. How the parser tests were run

Collecting LLM training data from the web comes down to five jobs. Crawl under a named agent at a pace each host can bear. Read robots.txt the way RFC 9309 defines it, which your parser may not. Keep the raw response with its provenance. Remove duplicates in layers, cheapest first. And check that every 200 is the document you asked for, because a crawl that stores challenge pages reports a healthy success rate the whole time.

Fetching is the easy half. The rest decides whether the corpus can be filtered, audited or partly deleted later, and all three get asked for. Every snippet below was run, on Python 3.15.0 unless noted, and the robots.txt section compares how three Python releases read the same files.

Read robots.txt the way the standard does

robots.txt is advisory. For a training corpus the question is less whether you can be stopped than whether you can show what you were told. RFC 9309, published in 2022, fixed the rules that matter:

  • Consecutive User-agent: lines share the rules that follow them. A parser that starts a new group on each line misreads most large files. In our survey of 46 sites, one file groups about twenty agents under a single Disallow: /.
  • Several groups for the same agent are combined. The rules of every group naming you apply, not just the first.
  • The most specific rule wins. Specificity is the length of the matching path in octets, and an Allow beats an equivalent Disallow.
  • Status codes have meanings. A 4xx means the file is unavailable and the crawler may proceed. A 5xx or a network error means it is unreachable, and the crawler must assume complete disallow.
# Your agent matches this group only. The * group below does not apply.
User-agent: ExampleCorpBot
Disallow: /private/

User-agent: *
Disallow: /

Your parser may not agree

Python’s standard-library parser is what many crawlers use, and until recently it did not implement RFC 9309. We fed the same cases to urllib.robotparser on three releases. “Blocked” and “allowed” are what can_fetch() returned.

Case RFC 9309 Python 3.9.6 Python 3.12.13 Python 3.15.0
Disallow: /docs/ then Allow: /docs/public/, fetch /docs/public/page Allowed Blocked Blocked Allowed
Disallow: /*.pdf$, fetch /files/report.pdf Blocked Allowed Allowed Blocked
Same agent in two groups, /a/ then /b/ disallowed, fetch /b/x Blocked Allowed Allowed Blocked
Two consecutive User-agent: lines above Disallow: / Blocked Blocked Blocked Blocked
User-agent: bot with Disallow: /, agent ExampleCorpBot Allowed Blocked Blocked Blocked
A bot group plus an ExampleCorpBot group with Allow: /, agent passed as ExampleCorpBot Allowed Blocked Blocked Allowed
Same file, agent passed as the full header ExampleCorpBot/1.0 (+https://example.com/bot) Allowed Blocked Blocked Blocked
robots.txt answers 401 or 403 May crawl Blocked Blocked Blocked
robots.txt answers 404 or 429 May crawl Allowed Allowed Allowed
robots.txt answers 500 or 503 Assume disallow Blocked Blocked Blocked
Nothing listening on the port Assume disallow Raises URLError Raises URLError Raises URLError
Server accepts the connection and never answers Assume disallow read() still blocked after 20 s Still blocked after 20 s Still blocked after 20 s

The change came from CPython issue gh-138907, closed in May 2026, with backport pull requests for the 3.13 and 3.14 branches. We did not test those branches, so check which patch release you run before trusting the newer column. Four rows stay wrong on every version we tried, and they are the ones a crawler should code around.

Agent matching is a substring test. User-agent: bot captured ExampleCorpBot on all three releases, where the standard asks for a case-insensitive match on the whole product token. Python 3.15.0 looks for an exact group first, but only for the string you pass, so passing the full User-Agent header instead of the bare token sends it back to the substring search. Pass the token.

401 and 403 mean “blocked”. RFC 9309 puts the whole 400 to 499 range under “unavailable”, where the crawler may proceed. Google’s documentation says its crawlers treat every 4xx except 429 as if no robots.txt existed. Python disallows everything on a 401 or 403, which is the cautious reading and a defensible one, but it is not the standard’s.

read() has no timeout. It calls urlopen() without one, so a server that accepts the connection and never replies holds the call indefinitely. Our test thread was still waiting when we stopped watching at 20 seconds. One stalled host can freeze a single-threaded crawler.

Network errors escape. A refused connection raises URLError out of read() rather than being treated as unreachable. Without a try, the crawler crashes; with a careless one, it proceeds as if allowed.

The way around all four is to fetch the file yourself, with a timeout, and give the parser only the text:

import time
import urllib.error
import urllib.request
import urllib.robotparser

AGENT = "ExampleCorpBot"          # the product token, not the full User-Agent
UA = "ExampleCorpBot/1.0 (+https://example.com/bot)"


def load_robots(origin: str, timeout: float = 10.0) -> urllib.robotparser.RobotFileParser:
    """Fetch robots.txt with a timeout and apply RFC 9309's status rules.

    RobotFileParser.read() has no timeout and treats 401/403 as disallow-all,
    so fetch it ourselves and hand the parser the result.
    """
    rp = urllib.robotparser.RobotFileParser(origin + "/robots.txt")
    req = urllib.request.Request(rp.url, headers={"User-Agent": UA})
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            body = r.read(500 * 1024)             # RFC 9309 2.5: parse at least 500 KiB
        rp.parse(body.decode("utf-8", "replace").splitlines())
        rp.verdict = "parsed"
    except urllib.error.HTTPError as e:
        if 400 <= e.code < 500 and e.code != 429: # 2.3.1.3 unavailable: may crawl
            rp.allow_all, rp.verdict = True, f"HTTP {e.code}: unavailable, allow"
        else:                                     # 5xx, and 429 (our choice: a rate limit is not an answer)
            rp.disallow_all, rp.verdict = True, f"HTTP {e.code}: unreachable, disallow"
    except (urllib.error.URLError, TimeoutError) as e:
        rp.disallow_all, rp.verdict = True, f"{type(e).__name__}: unreachable, disallow"
    rp.modified()                                 # stamp the fetch time for caching
    return rp

Run on Python 3.15.0 against local servers answering each status, a closed port, a server that never replies, and one real site. The local addresses are replaced with a description:

local server answering 401             HTTP 401: unavailable, allow         can_fetch=True  0.0s
local server answering 403             HTTP 403: unavailable, allow         can_fetch=True  0.0s
local server answering 404             HTTP 404: unavailable, allow         can_fetch=True  0.0s
local server answering 429             HTTP 429: unreachable, disallow      can_fetch=False  0.0s
local server answering 503             HTTP 503: unreachable, disallow      can_fetch=False  0.0s
nothing listening                      URLError: unreachable, disallow      can_fetch=False  0.0s
server that never answers              TimeoutError: unreachable, disallow  can_fetch=False  5.0s
https://www.rfc-editor.org             parsed                               can_fetch=True  0.2s

Treating 429 as unreachable is our choice, not the RFC’s: a rate limit on the policy file is not an answer about the site. Cache the result per host and refetch at least daily, since RFC 9309 says a cached copy should not be used for more than 24 hours unless the file is unreachable. Store the verdict beside every document fetched under it.

Record provenance at fetch time

Every downstream question (may we use this, where did it come from, is it stale, was it duplicated) needs metadata that is cheap at fetch time and impossible to reconstruct later.

Field Why it is needed later
URL, final after redirects The URL you requested is often not the one you got
Fetch timestamp, UTC Freshness, and evidence of what the policy said at the time
HTTP status and content type A 200 serving HTML where you expected JSON is a silent corruption
robots.txt verdict at fetch time Policies change; what you were told then is not recoverable now
Content hash Deduplication, and detecting silent edits on re-crawl
Response byte length Catches truncation that parsing will not
Canonical URL from the page The site’s own statement about which copy is authoritative
Address the response came from Which exit or region produced a page that varies by location

You do not need to invent a format for this. WARC, the web archive format (version 1.0 is ISO 28500:2009; the IIPC publishes 1.1), stores the raw request and response as separate records and gives each one header fields for most of the table: WARC-Target-URI, WARC-Date, WARC-IP-Address, WARC-Payload-Digest. It also defines a revisit record type, written when a re-crawl finds content identical to an earlier capture, so unchanged pages cost a header instead of a second copy.

Store the raw response, not only the extracted text. Extraction rules improve, and re-extracting from storage costs nothing, while re-fetching a million pages costs a great deal and may no longer be permitted. Partition by source and by fetch date, so “remove this domain” and “rebuild everything older than March” are cheap operations rather than a full scan.

Deduplicate in layers

The web repeats itself constantly, and a corpus that skips deduplication is largely boilerplate. Three layers, cheapest first:

  • URL canonicalisation. Lowercase the host, drop default ports, tracking parameters and fragments, respect rel="canonical". Much duplication is one document behind many addresses.
  • Exact hash of the normalised body. It catches mirrors, syndication and the same page under two URLs.
  • Near-duplicate detection. Templated pages differ by a product name or a date. MinHash and SimHash exist because exact hashing misses them, and this is the expensive layer.
import hashlib, re, unicodedata
from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

TRACKING = {"utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content", "gclid", "fbclid", "ref"}

def canonical_url(url: str) -> str:
    s = urlsplit(url)
    host = (s.hostname or "").lower().rstrip(".")
    port = f":{s.port}" if s.port and s.port not in (80, 443) else ""
    query = urlencode(sorted((k, v) for k, v in parse_qsl(s.query, keep_blank_values=True)
                             if k.lower() not in TRACKING))
    path = re.sub(r"/{2,}", "/", s.path) or "/"
    return urlunsplit((s.scheme.lower(), host + port, path, query, ""))

def body_hash(text: str) -> str:
    text = unicodedata.normalize("NFKC", text)
    text = re.sub(r"\s+", " ", text).strip().lower()
    return hashlib.sha256(text.encode()).hexdigest()[:16]

Four URLs in, two out, on Python 3.15.0:

https://Docs.Example.com/guide/install?utm_source=newsletter&lang=en -> https://docs.example.com/guide/install?lang=en
https://docs.example.com:443/guide//install?lang=en#linux            -> https://docs.example.com/guide/install?lang=en
https://docs.example.com./guide/install?lang=en&ref=sidebar          -> https://docs.example.com/guide/install?lang=en
https://docs.example.com/guide/install?lang=de                       -> https://docs.example.com/guide/install?lang=de
2 distinct of 4

The hash shows where the cheap layers stop. "Install the CLI\n\n Run the installer." and "install the cli run the installer." both hashed to 5045e8c5ea3c30d1. The same sentence with a full stop added and an “Updated 3 days ago” footer hashed to 72bee549060abb8e, a different document as far as exact matching knows. That is the job of the near-duplicate layer. Run the first two at write time; they cost almost nothing and shrink what the third has to examine. Which query parameters are tracking noise is per site, so treat the list above as a starting point.

Pace each host, and why more addresses do not help

The instinct at scale is to add addresses. That works for per-address rate limits and for nothing else. Our measurements of what HTTP clients send by default and how their TLS handshakes differ point the same way: most of what identifies a bulk client is produced by the client, and a new address changes none of it. A per-host rate limit sees your concurrency against that host, not your total volume.

Pace by host, spaced by that host’s own response time, and obey Retry-After when it comes:

import email.utils
import time
from datetime import datetime, timezone
from urllib.parse import urlsplit


def retry_after_seconds(value):
    """Retry-After is delta-seconds or an HTTP-date (RFC 9110, 10.2.3)."""
    if not value:
        return None
    if value.strip().isdigit():
        return int(value)
    try:
        when = email.utils.parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None
    return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())


class HostClock:
    """One request per host at a time, spaced by that host's own response time."""

    def __init__(self, min_gap=1.0, factor=4.0, max_gap=3600.0):
        self.min_gap, self.factor, self.max_gap = min_gap, factor, max_gap
        self.next_ok = {}                      # persist this, or a restart forgets every backoff

    def wait(self, url):
        host = urlsplit(url).hostname
        delay = self.next_ok.get(host, 0.0) - time.monotonic()
        if delay > 0:
            time.sleep(delay)
        return host

    def record(self, host, elapsed, status, retry_after=None):
        gap = max(self.min_gap, elapsed * self.factor)
        if status in (429, 503):
            gap = max(gap, retry_after_seconds(retry_after) or self.min_gap * 30)
        self.next_ok[host] = time.monotonic() + min(gap, self.max_gap)
        return gap

Against a local server, with a minimum gap of half a second, the third request drew a 429 carrying Retry-After: 30 and the clock pushed that host out by exactly that:

/ok        waited  0.0s  HTTP 200  next gap   0.5s
/ok        waited  0.5s  HTTP 200  next gap   0.5s
/429-long  waited  0.5s  HTTP 429  next gap  30.0s

The factor, the floor and the ceiling are choices, not findings. The comment on next_ok is the part that bites: a crawler that persists its queue but not its per-host backoff hammers every host it had backed off from the moment it restarts. Reading the status codes themselves, including which ones come from your own proxy rather than the site, is covered in 403 vs 407 vs 429.

Identify yourself, and mean it

At this scale the user agent is a claim you will be held to.

Position What it means in practice
A named agent with a contact URL Operators can reach you and allow you specifically. The only position that lets a site say yes
An honest library default Unremarkable and truthful. Fine for small, permitted work
No user agent at all Stands out, because ordinary traffic almost always carries one
A copied browser string Contradicted by everything else the client sends, and gives up any claim of good faith
User-Agent: ExampleCorpBot/1.0 (+https://example.com/bot)

The browser string fails on its own terms. Our header capture recorded a Chromium navigation sending 11 headers where the median of nine libraries sent 4, in a different order and without the Sec-Fetch-* set, so a browser name over a library connection is a contradiction rather than a disguise. The TLS handshake sits underneath and nothing at the HTTP layer changes it. The same survey of 46 sites found publishers blocking GPTBot 16 times and OAI-SearchBot 7 times: they treat bulk training crawlers and on-demand fetchers differently, so say which one you are.

What breaks once the crawl is large

Failure Symptom Guard
Challenge pages stored as text Thousands of near-identical “verify you are human” documents Assert on content, not on a 200
Silent extraction drift A parser change quietly empties a field for one site Per-source populated-field counts, compared run to run
robots.txt fetch hangs Crawler stalls on one host with no error Fetch the file yourself with a timeout
Parser reads robots.txt differently from the RFC Pages fetched that a compliant crawler would skip, or the reverse Test your parser against the cases above
Redirect loops into one page One document, thousands of URLs Hash before store; record the final URL
Encoding damage Mojibake in a fraction of documents Trust the declared charset, verify by decoding
Unbounded frontier Memory exhaustion days in Bound the queue and persist it
Backoff lost on restart A restart hammers hosts you had backed off from Persist per-host state, not just the queue

The first two are the expensive ones because they are invisible. A success rate measured on status codes stays healthy while the corpus fills with interstitials, since a 200 says a response arrived, not that it was the document. Keep a per-source contract of text the page must contain and an expected minimum length, and alert when either breaks. The RAG pipeline guide shows that check in code; it matters even more there, where one source can be the only answer to a question.

Decide what you will not collect

Some of this is legal and jurisdiction-specific, and none of it is advice. Three categories deserve an explicit decision before the crawl starts:

  • Personal data. Obligations attach to it whether or not the page was public. Being able to find and delete every document from one source is a design requirement.
  • Material behind a login or a paywall. Public and accessible are different things.
  • Sites that asked you not to. Whether or not the file binds you, ignoring it after reading it is a different position from never having looked.

The provenance fields are what make any of this actionable. Without a per-document record of source and fetch time, “remove everything from that domain” is not a query you can run. If the same pipeline also feeds an agent that fetches pages live for users, its traffic and its rules differ; see AI agent proxy setup.

How the parser tests were run

On 21 October 2026, on macOS, the same script ran under Python 3.9.6 (the system interpreter), 3.12.13 and 3.15.0. Eight parsing cases were passed to RobotFileParser.parse() as text, with no network. Seven status cases were fetched with read() from a local HTTP server answering 200, 401, 403, 404, 429, 500 and 503. The refused case used a closed port, and the stalled case used a local listener that accepts and never writes, watched by a thread that gave up after 20 seconds. The RFC 9309 column is our reading of sections 2.2.1, 2.2.2, 2.3.1.3 and 2.3.1.4. The robots.txt loader and the deduplication snippet ran on Python 3.15.0. The pacing test ran on Python 3.12.13, with httpx 0.28.1 making the requests to a local server.

Frequently asked questions

How do you collect training data for an LLM from the web?
Crawl under a named user agent with a contact URL, pace each host by its own response time and Retry-After, read robots.txt per RFC 9309, store the raw response with its URL, timestamp, status, robots verdict and hash, deduplicate by canonical URL, exact hash and then near-duplicate detection, and assert that each page contains what it should before keeping it.
Does Python's urllib.robotparser follow RFC 9309?
Partly, and it depends on the version. In our test Python 3.15.0 applied longest-match, the * and $ wildcards and group merging, while 3.9.6 and 3.12.13 did not. All three still match agents by substring, treat a 401 or 403 as disallow-all, and call urlopen without a timeout in read(). The change is CPython issue gh-138907.
Does robots.txt legally prevent me collecting training data?
No. It is a voluntary protocol, standardised in RFC 9309, that records what a site operator asks crawlers to do. It is not access control and not a licence. It does give you a record of what you were told, which is a different position from never having looked.
What should happen when robots.txt returns an error?
RFC 9309 says a 4xx means the file is unavailable and the crawler may proceed, while a 5xx or a network error means it is unreachable and the crawler must assume complete disallow. Google's documentation excludes 429 from its 4xx rule. Python's parser blocks on 401 and 403, and raises URLError on a refused connection.
Should my crawler use a browser user agent?
No. A browser string over a library connection is contradicted by the header set, the header order and the TLS handshake, which we have measured. It makes the request more distinctive and removes any claim of good faith. A named agent with a contact URL is the only position that lets an operator allow you deliberately.
Why store raw HTML when I only need the text?
Extraction rules improve, and re-extracting from storage costs nothing, while re-fetching millions of pages costs a great deal and may no longer be permitted. WARC stores the raw request and response with their URL, date, address and payload digest, and its revisit record avoids storing unchanged content twice.

Sources

  1. RFC 9309: Robots Exclusion Protocol, grouping, longest match, status codes, caching and the 500 KiB limit
  2. CPython gh-138907: Support RFC 9309 in robotparser
  3. Python 3.15 documentation: urllib.robotparser
  4. Google Search Central: how Google interprets robots.txt status codes
  5. IIPC: The WARC Format 1.1, record types and header fields
  6. RFC 9110: HTTP Semantics, Retry-After (10.2.3)
  7. RFC 6585: status code 429, Too Many Requests

Read this page as Markdown · Quote it freely under CC BY 4.0 with a link back.