Skip to content
Topic

Web scraping

Collecting data from the web at scale: client configuration, reading the failures correctly, and the silent bug that returns nothing.

Most scraping problems are not scraping problems. They are HTTP problems, session problems or measurement problems wearing a different hat, and they are much easier to fix once named correctly. That is what this section is organised around.

Start with the client

Whatever language you use, the first thing to get right is the request itself. Each of these guides was written by running the commands and recording what happened, not by summarising documentation:

  • Proxies with curl — the fastest way to prove an endpoint works at all, and the socks5h trap that fails silently.
  • Proxies in Python — requests, httpx and aiohttp, with session reuse and the TLS caveat.
  • Proxies in Node.js — the runtime that ignores the proxy environment variables entirely.
  • The environment variables — why HTTP_PROXY in uppercase is the one spelling curl deliberately refuses.

Then read the failures correctly

A block tells you which layer refused you, and that decides what to change. Changing your address when the refusal came from the transport layer wastes money and time.

  • 403, 407 and 429 — three refusals with three different meanings and three different fixes.
  • Rate limiting — what Retry-After obliges you to do, and why rotating around a limit often fails.
  • Anti-bot systems — the layers a request passes through, and which of them a new address actually affects.

The failure nobody notices

The most expensive scraping bug is not a block. It is a job that reports success while collecting nothing.

A challenge page returns 200. A consent wall returns 200. A page served to a suspected bot with the data stripped out returns 200. Any pipeline that measures health by status code is blind to all three, and it stays blind until somebody opens the output weeks later.

The fix is to assert on content. Require a field you actually need to be present and plausible before a record is accepted, and compare each run's populated-field counts against the previous one. This is covered in success rate, which also explains why published success figures are usually not comparable with each other.

Pacing and etiquette

Concurrency causes more blocks than total volume does, because parallelism turns an unremarkable request rate into a burst. Raise it in steps and watch the measured success rate rather than the throughput; the right setting is the highest one at which quality has not started to fall.

robots.txt is worth reading before a large crawl. It is advisory rather than enforceable, standardised in RFC 9309, and it records what the site operator asks of crawlers. Ignoring it rarely blocks you and does remove any claim that you were not told.

Browsers, and when you do not need one

A headless browser costs orders of magnitude more than an HTTP request, so establish that you need one first. Open the network panel and watch what the page fetches: sites that appear to require rendering very often pull their content from a single API endpoint you can call directly.

Where you do need one, note that its defaults announce it. Automation flags, default window sizes and missing plugin lists are all readable by the page, and a browser left at its defaults is easier to identify than a plain HTTP client.

In this section

29 pages
Glossary Term

What is a headless browser?

A real browser engine run from code with no visible window. It executes JavaScript like a normal browser, at many times the cost of…

4 min read

Glossary Term

What is a proxy success rate?

The share of requests that return a usable response. Without a sample size, a target, a date and a definition of success, the number…

3 min read

Glossary Term

What is a user agent?

The User-Agent request header, in which a client names its software and version. The client sets it, so servers treat it as a claim.

3 min read

Glossary Term

What is crawl budget?

Crawl budget is Google's term for the set of URLs Googlebot can and wants to crawl on a site, set by server capacity and…

4 min read

Glossary Term

What is exponential backoff?

Exponential backoff is a retry policy in which the wait before each new attempt grows by a fixed factor, usually doubling, up to a…

3 min read

Glossary Term

What is proxy concurrency?

How many requests or connections run at the same time. Often the limit you hit before bandwidth, and the setting that decides when a…

3 min read

Glossary Term

What is rate limiting?

A cap on how many requests a client may make in a time window. Going over it usually returns 429 Too Many Requests, often…

4 min read

Glossary Term

What is robots.txt?

A plain-text file at the root of a site that tells crawlers, by name, which paths to stay out of.

3 min read

Glossary Term

What is the Retry-After header?

Retry-After is an HTTP response header telling the client how long to wait, in seconds or until an HTTP-date, before trying again.

3 min read

Glossary Term

What is web scraping?

Using software to fetch web pages and turn them into structured records. The hard part is knowing the page you got is the page…

4 min read