408 Request Timeout: what it means and how to fix it
408 Request Timeout means the server stopped waiting for the rest of your request and closed the connection. Resend it once on a new…
Collecting data from the web at scale: client configuration, reading the failures correctly, and the silent bug that returns nothing.
Most scraping problems are not scraping problems. They are HTTP problems, session problems or measurement problems wearing a different hat, and they are much easier to fix once named correctly. That is what this section is organised around.
Whatever language you use, the first thing to get right is the request itself. Each of these guides was written by running the commands and recording what happened, not by summarising documentation:
socks5h trap that fails silently.HTTP_PROXY in uppercase is the one spelling curl deliberately refuses.A block tells you which layer refused you, and that decides what to change. Changing your address when the refusal came from the transport layer wastes money and time.
Retry-After obliges you to do, and why rotating around a limit often fails.The most expensive scraping bug is not a block. It is a job that reports success while collecting nothing.
A challenge page returns 200. A consent wall returns 200. A page served to a suspected bot with the data stripped out returns 200. Any pipeline that measures health by status code is blind to all three, and it stays blind until somebody opens the output weeks later.
The fix is to assert on content. Require a field you actually need to be present and plausible before a record is accepted, and compare each run's populated-field counts against the previous one. This is covered in success rate, which also explains why published success figures are usually not comparable with each other.
Concurrency causes more blocks than total volume does, because parallelism turns an unremarkable request rate into a burst. Raise it in steps and watch the measured success rate rather than the throughput; the right setting is the highest one at which quality has not started to fall.
robots.txt is worth reading before a large crawl. It is advisory rather than enforceable, standardised in RFC 9309, and it records what the site operator asks of crawlers. Ignoring it rarely blocks you and does remove any claim that you were not told.
A headless browser costs orders of magnitude more than an HTTP request, so establish that you need one first. Open the network panel and watch what the page fetches: sites that appear to require rendering very often pull their content from a single API endpoint you can call directly.
Where you do need one, note that its defaults announce it. Automation flags, default window sizes and missing plugin lists are all readable by the page, and a browser left at its defaults is easier to identify than a plain HTTP client.
408 Request Timeout means the server stopped waiting for the rest of your request and closed the connection. Resend it once on a new…
431 means the server refused your request because its headers are too big, in total or in one field such as Cookie. Trim them…
curl exit code 52 means the connection closed before one response byte arrived. Eight causes reproduced, each with the check that isolates it and…
Set a git proxy with http.proxy; https.proxy is silently ignored. npm and pip equivalents and their real errors, tested on git 2.54, npm 11.6,…
Node fetch proxy setup that works: NODE_USE_ENV_PROXY, undici ProxyAgent, axios and got, with the version traps and exact errors from a logging proxy.
curl -x syntax for HTTP and SOCKS5 proxies, plus 15 proxy failures with the exact error curl 8.7.1 and 8.23.0 printed for each.
A 410 Gone means the server removed the URL on purpose and expects it to stay gone. What that means for a scraper or…
HTTP 412 Precondition Failed means an If-Match, If-None-Match or If-Unmodified-Since check came out false. Causes, a local repro and fixes.
HTTP 413 means a server or proxy refused the request because its body is bigger than the limit it accepts. Shrink the body or…
HTTP 414 URI Too Long means the request target exceeds a server or proxy limit. Shorten the query or move state into a request…
HTTP 423 Locked means a WebDAV resource is protected by a write lock. Find the lock token or wait for expiry before retrying the…
HTTP 424 Failed Dependency means a request failed because another action it relied on failed. Where it comes from, a local repro and fixes.
HTTP 429 Too Many Requests means a server or proxy is rate limiting you. Honour Retry-After, lower concurrency and bound retries.
HTTP 507 Insufficient Storage means the server cannot store the representation needed for the request. Check quota and disk before retrying uploads.
HTTP 508 Loop Detected means a WebDAV operation or proxy route found a cycle. Inspect bindings and Via hops before retrying.
httpx vs requests, run side by side: timeouts, redirects, proxy precedence, HTTP/2, async and CPU cost, with real output from both.
Tested proxy setup for requests 2.34.2, httpx 0.28.1 and aiohttp 3.14.5, with the exact exception each raised when the proxy failed.
SSL handshake failed, reproduced in curl, Python, Node and Chrome: what each error string means, how to confirm the cause, and the fix.
A real browser engine run from code with no visible window. It executes JavaScript like a normal browser, at many times the cost of…
The share of requests that return a usable response. Without a sample size, a target, a date and a definition of success, the number…
The User-Agent request header, in which a client names its software and version. The client sets it, so servers treat it as a claim.
Crawl budget is Google's term for the set of URLs Googlebot can and wants to crawl on a site, set by server capacity and…
Exponential backoff is a retry policy in which the wait before each new attempt grows by a fixed factor, usually doubling, up to a…
How many requests or connections run at the same time. Often the limit you hit before bandwidth, and the setting that decides when a…
A cap on how many requests a client may make in a time window. Going over it usually returns 429 Too Many Requests, often…
A plain-text file at the root of a site that tells crawlers, by name, which paths to stay out of.
Retry-After is an HTTP response header telling the client how long to wait, in seconds or until an HTTP-date, before trying again.
Using software to fetch web pages and turn them into structured records. The hard part is knowing the page you got is the page…
robots.txt from 46 sites, scored for 18 AI agents. ClaudeBot was blocked by 22, GPTBot by 16, and every GPTBot block was also a…