Web scraping
Using proxies in Node.js: fetch, undici, axios and got
Node ignores the proxy environment variables entirely. Working configuration for every client, tested against a proxy that logs each request.
The proxy and web-scraping reference
Look it up. Then make it work.
Guides, definitions and working examples for proxies, scraping and anti-bot systems. Written for the person who has to ship the thing.
Start here
Four tracks, in the order the questions usually arrive.
- 1 Proxy fundamentals What a proxy is, how a request is routed through one, and why the four types exist at all. What is a proxy Forward vs reverse Rotation and sticky sessions
- 2 Choosing a type Match the category to the job before you compare vendors. Getting this wrong costs more than picking the wrong provider. Residential vs datacenter When ISP wins Reading a price per GB
- 3 Running requests Wiring a proxy into curl, Python and a headless browser, with the configuration that usually gets missed. curl and httpx Session reuse Concurrency and timeouts
- 4 When you get blocked Reading the response you actually got. A 403 and a 407 mean very different things and need different fixes. 403 vs 407 vs 429 TLS fingerprinting Retry that is not a loop
Start with the type
Picking the wrong category wastes more money than picking the wrong provider inside it.
Residential
Addresses assigned by consumer ISPs. Hardest to block, priced per GB, and the category with the widest quality spread.
Read more →Datacenter
Fast and cheap, hosted in server ranges. Easily identified as non-residential, so they suit targets that do not check.
Read more →ISP / static
Datacenter-hosted but registered to an ISP. Datacenter speed with a residential registration, billed per IP.
Read more →Mobile
Carrier addresses shared by many real handsets. The most trusted by anti-bot systems and the most expensive.
Read more →Free tools, no sign-up
They run against live connections. Nothing you type is stored.
Proxy checker
Test any endpoint and see the exit IP, country, ISP, ASN and response time it really presents.
IP and leak test
What your connection reveals: address, proxy headers, and a WebRTC check that runs in your browser.
Cost calculator
Enter your monthly volume and see what each provider charges, using prices we have on record with dates.
Latest guides
How the technology works, written to be used rather than skimmed.
Proxy fundamentals
HTTP_PROXY, HTTPS_PROXY and NO_PROXY explained
The four proxy environment variables, tested across curl, wget, Python, Go and Node, including the case rule that silently ignores HTTP_PROXY.
Buying guides
Residential vs datacenter vs ISP vs mobile: which to use
A decision procedure for choosing a proxy type, based on what the target actually checks rather than on price.
Original research
Benchmarks we ran ourselves, with the method and the sample published alongside.
Anti-bot systems
What HTTP clients send before you configure anything
Nine HTTP clients and a browser, captured at the byte level. The browser sent 11 headers; the median library sent 4, in a different…
Reference
Look-up material for when you already know what you need.
Glossary
One definition per term, in plain language, with the vendor spin removed.
How-to guides
Step-by-step walkthroughs with commands you can paste and run.
Diagnostics
Check an endpoint, read your own connection, and price a volume. No sign-up.
Machine-readable data
Every page as Markdown, the provider set as JSON, and llms.txt for retrieval.
Open data
Take our measurements and check them yourself.
Every study prints the method, the software versions and the date, so you can run it again rather than take our word for it. Every article is available as clean Markdown by adding ?format=md to its URL. All of it is CC BY 4.0 — quote it, cite it, argue with it.