AI and agents
Proxies for AI agents that browse
Agent traffic is neither human nor crawler, and its shape gives it away before any header is read. Identity per task, bounded retries, real…
The proxy and web-scraping reference
Look it up. Then make it work.
Guides, definitions and working examples for proxies, scraping and anti-bot systems. Written for the person who has to ship the thing.
Start here
Four tracks, in the order the questions usually arrive.
- 1 Proxy fundamentals What a proxy is, how a request is routed through one, and why the four types exist at all. What is a proxy Forward vs reverse Rotation and sticky sessions
- 2 Choosing a type Match the category to the job before you compare vendors. Getting this wrong costs more than picking the wrong provider. Residential vs datacenter When ISP wins Reading a price per GB
- 3 Running requests Wiring a proxy into curl, Python and a headless browser, with the configuration that usually gets missed. curl and httpx Session reuse Concurrency and timeouts
- 4 When you get blocked Reading the response you actually got. A 403 and a 407 mean very different things and need different fixes. 403 vs 407 vs 429 TLS fingerprinting Retry that is not a loop
Start with the type
Picking the wrong category wastes more money than picking the wrong provider inside it.
Residential
Addresses assigned by consumer ISPs. Hardest to block, priced per GB, and the category with the widest quality spread.
Read more →Datacenter
Fast and cheap, hosted in server ranges. Easily identified as non-residential, so they suit targets that do not check.
Read more →ISP / static
Datacenter-hosted but registered to an ISP. Datacenter speed with a residential registration, billed per IP.
Read more →Mobile
Carrier addresses shared by many real handsets. The most trusted by anti-bot systems and the most expensive.
Read more →Free tools, no sign-up
They run against live connections. Nothing you type is stored.
Proxy checker
Test any endpoint and see the exit IP, country, ISP, ASN and response time it really presents.
IP and leak test
What your connection reveals: address, proxy headers, and a WebRTC check that runs in your browser.
Cost calculator
Enter your monthly volume and see what each provider charges, using prices we have on record with dates.
Latest guides
How the technology works, written to be used rather than skimmed.
AI and agents
Proxies for RAG pipelines
A retrieval corpus fails quietly: it answers fluently from last month's page. How to revalidate rather than refetch, and spot a source changing shape.
AI and agents
Collecting training data for a language model
Provenance, deduplication, rate limits, and the 200 that is not the document — what a bulk text pipeline must record at fetch time or…
Original research
Benchmarks we ran ourselves, with the method and the sample published alongside.
Web scraping
Which AI crawlers do major sites actually block?
46 sites, 18 AI agents, read from robots.txt. ClaudeBot was blocked by 22 and GPTBot by 16, and the ordering is strict.
Anti-bot systems
Browsers vary every TLS handshake. Libraries never do.
Twelve connections from one Chromium browser produced twelve different JA3 hashes. curl and Go produced one each, every time.
Reference
Look-up material for when you already know what you need.
Glossary
One definition per term, in plain language, with the vendor spin removed.
How-to guides
Step-by-step walkthroughs with commands you can paste and run.
Diagnostics
Check an endpoint, read your own connection, and price a volume. No sign-up.
Machine-readable data
Every page as Markdown, the provider set as JSON, and llms.txt for retrieval.
Open data
Take our measurements and check them yourself.
Every study prints the method, the software versions and the date, so you can run it again rather than take our word for it. Every article is available as clean Markdown by adding ?format=md to its URL. All of it is CC BY 4.0 — quote it, cite it, argue with it.