What is a CAPTCHA, and why do scrapers get one?
Definition
A test a website shows when it doubts that a visitor is human, from distorted text and image grids to a single checkbox.
For a scraper, a challenge is the visible end of a decision already made from signals it sent earlier: the address, the handshake, the headers, the rate. The scoring is mostly invisible. Google’s reCAPTCHA v3 returns a score between 0.0 and 1.0 for every request and shows the visitor nothing, and Cloudflare describes its challenges as evaluating client-side signals or asking for a minimal action such as ticking a box. A visible puzzle is usually the fallback for a poor score.
The name is an acronym, Completely Automated Public Turing test to tell Computers and Humans Apart, from the 2003 paper by von Ahn, Blum, Hopper and Langford that formalised the idea. The test has changed shape since then. Its job has not: separate people from programs cheaply enough to run on every visit.
What a challenge looks like on the wire
Usually a 403 with a page that needs JavaScript. When we fetched Glassdoor’s home page, curl 8.7.1 got a 403 and a 9-byte body reading Forbidden. Headless Chrome 155 got a 403 too, but with a page titled “Just a moment…” that loaded 13 resources and waited for its script to run. Etsy answered curl with a 776-byte page whose only visible text was:
Please enable JS and disable any ad blocker
Its one script came from ct.captcha-delivery.com, and the response headers named DataDome. In all three cases the status code alone did not say whether the client was refused or challenged. The body did. A pipeline that logs only status codes cannot tell the two apart, and they call for different fixes.
Find the trigger before solving anything
A challenge is the output of a score. Work out what moved the score, cheapest test first, changing one thing at a time.
| Suspect | Test | If the challenge goes away |
|---|---|---|
| Your own setup, not the address | Load the page in a real browser on your own connection | If that is challenged too, no proxy will fix it |
| Address class | Repeat with the same client from a different category of address | The network layer was the trigger |
| TLS handshake | Repeat with a real browser over the same proxy | Your library’s handshake was the trigger |
| Header set | Send the headers a browser sends, in its order | Missing or reordered headers were the trigger |
| Rate | Halve concurrency and add jittered delays | The behaviour layer was the trigger |
| Missing cookies | Load the entry page first and keep the cookie jar | The site expected a session |
Run the first row before you buy anything. It takes a minute, and it is easy to skip when the instinct is to buy better proxies. The decision procedure in the proxy type guide starts from the same idea: find out whether the target checks at all before choosing what to buy. Telling a challenge from a plain refusal, and from a block that returns 200, is covered in 403 vs 407 vs 429: reading proxy and block errors.
The handshake and header rows are the ones people underestimate, because they are invisible from inside your own code. Both are measured in proxy.wiki studies: the TLS ClientHello fingerprint study and What HTTP clients send before you configure anything.
Why paying to solve is the last step
Solving treats the output. It costs money per challenge, adds latency to every request that gets one, and leaves the signal that triggered it in place, so the challenges keep coming. Fixing the trigger removes the cost instead of paying it again on each request.
Solving services also sit in contested territory. Many sites’ terms forbid getting around their challenges, and some jurisdictions restrict circumventing access controls. That is a legal and contractual question, separate from whether it works. Read the target’s terms before adopting one, and take advice if the work is at any scale. The anti-bot system entry describes the layers a challenge sits on top of.
How the examples were observed. 21 October 2026, one consumer connection in Europe, no proxy. curl 8.7.1 with no options, and Google Chrome 155.0.8059.85 with --headless and a fresh profile. One request per client per site, repeated minutes later with the same status.