Anti-bot systems

The single most useful thing to understand about blocking is that it happens in stages, and each stage is cheaper for the defender than the next. Work out which one refused you and the next step is obvious. Guess, and you will spend money on proxies to fix a problem proxies cannot reach.

The layers, in the order a request meets them

Only the first and fourth rows respond to a new address at all. This is why "we switched to residential proxies and still get blocked" is such a common report: the decision was never about where the request appeared to come from.

Why the handshake beats header disguise

The first message of a TLS connection is sent in plain text before any encryption, and it lists the cipher suites and extensions your TLS library supports, in your library's order. That is emitted before you have sent a single header, so the classification is finished before the user agent ever arrives.

We measured how far apart the clients actually are. A byte-level capture of nine HTTP clients and a browser found the browser sending eleven headers against a median of four, no two clients producing the same header order, and two clients sending no user agent at all. None of it changes with a proxy.

Reading what you were given

The response itself narrows the cause considerably:

• An instant refusal with no content — network or transport. Compare against a request from a different category of address.
• A challenge page — the client layer. Load the same page in a real browser on your own connection; if that is challenged too, the address is not the problem.
• Success, then failure after a while — behaviour. Reduce concurrency and pace the requests.
• Failure on some paths only — application rules. The address is not the variable.

The status-code guide covers the three refusals you will meet most often and what each one obliges you to do differently.

Consistency, not disguise

Fingerprinting rewards ordinariness and punishes contradiction. No single property is rare; the combination is. Changing one value to something unusual does not blend you in, it moves you to a smaller group.

This is why partial disguise reliably backfires. A browser user agent over a library handshake, or an exit in one country with a browser reporting another country's timezone, is more distinctive than sending nothing at all.