Skip to content
Topic

Anti-bot systems

How automated traffic is identified, layer by layer, and which layers a different address actually affects.

The single most useful thing to understand about blocking is that it happens in stages, and each stage is cheaper for the defender than the next. Work out which one refused you and the next step is obvious. Guess, and you will spend money on proxies to fix a problem proxies cannot reach.

The layers, in the order a request meets them

LayerDecides onDoes a new address help?
Network reputationThe ASN and address historyYes, if the new address is classified differently
TransportThe TLS handshakeNo. The handshake is unchanged
ProtocolHeader set, order and HTTP versionNo
BehaviourTiming, paths, volume per identityResets the counter, not the pattern
Client challengeJavaScript execution, CAPTCHANo

Only the first and fourth rows respond to a new address at all. This is why "we switched to residential proxies and still get blocked" is such a common report: the decision was never about where the request appeared to come from.

Why the handshake beats header disguise

The first message of a TLS connection is sent in plain text before any encryption, and it lists the cipher suites and extensions your TLS library supports, in your library's order. That is emitted before you have sent a single header, so the classification is finished before the user agent ever arrives.

We measured how far apart the clients actually are. A byte-level capture of nine HTTP clients and a browser found the browser sending eleven headers against a median of four, no two clients producing the same header order, and two clients sending no user agent at all. None of it changes with a proxy.

Reading what you were given

The response itself narrows the cause considerably:

  • An instant refusal with no content — network or transport. Compare against a request from a different category of address.
  • A challenge page — the client layer. Load the same page in a real browser on your own connection; if that is challenged too, the address is not the problem.
  • Success, then failure after a while — behaviour. Reduce concurrency and pace the requests.
  • Failure on some paths only — application rules. The address is not the variable.

The status-code guide covers the three refusals you will meet most often and what each one obliges you to do differently.

Consistency, not disguise

Fingerprinting rewards ordinariness and punishes contradiction. No single property is rare; the combination is. Changing one value to something unusual does not blend you in, it moves you to a smaller group.

This is why partial disguise reliably backfires. A browser user agent over a library handshake, or an exit in one country with a browser reporting another country's timezone, is more distinctive than sending nothing at all.

In this section

18 pages
Glossary Term

What is a honeypot link?

A honeypot link is a link placed where no human visitor will see or follow it, so that any client requesting it reveals itself…

4 min read

Glossary Term

What is an anti-bot system?

Software in front of a website that scores each request on its network, TLS handshake, headers and behaviour, then serves, challenges or blocks it.

4 min read

Glossary Term

What is browser fingerprinting?

Identifying a visitor from properties a page's JavaScript can read, such as screen, GPU, plugins, languages and timezone, combined into one profile.

4 min read

Glossary Term

What is HTTP/2 fingerprinting?

An HTTP/2 fingerprint identifies a client from the settings, window size and pseudo-header order it uses to open a connection, not from header values.

4 min read

Glossary Term

What is TLS fingerprinting?

Identifying the software behind a connection from its TLS ClientHello, which is sent in the clear before any HTTP header.

4 min read