Skip to content
Glossary

What is an anti-bot system?

Definition

Software in front of a website that scores each request on its network, TLS handshake, headers and behaviour, then serves, challenges or blocks it.

The decision is usually made before your request reaches the site’s own code, at a reverse proxy or CDN edge, and it is made from several independent signals. The address you connect from is one of them. It is often not the one that decides.

What it looks at, in the order it can see it

The network comes first: the ASN, whether the range belongs to a hosting company, and the history of the address. Next is the TLS handshake, which arrives before any HTTP and is written by your TLS library. Then the header set, including which headers are present and in what order. Then behaviour across requests: rate, the paths you follow, how regular the timing is. A page that runs JavaScript can add a browser fingerprint and a challenge on top.

Each layer is cheaper to evaluate than the one after it, so most traffic is settled early. Only two of them react to a new address at all. Rotating the address resets per-address counters in the behaviour layer and can change the network class. It does nothing to the handshake, the headers or the fingerprint, which travel through any proxy unchanged. That is why moving to residential exits so often leaves a block exactly where it was. The proxy type guide has a test for this that costs nothing to run.

What eight protected sites returned

We requested the home page of eight well-known sites from one consumer connection, with no proxy. Each was fetched with curl 8.7.1 at its defaults and with headless Chrome 155 at its defaults, and we recorded the status, the page title and the vendor the response names.

Site Status to curl Status to headless Chrome Title of the page Chrome received Vendor named in the response
crunchbase.com 200 200 Crunchbase | Private Company Data & Predictive Intelligence server: cloudflare
g2.com 403 403 g2.com Cloudflare edge, x-datadome: protected
glassdoor.com 403 403 Just a moment… server: cloudflare
indeed.com 403 403 Blocked – Indeed.com server: cloudflare
zillow.com 403 403 Access to this page has been denied PerimeterX script, px-captcha
etsy.com 403 403 etsy.com server: DataDome
leboncoin.fr 403 403 leboncoin.fr x-datadome: protected, via CloudFront
tripadvisor.com 403 403 tripadvisor.com server: DataDome

Seven of the eight refused both clients, so a real browser engine on its own did not get through. That is less surprising than it sounds: Chrome in headless mode writes HeadlessChrome/155.0.0.0 into its user agent, as the headless browser entry shows.

Two details say more than the headline. Glassdoor’s edge answered curl with a 9-byte body reading Forbidden, and answered Chrome with a JavaScript challenge page that loaded 13 resources. Same site, same moment, a different layer applied to each client. And Etsy’s 403 to curl carried x-datadome-riskscore: 0.9542847904048648, the score itself, in a response header.

This is one connection and one moment. Another address, another hour or another client version can give different answers, and nothing here ranks one vendor against another.

Working out which layer refused you

  • A flat refusal with no challenge, like the one curl got from Glassdoor, points at the network or transport layer. Repeat from a different class of address, then with a real browser on the same address. Whichever change flips the result names the layer.
  • A challenge page means the client layer wants JavaScript to run and pass its checks. A new address does not change that.
  • Success that turns into failure after a while is the behaviour layer. Reduce concurrency and pace the requests. 403 vs 407 vs 429: reading proxy and block errors has a decision tree for this case.
  • A 200 with the wrong content is a block too. Check for a field you need rather than trusting the status.

The two layers people underestimate each have a proxy.wiki study: the handshake in the TLS ClientHello fingerprint study, and the header set in What HTTP clients send before you configure anything.

How this was measured. 21 October 2026, one consumer connection in Europe, no proxy. curl 8.7.1 with no options. Chrome 155.0.8059.85 launched with --headless and a fresh profile, driven over the DevTools protocol, which recorded the document status, the server header and document.title. Each client was run against each site twice, minutes apart, and got the same status both times. Vendor names come from headers and script references in the responses.

Frequently asked questions

Why am I still blocked after switching to residential proxies?
Because the address is one layer of several. The TLS handshake, the header set and any JavaScript challenge are produced by your client and pass through any proxy unchanged, so a new address leaves them exactly as they were.
How can I tell which anti-bot vendor a site uses?
Read the response. Vendors usually name themselves in a header or a script reference. In our captures that meant server: DataDome or x-datadome: protected, server: cloudflare with a challenge script, and a px-captcha script on a PerimeterX block page.
Do anti-bot systems always return an error?
No. Seven of the eight sites we tested returned 403, but systems can also serve a page with the data missing or slow the response down. A pipeline that checks only status codes records those as successes, so validate the content.

Sources

  1. RFC 9110: HTTP Semantics, 403 Forbidden
  2. RFC 8446: TLS 1.3, the ClientHello sent before any HTTP
  3. Cloudflare documentation: Challenges

All terms, A to Z