TLS fingerprinting
Identifying a client from the structure of its TLS handshake, before any HTTP request is sent.
TLS fingerprinting derives an identifier from how a client opens an encrypted connection. The Client Hello message exposes the cipher suites offered, their order, supported extensions and their order, and version preferences. Different software produces reliably different combinations.
#Why it defeats naive scraping
A script can send any User-Agent it likes, but the TLS handshake is produced by the underlying library. A Python HTTP client claiming to be Chrome still handshakes like Python — and the mismatch between the two is itself a strong signal, arguably stronger than either alone.
This happens before the HTTP request, so the server can decide to block you without ever seeing the request you intended to make.
#What can be done about it
Use a client that reproduces a real browser’s handshake, or drive a real browser. Header-level disguises alone do not address it. Fingerprint databases such as JA3 and its successors are widely used to catalogue these signatures.
#Related
Browser fingerprinting works at a higher layer, using JavaScript. TLS fingerprinting needs no JavaScript at all.
#Where the signal comes from
The first message of a TLS connection, the ClientHello, is sent in plain text before any encryption begins. It lists the cipher suites, extensions, elliptic curves and versions the client supports, in the client’s own order.
Those choices come from the TLS library, not from your code. Every HTTP library therefore has a characteristic handshake, and it is emitted before you have sent a single header. That is why the technique defeats header-level disguise completely: the classification is finished before the user agent arrives.
#What can be changed and what cannot
| Approach | Changes the handshake? |
|---|---|
| Setting headers, including the user agent | No |
| Changing proxy, country or address | No |
| Switching HTTP library within the same TLS stack | Rarely |
| Using a library that mimics a browser stack | Yes |
| Driving a real browser engine | Yes, it is a browser handshake |
The two rows that work are the two that replace the TLS implementation. Everything above them operates at a layer the fingerprint never reaches.
#Checking your own handshake
Services exist that echo the handshake they observed. Compare the result from your scraper with the result from a real browser on the same machine. If they differ, the difference is what the target sees, whatever your headers claim.