User agent
The header a client uses to identify itself. Trivially changed, and therefore weak evidence on its own.
The User-Agent header is a string by which a client describes itself. It is set by the client, so it is a claim rather than a fact.
#Why changing it is not a disguise
Because anyone can set any value, servers treat it as weak evidence and cross-check it against things that are harder to control: the TLS fingerprint, HTTP/2 settings, header ordering and, where JavaScript runs, the browser fingerprint.
An inconsistent claim is worse than an honest one. A string announcing Chrome on Windows, sent with a Python TLS handshake and header order, is a clearer automation signal than leaving the library’s default in place.
#Practical guidance
- Keep it consistent with everything else your client actually does.
- Use current, real strings — versions long out of support are conspicuous.
- Randomising it per request is a strong bot signal; real clients are stable.
- If you are identifying yourself honestly as a crawler, say so and publish contact details.
#Why the header alone convinces nobody
A user agent string is a claim the client makes about itself. Everything else about the connection is evidence, and the evidence is not under the header’s control. When they disagree, the disagreement is itself the signal.
| The header claims | The connection reveals |
|---|---|
| A current Chrome release | A TLS handshake no Chrome produces |
| A desktop browser | No Accept-Language and no Accept-Encoding |
| A browser | Header order no browser sends |
| A human visitor | Perfectly regular request intervals |
Any one of these mismatches is more distinctive than the honest string would have been. A default library user agent is unremarkable; a browser string over a library handshake is not.
#Practical guidance
- Send a realistic, current string if you send a browser one. Versions that no longer exist stand out.
- Send the headers that accompany it. A real browser always sends Accept, Accept-Language and Accept-Encoding.
- Keep it consistent across the session. A client that changes identity between requests is not a browser.
- Consider identifying yourself honestly where the target has no objection. Some operators allow named crawlers they can contact.