---
title: "What HTTP clients send before you configure anything"
url: https://proxy.wiki/research/default-request-headers/
type: Research Report
author: "proxy.wiki editorial"
published: 2026-08-29
updated: 2026-08-29
site: proxy.wiki
topics: ["Anti-bot systems"]
license: CC BY 4.0 — quote freely with attribution to https://proxy.wiki/
---

# What HTTP clients send before you configure anything

> Nine HTTP clients and a browser, captured at the byte level. The browser sent 11 headers; the median library sent 4, in a different order.

## Key takeaways

- A Chromium navigation sent 11 headers. The nine libraries tested sent between 2 and 7, with a median of 4.
- No two clients produced the same header order, and none matched the browser, so order alone separates all ten.
- Node 24 global fetch is the only client that lowercases every header name, and it sends the user agent "node".
- node:http and PHP file_get_contents send no User-Agent at all, only Host and Connection.
- Only python-requests and python-httpx advertise the same Accept-Encoding set as the browser.
- None of this changes with a proxy: every signal here is produced by the client, not by the network path.

Every HTTP client sends headers you did not write. They come from the library, they are consistent, and they differ enough between libraries to identify which one made the request. This study records exactly what nine clients send by default, captured at the byte level, and compares them against a real browser.

The practical question behind it: how far is a default HTTP client from a browser, before you change anything? The answer is further than most people assume, and the gap is visible in the first four lines of the request.

## Method

A raw TCP listener on `127.0.0.1` accepted a connection, read bytes until the end of the header block, recorded them verbatim, and returned a minimal 200. No proxy, no TLS and no framework sat in the path, so what is recorded is what the client put on the wire.

Each client made one plain `GET` to the same path with no options, no configured headers and no session reuse. The browser row is a top-level navigation from a Chromium-based browser to the same address. The full listener is at the end of this page, so any result here can be reproduced in about a minute.

All measurements were taken on macOS on 29 August 2026. Versions are given in the table. Behaviour changes between releases, so treat every row as describing that version rather than the library in general.

## What each client sent

| Client | Headers | User-Agent | Header order as sent |
| --- | --- | --- | --- |
| curl 8.7.1 | 3 | Yes | Host, User-Agent, Accept |
| GNU wget 1.25.0 | 5 | Yes | Host, User-Agent, Accept, Accept-Encoding, Connection |
| python-requests 2.32.3 | 5 | Yes | Host, User-Agent, Accept-Encoding, Accept, Connection |
| python-urllib 3.9 | 4 | Yes | Accept-Encoding, Host, User-Agent, Connection |
| python-httpx 0.27.2 | 5 | Yes | Host, Accept, Accept-Encoding, Connection, User-Agent |
| Node 24.2.0 global fetch | 7 | Yes | host, connection, accept, accept-language, sec-fetch-mode, user-agent, accept-encoding |
| Node 24.2.0 node:http | 2 | No | Host, Connection |
| Go 1.24 net/http | 3 | Yes | Host, User-Agent, Accept-Encoding |
| PHP file_get_contents | 2 | No | Host, Connection |
| Chromium, navigation | 11 | Yes | Host, Connection, Upgrade-Insecure-Requests, User-Agent, Accept, Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-User, Sec-Fetch-Dest, Accept-Encoding, Accept-Language |

**No two clients produced the same header order.** Order alone separates all ten, and none of the nine libraries matches the browser.

## Finding 1: the browser sends nearly four times as many headers

Eleven against a median of four. More importantly, the extra headers are not decorative. Four of them belong to the Fetch Metadata set, which a browser emits to describe the context of the request:

```
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
```

Not one of the nine libraries sent these on a plain request. Node’s `fetch` sent a single `sec-fetch-mode: cors` and nothing else from the set. A server that expects a navigation and receives no `Sec-Fetch-Dest` has learned something before it reads the [user agent](/glossary/user-agent/).

## Finding 2: Node’s fetch lowercases every header name

This is the most distinctive single result in the study. HTTP header names are case-insensitive, so lowercasing is legal. It is also unique here: every other client, and the browser, sent canonical capitalisation.

```
# Node 24 global fetch, exactly as sent
GET /probe HTTP/1.1
host: 127.0.0.1:8799
connection: keep-alive
accept: */*
accept-language: *
sec-fetch-mode: cors
user-agent: node
accept-encoding: gzip, deflate
```

Two further details in those eight lines. The user agent is the bare string `node`. And `accept-language: *` is a value no browser sends, because a browser states real language preferences. A request claiming to be a browser through this client contradicts itself three ways before the first byte of the body.

## Finding 3: two clients send no user agent at all

`node:http` and PHP’s `file_get_contents` each sent two headers: `Host` and `Connection`. Nothing else. A two-header request is not a browser request, and it is not close enough for the absence to be ambiguous.

This matters more than a wrong user agent does. Filters can treat a missing `User-Agent` as a stronger signal than an unrecognised one, because ordinary traffic almost always carries something.

## Finding 4: Accept-Encoding is the widest divergence

| Value sent | Clients |
| --- | --- |
| Not sent at all | curl, node:http, PHP |
| identity | wget, python-urllib |
| gzip | Go net/http |
| gzip, deflate | Node global fetch |
| gzip, deflate, br, zstd | python-requests, python-httpx, Chromium |

Only two of the nine libraries advertise the same encoding set as the browser. The rest are distinguishable on this line alone. There is also a cost implication: a client asking for `identity` receives uncompressed responses, and on [metered bandwidth](/glossary/bandwidth-billing/) that is billable traffic spent on nothing.

## Finding 5: header order is stable, and it is a signature

Repeated runs produced identical ordering for each client. Order is decided by the library’s internal structures, not by the request, so it is consistent and it is not something you set. Three orderings are worth noting:

- **python-urllib puts `Accept-Encoding` before `Host`.** No other client tested places anything before `Host`.

- **python-httpx puts `User-Agent` last.** Every other client that sends one places it second or third.

- **The browser puts `Connection` second**, then `Upgrade-Insecure-Requests`, which no library sent at all.

Setting a header explicitly usually replaces the value, not the position, so a copied browser header set arrives in the library’s order rather than the browser’s.

## What this means in practice

Three conclusions follow from the data, and one non-conclusion.

**Copying a browser user agent onto a default client makes the request more distinctive, not less.** The claim is one line; the contradiction is spread across the header count, the ordering, the missing Fetch Metadata set and the encoding list. Changing one field while leaving the rest untouched creates a combination that neither a browser nor an honest library produces.

**A proxy changes none of this.** Every result here is produced by the client, so it is identical through any proxy, in any country, from any [address category](/glossary/residential-proxy/). This is the same reason [TLS fingerprinting](/glossary/tls-fingerprinting/) is unaffected by proxy choice, and it explains a common and expensive misunderstanding about what better proxies buy you.

**If you want to look like a browser, send what a browser sends.** That means all eleven headers, in the browser’s order, with the Fetch Metadata set matching the kind of request you are making. Anything less is a partial disguise, and a partial disguise is a distinctive one.

The non-conclusion: none of this establishes that any particular site uses these signals, or how it weights them. This study measures what clients emit, not what servers do with it. Treat it as a map of what is visible, not as evidence about any given target’s defences.

## How much of the gap you can actually close

The header layer is the cheapest layer to fix and the one that closes least. It is worth being precise about what each step buys.

| Step | Closes | Leaves open |
| --- | --- | --- |
| Set a current browser user agent | One line | Count, order, Fetch Metadata, encodings, handshake |
| Send the browser’s full header set | Count and content | Order, the handshake |
| Send them in the browser’s order | Order, where the library permits it | The handshake |
| Use a client that mimics a browser TLS stack | The handshake | Behaviour and timing |
| Drive a real browser engine | Everything above | Cost, and automation defaults |

Two things follow. The first three rows are free and none of them reaches the [handshake](/glossary/tls-fingerprinting/), which is sent before any header. And most libraries do not let you control header order at all, so row three is often unavailable without dropping to a lower-level client.

This is why the ladder matters more than any single step. Stopping after row one leaves the largest signal untouched while adding a contradiction, which is measurably worse than sending nothing. If the target is defended enough that headers matter, the work does not end at the header layer, and if it is not, the default user agent was never the problem.

## Reproducing it

The listener is forty lines. Run it, point a client at the port, and read the file it writes.

```
import socket, json, threading

def handle(conn):
    conn.settimeout(3)
    data = b""
    while b"\r\n\r\n" not in data and len(data) < 65536:
        chunk = conn.recv(4096)
        if not chunk:
            break
        data += chunk
    conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 3\r\n"
                 b"Connection: close\r\n\r\nok\n")
    conn.close()
    with open("captures.jsonl", "a") as f:
        f.write(json.dumps({"raw": data.decode("latin-1")}) + "\n")

s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(("127.0.0.1", 8799))
s.listen(64)
while True:
    c, _ = s.accept()
    threading.Thread(target=handle, args=(c,), daemon=True).start()
```

Then send one request from each client you care about:

```
curl -s http://127.0.0.1:8799/probe -o /dev/null
wget -q -O /dev/null http://127.0.0.1:8799/probe
python3 -c "import requests; requests.get('http://127.0.0.1:8799/probe')"
node -e "fetch('http://127.0.0.1:8799/probe')"
```

Run it against your own versions. The point of publishing the method rather than only the table is that your results are the ones that apply to you.

## Limits of this study

- **One request per client, on one platform, on one date.** Ordering was stable across repeats, but defaults change between releases and can differ by operating system.

- **Plain HTTP only.** This measures the header layer. The [TLS handshake](/glossary/tls-fingerprinting/) carries a separate signature that arrives earlier and is not covered here.

- **One browser.** The browser row is a Chromium build. Firefox and Safari send different sets, and this study does not measure them.

- **HTTP/1.1 only.** The listener speaks HTTP/1.1, so every capture here is HTTP/1.1. Over HTTP/2 the request is carried as compressed header fields preceded by pseudo-headers, and the encoding differs, so these orderings should not be assumed to transfer unchanged to an HTTP/2 connection.

- **No claim about detection.** What servers do with these differences is outside what a listener can observe. A signal being visible is not evidence that it is used.

## Frequently asked questions

### Does changing the user agent make my scraper look like a browser?

No, and it can make things worse. In this study the browser sent 11 headers and the median library sent 4, in a different order, without the Fetch Metadata set and often with a different Accept-Encoding list. Changing one line while leaving the rest untouched creates a combination neither a browser nor an honest client produces.

### Will better proxies fix this?

No. Every result in this study is produced by the client, so it is identical through any proxy, from any country and from any category of address. A proxy changes which address the server sees and nothing about the request itself.

### Which client is closest to a browser by default?

Node's global fetch sent the most headers of the libraries tested, at seven, but it is also the most distinctive, because it is the only one that lowercases every header name and it sends the user agent string "node". By header count and content, python-requests and python-httpx are closer to ordinary browser traffic.

### What are the Sec-Fetch headers and why do they matter?

They are the Fetch Metadata request headers, which a browser sends to describe the context of a request: its origin relationship, its mode, whether a user initiated it and what kind of resource it expects. No library tested sent the full set, so their absence distinguishes a library from a browser regardless of the user agent.

### Can I reproduce these results?

Yes, and the full listener is printed in the article. It is about forty lines of Python with no dependencies. Run it, send one request from each client you use, and read the captured bytes. Your own versions are the ones that apply to you.

## Sources

1. [RFC 9110: HTTP Semantics, the User-Agent, Accept and Accept-Encoding fields](https://www.rfc-editor.org/rfc/rfc9110.html)
2. [W3C: Fetch Metadata Request Headers, the Sec-Fetch-* set](https://www.w3.org/TR/fetch-metadata/)