Skip to content
Topic

Proxy fundamentals

How proxies work, what the four categories actually differ on, and the protocol detail underneath.

Nearly every practical proxy question reduces to one of three things: who owns the address, how the connection is carried, and how long you keep the same identity. The rest is detail hanging off those three.

The one idea that explains the categories

The four categories do not differ on protocol. They differ on who the address is registered to, and everything else follows from that.

  • Datacenter — registered to a hosting company, so identifiable from the registration alone.
  • ISP — registered to a consumer provider, hosted in a datacentre, assigned to you exclusively.
  • Residential — a real consumer line, usually shared and rotating.
  • Mobile — a carrier address shared with many real subscribers behind carrier-grade NAT.

The comparison guide works through which to use when. The short version: registration decides how the address is treated before it does anything, and the ASN is where that decision is read from.

How the connection is carried

An HTTPS request through an HTTP proxy is not forwarded, it is tunnelled. The client asks the proxy to open a tunnel with CONNECT, the proxy answers, and the encrypted traffic passes through untouched. This is why the proxy sees the destination hostname and the byte counts but not the contents.

SOCKS5 works one layer lower and carries any TCP stream. It cannot add HTTP headers, which makes it structurally quieter, and it has one option that catches everybody: whether your machine or the proxy resolves the hostname.

Identity over time

A rotating endpoint gives you a different address per request or per interval. A sticky session holds one for a while. Neither is better; they answer different questions.

The guide on choosing between them covers the failure this causes most often: a crawler rotating per request that cannot understand why a paginated list keeps returning page 1. The site bound the cursor to a session, the session to an address, and the address changed.

Getting in

Two authentication models exist and they fail in opposite ways. Credentials travel with the request and survive a changing address, but leak into logs and shell history. An address allow-list sends nothing secret and breaks silently the moment your address changes.

On residential networks the username usually carries more than identity. Country, city, session identifier and rotation behaviour are commonly appended to it, because the gateway hostname is fixed and the credential is the only field a client can vary per request.

Configuring it

The environment variables look like the simplest approach and behave differently in every runtime. Tested across curl, wget, Python, Go and Node, the clients disagree on the case of HTTP_PROXY, on what a leading dot in NO_PROXY matches, and on whether CIDR ranges work at all.

In this section

40 pages
Glossary Term

ASN

Autonomous System Number — the identifier of the network that owns an IP range, and a primary classification signal.

2 min read

Glossary Term

Backconnect proxy

A single gateway address that transparently routes each request through a different address in the provider's pool.

2 min read

Glossary Term

Carrier-grade NAT

Carrier-grade NAT is address translation performed inside an operator's own network, so that many subscribers share a single public address.

2 min read

Glossary Term

Certificate pinning

Certificate pinning is the practice of accepting only a specific key or certificate for a host, rather than any certificate that chains to a…

2 min read

Glossary Term

CIDR

CIDR is the notation and allocation scheme that writes a block of IP addresses as an address followed by a prefix length, such as…

2 min read

Glossary Term

Cookie jar

A cookie jar is the store a client keeps of the cookies it has received, keyed by domain and path, and replayed on later…

2 min read

Glossary Term

Crawl budget

Crawl budget is the informal name for how much of a site a crawler will fetch in a given period.

2 min read

Glossary Term

Datacenter proxy

A proxy hosted in a commercial datacentre. Fast and inexpensive, but easily identified as non-residential.

2 min read

Glossary Term

DNS leak

A DNS leak is a name lookup that travels outside the path you intended, so a resolver you did not choose learns the hostname…

2 min read

Glossary Term

Exit node

The final address in a proxy chain — the one the destination actually sees and records.

2 min read

Glossary Term

Exponential backoff

Exponential backoff is a retry policy in which the wait between attempts grows by a constant factor each time, usually doubling.

2 min read

Glossary Term

Forward proxy

A proxy that acts for the client. It sits between you and the internet and hides your address from destinations.

2 min read

Glossary Term

Geo-targeting

Choosing where your proxy exits, from country level down to city or network in some services.

2 min read

Glossary Term

Honeypot link

A honeypot link is a link placed on a page so that a person will never follow it, in order to identify clients that…

2 min read

Glossary Term

HTTP CONNECT

The HTTP method that asks a proxy to open a raw tunnel, which is how HTTPS travels through an HTTP proxy.

2 min read

Glossary Term

HTTP/2 fingerprint

An HTTP/2 fingerprint identifies a client from how it opens and drives the connection, rather than from the content of its headers.

2 min read

Glossary Term

IP whitelisting

Authorising a proxy by the address you connect from, instead of sending a username and password.

2 min read

Glossary Term

IPv6

IPv6 is the current version of the Internet Protocol, with addresses of 128 bits written as eight groups of four hexadecimal digits.

2 min read

Glossary Term

ISP proxy

A datacentre-hosted address registered to a consumer ISP. Datacentre speed with residential registration, billed per IP.

2 min read

Glossary Term

JA3 fingerprint

A JA3 fingerprint is a hash computed from selected fields of the TLS ClientHello, taken in the order the client sent them.

2 min read

Glossary Term

Mobile proxy

A proxy exiting through a mobile carrier address, shared by many real handsets via carrier-grade NAT.

2 min read

Glossary Term

Port

A port is a 16-bit number identifying one endpoint of a transport connection, so a single address can carry many independent conversations at once.

2 min read

Glossary Term

Proxy authentication

How a proxy verifies who you are, and the meaning of the 407 status code defined in RFC 9110.

2 min read

Glossary Term

Proxy server

A server that makes requests on your behalf, so the destination sees the proxy's address instead of yours.

2 min read

Glossary Term

Referer header

Referer is a request header naming the resource from which the target URI was obtained.

2 min read

Glossary Term

Residential proxy

A proxy whose exit address belongs to a consumer ISP, so the traffic appears to come from an ordinary home connection.

2 min read

Glossary Term

Retry-After

Retry-After is a response header giving either a number of seconds to wait or an HTTP-date after which to retry.

2 min read

Glossary Term

Reverse proxy

A proxy deployed in front of a server. Clients connect to it as if it were the origin, and it forwards to the real…

2 min read

Glossary Term

Rotating proxy

A proxy endpoint that changes the exit address automatically, either on every request or on a timer.

2 min read

Glossary Term

SOCKS5

A protocol that proxies any TCP or UDP traffic without inspecting it, defined in RFC 1928.

2 min read

Glossary Term

Sticky session

A setting that keeps the same exit address for a fixed period, so a multi-step flow appears to come from one user.

2 min read

Glossary Term

Subnet

A subnet is a contiguous block of addresses that share a common leading prefix and are administered as one network.

2 min read

Glossary Term

TLS interception

TLS interception is the practice of terminating a client's encrypted connection at a middlebox that reads the plaintext and re-encrypts onward.

2 min read

Glossary Term

Transparent proxy

In everyday use, a transparent proxy is one that intercepts traffic without any configuration on the client, so the client does not know it…

2 min read

Glossary Term

Via header

Via is a standard HTTP header in which each intermediary records the protocol version it received the message on, together with a name for…

2 min read

Glossary Term

WebRTC leak

A WebRTC leak is the disclosure of a network address by the browser's real-time communication stack.

2 min read

Guide

What is a proxy server?

How a proxy actually routes your request, what the destination can still see, and where each of the four types fits.

8 min read

Glossary Term

X-Forwarded-For

X-Forwarded-For is a de facto request header in which a proxy records the address of the client it received the request from.

2 min read