On this page 13 sections
- requests
- SOCKS5 with requests
- httpx
- Async
- aiohttp
- The same faults in all three libraries
- Reuse the Session: measured
- Retries that honour Retry-After
- The limit no proxy setting fixes
- Validate the response, not the status
- Rotating through your own list
- Keeping one address for a whole flow
- Check the exit before you rely on it
- A small fetcher that puts it together
- Choosing between the three libraries
To use a proxy with Python requests, pass a proxies dict with both an "http" and an "https" key, and set it on a Session so connections are reused:
import requests
PROXY = "http://user:pass@gw.example.net:8000"
with requests.Session() as s:
s.proxies.update({"http": PROXY, "https": PROXY})
r = s.get("https://example.com/", timeout=(5, 30))
print(r.status_code)
Both keys matter. The key is the scheme of the site you are requesting; the value is how to reach the proxy. When we gave requests 2.34.2 only an "http" key, it sent our HTTPS request straight to the site, without the proxy and without an error. httpx takes a single proxy= argument, and aiohttp takes proxy= on the session or on each request. The rest of this page is the setup for each, and the exact exception each library raised when we broke the proxy on purpose.
requests
The Session example above is the whole setup for an HTTP proxy. Three details decide whether it behaves:
- The
"https"key still points at anhttp://proxy URL. That is correct: requests connects to the proxy over plain HTTP and opens a CONNECT tunnel for the HTTPS site inside it. - The timeout is a tuple of connect and read. A bare
timeout=30applies 30 seconds to each phase, which is far too long to wait for a dead proxy to accept a connection. - requests reads
HTTP_PROXY,HTTPS_PROXYandNO_PROXYfrom the environment unless you tell it not to (s.trust_env = False). WithHTTPS_PROXYpointing at a closed port and no proxy in the code, requests used the variable and failed withProxyError, as it should. More on which tools honour which variables is in HTTP_PROXY, HTTPS_PROXY and NO_PROXY explained.
Passwords with special characters are safer percent-encoded. requests 2.34.2 accepted p@ss:word unencoded in the proxy URL, but curl rejected the same string, and the next tool in your pipeline may too:
from urllib.parse import quote
PROXY = f"http://user2:{quote('p@ss:word', safe='')}@gw.example.net:8000"
# http://user2:p%40ss%3Aword@gw.example.net:8000
SOCKS5 with requests
pip install "requests[socks]"
PROXY = "socks5h://user:pass@gw.example.net:1080"
r = requests.get("https://example.com/", proxies={"http": PROXY, "https": PROXY}, timeout=(5, 30))
Without the extra, requests fails before it opens a connection, with requests.exceptions.InvalidSchema: Missing dependencies for SOCKS support.
Use socks5h, not socks5. Our SOCKS server logged what it received for the same request:
requests socks5:// CONNECT atyp=IPv4 target=104.20.23.154:443
requests socks5h:// CONNECT atyp=DOMAIN target=example.com:443
With socks5://, requests resolved the hostname on your machine and sent the proxy a bare address. That leaks every hostname to your local resolver, and on a site that answers DNS by region it can send you to the wrong regional server even though the proxy exits in the right country.
httpx
import httpx
PROXY = "http://user:pass@gw.example.net:8000"
with httpx.Client(proxy=PROXY, timeout=httpx.Timeout(30.0, connect=5.0), follow_redirects=True) as client:
r = client.get("https://example.com/")
print(r.status_code)
The argument is proxy=, singular. httpx added it in 0.26.0 and removed the old proxies= dict in 0.28.0. Code copied from older tutorials now fails at construction:
TypeError: Client.__init__() got an unexpected keyword argument 'proxies'
For different proxies per scheme or per host, httpx uses mounts= with one transport each, not a dict.
SOCKS is not built in. httpx needs the socksio package, and says so:
pip install "httpx[socks]"
ImportError: Using SOCKS proxy, but the 'socksio' package is not installed. Make sure to install httpx using `pip install httpx[socks]`.
One difference from requests showed up in the SOCKS log. httpx 0.28.1 sent the hostname to the proxy for both socks5:// and socks5h://, so it never resolved locally. Writing socks5h is still the clearer habit, because the same URL in requests or curl behaves differently.
Async
import asyncio
import httpx
PROXY = "http://user:pass@gw.example.net:8000"
async def fetch_all(urls, limit=10):
sem = asyncio.Semaphore(limit) # cap concurrency yourself
async with httpx.AsyncClient(proxy=PROXY, timeout=httpx.Timeout(30.0, connect=5.0)) as client:
async def one(u):
async with sem:
try:
r = await client.get(u)
return u, r.status_code, len(r.content)
except httpx.HTTPError as e:
return u, None, repr(e)
return await asyncio.gather(*(one(u) for u in urls))
print(asyncio.run(fetch_all(["https://example.com/"] * 5)))
aiohttp
import asyncio
import aiohttp
PROXY = "http://user:pass@gw.example.net:8000"
async def main():
timeout = aiohttp.ClientTimeout(total=30, connect=5)
async with aiohttp.ClientSession(proxy=PROXY, timeout=timeout) as session:
async with session.get("https://example.com/") as r:
print(r.status)
asyncio.run(main())
Older advice, including an earlier version of this page, says aiohttp only takes a proxy per request. That stopped being true in aiohttp 3.11.0, which added proxy to ClientSession; both forms worked in 3.14.5. Credentials can sit in the proxy URL, as above. The separate proxy_auth=aiohttp.BasicAuth(...) form still works but is deprecated since 3.14.0, and 3.14.5 printed two warnings for it:
DeprecationWarning: BasicAuth is deprecated and will be removed in aiohttp 4.0; use aiohttp.encode_basic_auth() with headers={'Authorization': ...} instead
DeprecationWarning: The 'proxy_auth' parameter is deprecated and will be removed in v4; pass proxy_headers={'Proxy-Authorization': aiohttp.encode_basic_auth(login, password)} instead
Two more aiohttp behaviours differ from the other libraries. It ignores proxy environment variables unless you pass trust_env=True: with HTTPS_PROXY pointing at a closed port, a default session went straight to the site and returned 200. And it does not speak SOCKS. Given a socks5:// URL, aiohttp 3.14.5 raised no configuration error. It sent an HTTP CONNECT to the SOCKS port (our server logged a first byte of 0x43, the letter C) and then failed with ServerDisconnectedError: Server disconnected, which reads like a network fault.
For SOCKS, use the aiohttp-socks package:
import asyncio
import aiohttp
from aiohttp_socks import ProxyConnector
async def main():
connector = ProxyConnector.from_url("socks5://user:pass@gw.example.net:1080")
async with aiohttp.ClientSession(connector=connector) as session:
async with session.get("https://example.com/") as r:
print(r.status)
asyncio.run(main())
aiohttp-socks 0.12.0 sent the hostname to the proxy by default. Passing rdns=False made it resolve locally and send an IP, like socks5:// in requests. It rejects the socks5h spelling outright with ValueError: Invalid scheme component: socks5h.
The same faults in all three libraries
Each fault was triggered against the local proxies with an HTTPS target. A dash means the case does not apply to that library.
| Fault | requests 2.34.2 | httpx 0.28.1 | aiohttp 3.14.5 |
|---|---|---|---|
| Wrong proxy password | ProxyError: ... (Caused by ProxyError('Unable to connect to proxy', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))) |
httpx.ProxyError: 407 Proxy Authentication Required |
ClientHttpProxyError: 407, message='Proxy Authentication Required', url='http://user:wrong@127.0.0.1:3128' |
HTTPS_PROXY set to a closed port, no proxy in code |
Used it: ProxyError: ... Failed to establish a new connection: [Errno 61] Connection refused |
Used it: httpx.ConnectError: [Errno 61] Connection refused |
Ignored it: 200, direct. With trust_env=True: ClientProxyConnectionError: Cannot connect to host 127.0.0.1:3999 |
| SOCKS URL, optional package not installed | InvalidSchema: Missing dependencies for SOCKS support. |
ImportError: Using SOCKS proxy, but the 'socksio' package is not installed. |
ServerDisconnectedError: Server disconnected |
Hostname sent to a SOCKS5 proxy for socks5:// |
No, resolved locally | Yes | Yes, with aiohttp-socks defaults |
Only an "http" key in the proxy mapping |
HTTPS request went direct, no error | — | — |
| Old-style argument | — | TypeError on proxies= |
DeprecationWarning on proxy_auth= |
Two things in that table cost real debugging time. The httpx error for a dead proxy, ConnectError: [Errno 61] Connection refused, does not mention the proxy at all, so it looks like the site is down. And the aiohttp exception text includes the full proxy URL with the password. If you log exceptions as strings, you are logging your proxy credentials.
r.status_code == 407 never sees it, because there is no response object. Only a plain-HTTP target returns the proxy’s 407 as an ordinary response: requests gave us status_code == 407 for http://example.com/.Reuse the Session: measured
Calling requests.get() in a loop builds a fresh connection pool every time, so every call opens a new tunnel through the proxy and a new TLS handshake with the site. We counted tunnels in Squid’s access log for 20 requests to https://example.com/ each way, three rounds per run, two runs:
| Round | requests.get() x20 (s) | Session x20 (s) |
|---|---|---|
| Run 1, round 1 | 0.76 | 0.33 |
| Run 1, round 2 | 0.83 | 0.33 |
| Run 1, round 3 | 0.70 | 0.30 |
| Run 2, round 1 | 1.04 | 0.34 |
| Run 2, round 2 | 0.93 | 0.34 |
| Run 2, round 3 | 0.91 | 0.32 |
Every round, requests.get() opened 20 CONNECT tunnels and the Session opened one. The Session took 0.30 to 0.34 seconds against 0.70 to 1.04 for the loop. The proxy was on the same machine, so the gap here is almost entirely TLS handshakes with the site. Through a remote proxy each extra tunnel also adds a round trip to the gateway, and on a per-gigabyte plan each handshake is traffic you pay for. The same applies to httpx: create one Client or AsyncClient and keep it.
Retries that honour Retry-After
Many retry snippets, including an earlier version of this page, read the header with float(r.headers["Retry-After"]). That crashes on one of the two forms the standard allows. RFC 9110 permits either a number of seconds or an HTTP date, and a date gives you this:
ValueError: could not convert string to float: 'Wed, 21 Oct 2026 23:59:00 GMT'
A parser that handles both forms, and the retry loop around it:
import random
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
import httpx
def retry_after_seconds(value, default):
"""Retry-After is delay-seconds or an HTTP-date (RFC 9110 section 10.2.3)."""
if value is None:
return default
value = value.strip()
if value.isdigit():
return int(value)
try:
when = parsedate_to_datetime(value)
except (TypeError, ValueError):
return default
return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
def get_with_retry(client, url, attempts=4, max_wait=120):
for i in range(attempts):
backoff = (2 ** i) * (0.5 + random.random() * 0.5) # jitter: 50-100%
try:
r = client.get(url)
except httpx.ProxyError:
raise # the proxy refused you; retrying cannot help
except httpx.TransportError:
time.sleep(backoff)
continue
if r.status_code == 429:
wait = retry_after_seconds(r.headers.get("Retry-After"), backoff)
if wait > max_wait:
return r # the server asked for a long pause; stop
time.sleep(wait)
continue
if r.status_code >= 500:
time.sleep(backoff)
continue
return r
return None
Against a test server answering 429, the parser returned 2 for Retry-After: 2 and more than 15,000 seconds for a date a little over four hours away. That second value is why max_wait exists: a date can legitimately tell you to come back tomorrow, and a worker that sleeps on it looks like a hang. Jitter is the other load-bearing line. Without it, parallel workers that failed together retry together and rebuild the burst that caused the rate limit. Which status codes deserve a retry at all is covered in 407 vs 403 vs 429.
The limit no proxy setting fixes
requests, httpx and aiohttp each send their own TLS fingerprint, and a site sees it before any header you set. Our TLS ClientHello study captured python-requests 2.32.3 offering 38 cipher suites and 4 extensions where Chromium offered 15 and 15, and found curl and Go keeping the same extension order across three connections each while Chromium varied across 12 captured connections. The other five libraries were sampled once, so their rows are not a repeatability result. The default request headers study recorded python-requests 2.32.3 and httpx 0.27.2 sending 5 headers each, where a Chromium navigation sent 11. If a site fingerprints TLS, changing the User-Agent will not help; you need a client that reproduces a browser handshake, or a real headless browser.
Validate the response, not the status
r = client.get(url)
if r.status_code == 200 and "expected-marker" in r.text:
handle(r)
else:
record_failure(url, r.status_code, len(r.content))
An anti-bot system can answer with a 200 and a challenge page or an empty result. A pipeline that checks only the status reports a high success rate while collecting nothing.
Rotating through your own list
With a backconnect gateway the provider rotates for you. With a list of individual proxies, rotate in order and retire the ones that fail. The retirement rule has to catch exceptions, because that is how a 407 arrives:
import itertools
import threading
import httpx
class ProxyRing:
def __init__(self, proxies):
self._live = list(proxies)
self._cycle = itertools.cycle(self._live)
self._lock = threading.Lock()
def next(self):
with self._lock:
return next(self._cycle)
def drop(self, proxy):
"""Retire a proxy that keeps failing, and rebuild the cycle."""
with self._lock:
if proxy in self._live and len(self._live) > 1:
self._live.remove(proxy)
self._cycle = itertools.cycle(self._live)
ring = ProxyRing([
"http://user:pass@a.example.net:8000",
"http://user:pass@b.example.net:8000",
])
def fetch(url):
proxy = ring.next()
try:
with httpx.Client(proxy=proxy, timeout=20) as c:
return c.get(url)
except (httpx.ProxyError, httpx.ConnectError, httpx.ConnectTimeout):
ring.drop(proxy) # 407, dead port, or no answer from the proxy
return None
A 403 from the site is not a reason to drop a proxy on its own. The site may be refusing your client rather than that address. Count 403s per proxy and retire one only when it fails where the others succeed.
Keeping one address for a whole flow
Providers that support sticky sessions typically hold the exit while the username carries the same session identifier. The syntax varies by provider; -session- below is an example:
import uuid
import httpx
def flow_client(base_user, password, host, port):
session = f"s-{uuid.uuid4().hex[:8]}"
proxy = f"http://{base_user}-session-{session}:{password}@{host}:{port}"
return httpx.Client(proxy=proxy, timeout=30, follow_redirects=True)
with flow_client("user", "pass", "gw.example.net", 8000) as c:
c.get("https://example.com/step-one")
r = c.get("https://example.com/step-two")
The client keeps cookies, and the session identifier keeps the address. A flow needs both; see rotating vs sticky sessions.
Check the exit before you rely on it
import httpx
with httpx.Client(proxy=PROXY, timeout=20) as c:
me = c.get("http://ip-api.com/json/?fields=status,query,country,city,isp,as").json()
assert me.get("status") == "success", me
print(me["query"], me["country"], me.get("isp"))
Log this at the start of a job. When the output looks wrong three hours later, knowing which country you were actually exiting from turns a guess into a fact.
A small fetcher that puts it together
This reuses one client, caps concurrency below the provider’s limit, honours Retry-After with a ceiling, stops on a proxy refusal, and checks content rather than trusting the status:
import asyncio
import random
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
import httpx
PROXY = "http://user:pass@gw.example.net:8000"
BLOCK_MARKERS = ("captcha", "unusual traffic", "access denied")
MAX_WAIT = 120 # seconds; a longer Retry-After ends the attempt
def retry_after_seconds(value, default):
if value is None:
return default
value = value.strip()
if value.isdigit():
return int(value)
try:
when = parsedate_to_datetime(value)
except (TypeError, ValueError):
return default
return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
def usable(r, min_bytes=500):
if r.status_code != 200 or len(r.content) < min_bytes:
return False
low = r.text.lower()
return not any(m in low for m in BLOCK_MARKERS)
async def fetch(client, url, attempts=4):
for i in range(attempts):
backoff = (2 ** i) * (0.5 + random.random() * 0.5)
try:
r = await client.get(url)
except httpx.ProxyError:
raise # credentials or proxy policy: stop the job
except httpx.TransportError:
await asyncio.sleep(backoff)
continue
if r.status_code == 429:
wait = retry_after_seconds(r.headers.get("Retry-After"), backoff)
if wait > MAX_WAIT:
return None
await asyncio.sleep(wait)
continue
if usable(r):
return r
if r.status_code == 403:
return None # refused; retrying the same request will not change that
await asyncio.sleep(backoff)
return None
async def main(urls, workers=8):
limits = httpx.Limits(max_connections=workers, max_keepalive_connections=workers)
async with httpx.AsyncClient(proxy=PROXY, timeout=httpx.Timeout(30, connect=5),
limits=limits, follow_redirects=True) as client:
sem = asyncio.Semaphore(workers)
async def one(u):
async with sem:
r = await fetch(client, u)
await asyncio.sleep(0.1) # pace, do not sprint
return u, (r.status_code if r else None)
return await asyncio.gather(*(one(u) for u in urls))
if __name__ == "__main__":
print(asyncio.run(main(["https://example.com/"] * 3)))
Through the test proxy it printed [('https://example.com/', 200), ('https://example.com/', 200), ('https://example.com/', 200)]. With a wrong password it raised httpx.ProxyError: 407 Proxy Authentication Required on the first request instead of retrying four times. min_bytes=500 suits example.com, whose page is 577 bytes; set it from a real page on your target. Log four fields per request and most incidents explain themselves: status, response size, exit address and elapsed time.
Choosing between the three libraries
| requests 2.34.2 | httpx 0.28.1 | aiohttp 3.14.5 | |
|---|---|---|---|
| Async | No | Yes, alongside sync | Async only |
| HTTP/2 | No | Yes, with the http2 extra |
No |
| Where the proxy goes | proxies dict, per request or on a Session |
proxy= on the client; mounts= for per-scheme rules |
proxy= on the session or per request |
| SOCKS | requests[socks] (PySocks) |
httpx[socks] (socksio) |
Separate aiohttp-socks package |
| Reads proxy environment variables | Yes, by default | Yes, by default | Only with trust_env=True |
| How a 407 on HTTPS arrives | ProxyError |
httpx.ProxyError |
ClientHttpProxyError, e.status == 407 |
httpx is a reasonable default if you want sync and async code with one API. requests is still the simplest choice for a synchronous script. aiohttp suits code that is already built on it. None of them changes your TLS fingerprint, removes the need to validate content, or makes an uncapped retry loop safe. The same setup for the command line is in how to use curl with a proxy, and for JavaScript in proxies in Node.js.
Frequently asked questions
How do I set a proxy in Python requests?
Why does requests ignore my proxy for HTTPS sites?
Does httpx still accept proxies=?
Does requests support SOCKS5 proxies?
Why does my code never see a 407 status?
Does aiohttp use the HTTPS_PROXY environment variable?
Sources
- requests documentation: proxies and SOCKS
- httpx documentation: proxies
- httpx changelog: proxy= added in 0.26.0, proxies= removed in 0.28.0
- aiohttp client reference: ClientSession proxy and trust_env
- aiohttp changelog: session-level proxy in 3.11.0, BasicAuth and proxy_auth deprecated in 3.14.0
- aiohttp-socks on PyPI
- RFC 9110 section 10.2.3: Retry-After
- RFC 1928: SOCKS Protocol Version 5