Skip to content
Web scraping

Python requests proxy: requests, httpx and aiohttp

Tested proxy setup for requests 2.34.2, httpx 0.28.1 and aiohttp 3.14.5, with the exact exception each raised when the proxy failed.

On this page 13 sections
  1. requests
  2. SOCKS5 with requests
  3. httpx
  4. Async
  5. aiohttp
  6. The same faults in all three libraries
  7. Reuse the Session: measured
  8. Retries that honour Retry-After
  9. The limit no proxy setting fixes
  10. Validate the response, not the status
  11. Rotating through your own list
  12. Keeping one address for a whole flow
  13. Check the exit before you rely on it
  14. A small fetcher that puts it together
  15. Choosing between the three libraries

To use a proxy with Python requests, pass a proxies dict with both an "http" and an "https" key, and set it on a Session so connections are reused:

import requests

PROXY = "http://user:pass@gw.example.net:8000"

with requests.Session() as s:
    s.proxies.update({"http": PROXY, "https": PROXY})
    r = s.get("https://example.com/", timeout=(5, 30))
    print(r.status_code)

Both keys matter. The key is the scheme of the site you are requesting; the value is how to reach the proxy. When we gave requests 2.34.2 only an "http" key, it sent our HTTPS request straight to the site, without the proxy and without an error. httpx takes a single proxy= argument, and aiohttp takes proxy= on the session or on each request. The rest of this page is the setup for each, and the exact exception each library raised when we broke the proxy on purpose.

Tested with Python 3.12.13, requests 2.34.2 (urllib3 2.8.0), httpx 0.28.1, aiohttp 3.14.5, PySocks 1.7.1, socksio 1.0.0 and aiohttp-socks 0.12.0 on macOS 26.6.2, 21 October 2026. These were the current PyPI releases that day. Proxies: Squid 6.13 with Basic authentication on 127.0.0.1:3128, and a SOCKS5 server we wrote that logs what each client sends. Every case ran twice with the same result. Every code block on this page ran against those proxies, with only the host, port and credentials changed.

requests

The Session example above is the whole setup for an HTTP proxy. Three details decide whether it behaves:

  1. The "https" key still points at an http:// proxy URL. That is correct: requests connects to the proxy over plain HTTP and opens a CONNECT tunnel for the HTTPS site inside it.
  2. The timeout is a tuple of connect and read. A bare timeout=30 applies 30 seconds to each phase, which is far too long to wait for a dead proxy to accept a connection.
  3. requests reads HTTP_PROXY, HTTPS_PROXY and NO_PROXY from the environment unless you tell it not to (s.trust_env = False). With HTTPS_PROXY pointing at a closed port and no proxy in the code, requests used the variable and failed with ProxyError, as it should. More on which tools honour which variables is in HTTP_PROXY, HTTPS_PROXY and NO_PROXY explained.

Passwords with special characters are safer percent-encoded. requests 2.34.2 accepted p@ss:word unencoded in the proxy URL, but curl rejected the same string, and the next tool in your pipeline may too:

from urllib.parse import quote

PROXY = f"http://user2:{quote('p@ss:word', safe='')}@gw.example.net:8000"
# http://user2:p%40ss%3Aword@gw.example.net:8000

SOCKS5 with requests

pip install "requests[socks]"
PROXY = "socks5h://user:pass@gw.example.net:1080"
r = requests.get("https://example.com/", proxies={"http": PROXY, "https": PROXY}, timeout=(5, 30))

Without the extra, requests fails before it opens a connection, with requests.exceptions.InvalidSchema: Missing dependencies for SOCKS support.

Use socks5h, not socks5. Our SOCKS server logged what it received for the same request:

requests socks5://   CONNECT atyp=IPv4 target=104.20.23.154:443
requests socks5h://  CONNECT atyp=DOMAIN target=example.com:443

With socks5://, requests resolved the hostname on your machine and sent the proxy a bare address. That leaks every hostname to your local resolver, and on a site that answers DNS by region it can send you to the wrong regional server even though the proxy exits in the right country.

httpx

import httpx

PROXY = "http://user:pass@gw.example.net:8000"

with httpx.Client(proxy=PROXY, timeout=httpx.Timeout(30.0, connect=5.0), follow_redirects=True) as client:
    r = client.get("https://example.com/")
    print(r.status_code)

The argument is proxy=, singular. httpx added it in 0.26.0 and removed the old proxies= dict in 0.28.0. Code copied from older tutorials now fails at construction:

TypeError: Client.__init__() got an unexpected keyword argument 'proxies'

For different proxies per scheme or per host, httpx uses mounts= with one transport each, not a dict.

SOCKS is not built in. httpx needs the socksio package, and says so:

pip install "httpx[socks]"
ImportError: Using SOCKS proxy, but the 'socksio' package is not installed. Make sure to install httpx using `pip install httpx[socks]`.

One difference from requests showed up in the SOCKS log. httpx 0.28.1 sent the hostname to the proxy for both socks5:// and socks5h://, so it never resolved locally. Writing socks5h is still the clearer habit, because the same URL in requests or curl behaves differently.

Async

import asyncio
import httpx

PROXY = "http://user:pass@gw.example.net:8000"

async def fetch_all(urls, limit=10):
    sem = asyncio.Semaphore(limit)          # cap concurrency yourself
    async with httpx.AsyncClient(proxy=PROXY, timeout=httpx.Timeout(30.0, connect=5.0)) as client:
        async def one(u):
            async with sem:
                try:
                    r = await client.get(u)
                    return u, r.status_code, len(r.content)
                except httpx.HTTPError as e:
                    return u, None, repr(e)
        return await asyncio.gather(*(one(u) for u in urls))

print(asyncio.run(fetch_all(["https://example.com/"] * 5)))

aiohttp

import asyncio
import aiohttp

PROXY = "http://user:pass@gw.example.net:8000"

async def main():
    timeout = aiohttp.ClientTimeout(total=30, connect=5)
    async with aiohttp.ClientSession(proxy=PROXY, timeout=timeout) as session:
        async with session.get("https://example.com/") as r:
            print(r.status)

asyncio.run(main())

Older advice, including an earlier version of this page, says aiohttp only takes a proxy per request. That stopped being true in aiohttp 3.11.0, which added proxy to ClientSession; both forms worked in 3.14.5. Credentials can sit in the proxy URL, as above. The separate proxy_auth=aiohttp.BasicAuth(...) form still works but is deprecated since 3.14.0, and 3.14.5 printed two warnings for it:

DeprecationWarning: BasicAuth is deprecated and will be removed in aiohttp 4.0; use aiohttp.encode_basic_auth() with headers={'Authorization': ...} instead
DeprecationWarning: The 'proxy_auth' parameter is deprecated and will be removed in v4; pass proxy_headers={'Proxy-Authorization': aiohttp.encode_basic_auth(login, password)} instead

Two more aiohttp behaviours differ from the other libraries. It ignores proxy environment variables unless you pass trust_env=True: with HTTPS_PROXY pointing at a closed port, a default session went straight to the site and returned 200. And it does not speak SOCKS. Given a socks5:// URL, aiohttp 3.14.5 raised no configuration error. It sent an HTTP CONNECT to the SOCKS port (our server logged a first byte of 0x43, the letter C) and then failed with ServerDisconnectedError: Server disconnected, which reads like a network fault.

For SOCKS, use the aiohttp-socks package:

import asyncio
import aiohttp
from aiohttp_socks import ProxyConnector

async def main():
    connector = ProxyConnector.from_url("socks5://user:pass@gw.example.net:1080")
    async with aiohttp.ClientSession(connector=connector) as session:
        async with session.get("https://example.com/") as r:
            print(r.status)

asyncio.run(main())

aiohttp-socks 0.12.0 sent the hostname to the proxy by default. Passing rdns=False made it resolve locally and send an IP, like socks5:// in requests. It rejects the socks5h spelling outright with ValueError: Invalid scheme component: socks5h.

The same faults in all three libraries

Each fault was triggered against the local proxies with an HTTPS target. A dash means the case does not apply to that library.

Fault requests 2.34.2 httpx 0.28.1 aiohttp 3.14.5
Wrong proxy password ProxyError: ... (Caused by ProxyError('Unable to connect to proxy', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))) httpx.ProxyError: 407 Proxy Authentication Required ClientHttpProxyError: 407, message='Proxy Authentication Required', url='http://user:wrong@127.0.0.1:3128'
HTTPS_PROXY set to a closed port, no proxy in code Used it: ProxyError: ... Failed to establish a new connection: [Errno 61] Connection refused Used it: httpx.ConnectError: [Errno 61] Connection refused Ignored it: 200, direct. With trust_env=True: ClientProxyConnectionError: Cannot connect to host 127.0.0.1:3999
SOCKS URL, optional package not installed InvalidSchema: Missing dependencies for SOCKS support. ImportError: Using SOCKS proxy, but the 'socksio' package is not installed. ServerDisconnectedError: Server disconnected
Hostname sent to a SOCKS5 proxy for socks5:// No, resolved locally Yes Yes, with aiohttp-socks defaults
Only an "http" key in the proxy mapping HTTPS request went direct, no error — —
Old-style argument — TypeError on proxies= DeprecationWarning on proxy_auth=

Two things in that table cost real debugging time. The httpx error for a dead proxy, ConnectError: [Errno 61] Connection refused, does not mention the proxy at all, so it looks like the site is down. And the aiohttp exception text includes the full proxy URL with the password. If you log exceptions as strings, you are logging your proxy credentials.

A 407 on an HTTPS request is an exception in all three libraries, not a response. Code that checks r.status_code == 407 never sees it, because there is no response object. Only a plain-HTTP target returns the proxy’s 407 as an ordinary response: requests gave us status_code == 407 for http://example.com/.

Reuse the Session: measured

Calling requests.get() in a loop builds a fresh connection pool every time, so every call opens a new tunnel through the proxy and a new TLS handshake with the site. We counted tunnels in Squid’s access log for 20 requests to https://example.com/ each way, three rounds per run, two runs:

Wall time for 20 HTTPS requests through a local Squid, requests 2.34.2, six rounds, 21 October 2026 Drawn from the table below.
Wall time for 20 HTTPS requests through a local Squid, requests 2.34.2, six rounds, 21 October 2026. Bar chart of requests.get() x20 (s), Session x20 (s) by Round.6 rows, drawn from the table that follows this chart.00.250.50.7511.25 sRun 1, round 1 · requests.get() x20 (s): 0.760.76Run 1, round 1 · Session x20 (s): 0.330.33Run 1, round 1Run 1, round 2 · requests.get() x20 (s): 0.830.83Run 1, round 2 · Session x20 (s): 0.330.33Run 1, round 2Run 1, round 3 · requests.get() x20 (s): 0.700.70Run 1, round 3 · Session x20 (s): 0.300.30Run 1, round 3Run 2, round 1 · requests.get() x20 (s): 1.041.04Run 2, round 1 · Session x20 (s): 0.340.34Run 2, round 1Run 2, round 2 · requests.get() x20 (s): 0.930.93Run 2, round 2 · Session x20 (s): 0.340.34Run 2, round 2Run 2, round 3 · requests.get() x20 (s): 0.910.91Run 2, round 3 · Session x20 (s): 0.320.32Run 2, round 3Wall time for 20 HTTPS requests through a local Squid, requests 2.34.2, six rounds, 21 October 2026. Bar chart of requests.get() x20 (s), Session x20 (s) by Round.6 rows, drawn from the table that follows this chart.Run 1, round 1Run 1, round 1 · requests.get() x20 (s): 0.760.76Run 1, round 1 · Session x20 (s): 0.330.33Run 1, round 2Run 1, round 2 · requests.get() x20 (s): 0.830.83Run 1, round 2 · Session x20 (s): 0.330.33Run 1, round 3Run 1, round 3 · requests.get() x20 (s): 0.700.70Run 1, round 3 · Session x20 (s): 0.300.30Run 2, round 1Run 2, round 1 · requests.get() x20 (s): 1.041.04Run 2, round 1 · Session x20 (s): 0.340.34Run 2, round 2Run 2, round 2 · requests.get() x20 (s): 0.930.93Run 2, round 2 · Session x20 (s): 0.340.34Run 2, round 3Run 2, round 3 · requests.get() x20 (s): 0.910.91Run 2, round 3 · Session x20 (s): 0.320.3200.511.5 s
Round requests.get() x20 (s) Session x20 (s)
Run 1, round 1 0.76 0.33
Run 1, round 2 0.83 0.33
Run 1, round 3 0.70 0.30
Run 2, round 1 1.04 0.34
Run 2, round 2 0.93 0.34
Run 2, round 3 0.91 0.32

Every round, requests.get() opened 20 CONNECT tunnels and the Session opened one. The Session took 0.30 to 0.34 seconds against 0.70 to 1.04 for the loop. The proxy was on the same machine, so the gap here is almost entirely TLS handshakes with the site. Through a remote proxy each extra tunnel also adds a round trip to the gateway, and on a per-gigabyte plan each handshake is traffic you pay for. The same applies to httpx: create one Client or AsyncClient and keep it.

Retries that honour Retry-After

Many retry snippets, including an earlier version of this page, read the header with float(r.headers["Retry-After"]). That crashes on one of the two forms the standard allows. RFC 9110 permits either a number of seconds or an HTTP date, and a date gives you this:

ValueError: could not convert string to float: 'Wed, 21 Oct 2026 23:59:00 GMT'

A parser that handles both forms, and the retry loop around it:

import random
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

import httpx

def retry_after_seconds(value, default):
    """Retry-After is delay-seconds or an HTTP-date (RFC 9110 section 10.2.3)."""
    if value is None:
        return default
    value = value.strip()
    if value.isdigit():
        return int(value)
    try:
        when = parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return default
    return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())

def get_with_retry(client, url, attempts=4, max_wait=120):
    for i in range(attempts):
        backoff = (2 ** i) * (0.5 + random.random() * 0.5)   # jitter: 50-100%
        try:
            r = client.get(url)
        except httpx.ProxyError:
            raise                      # the proxy refused you; retrying cannot help
        except httpx.TransportError:
            time.sleep(backoff)
            continue
        if r.status_code == 429:
            wait = retry_after_seconds(r.headers.get("Retry-After"), backoff)
            if wait > max_wait:
                return r               # the server asked for a long pause; stop
            time.sleep(wait)
            continue
        if r.status_code >= 500:
            time.sleep(backoff)
            continue
        return r
    return None

Against a test server answering 429, the parser returned 2 for Retry-After: 2 and more than 15,000 seconds for a date a little over four hours away. That second value is why max_wait exists: a date can legitimately tell you to come back tomorrow, and a worker that sleeps on it looks like a hang. Jitter is the other load-bearing line. Without it, parallel workers that failed together retry together and rebuild the burst that caused the rate limit. Which status codes deserve a retry at all is covered in 407 vs 403 vs 429.

The limit no proxy setting fixes

requests, httpx and aiohttp each send their own TLS fingerprint, and a site sees it before any header you set. Our TLS ClientHello study captured python-requests 2.32.3 offering 38 cipher suites and 4 extensions where Chromium offered 15 and 15, and found curl and Go keeping the same extension order across three connections each while Chromium varied across 12 captured connections. The other five libraries were sampled once, so their rows are not a repeatability result. The default request headers study recorded python-requests 2.32.3 and httpx 0.27.2 sending 5 headers each, where a Chromium navigation sent 11. If a site fingerprints TLS, changing the User-Agent will not help; you need a client that reproduces a browser handshake, or a real headless browser.

Validate the response, not the status

r = client.get(url)
if r.status_code == 200 and "expected-marker" in r.text:
    handle(r)
else:
    record_failure(url, r.status_code, len(r.content))

An anti-bot system can answer with a 200 and a challenge page or an empty result. A pipeline that checks only the status reports a high success rate while collecting nothing.

Rotating through your own list

With a backconnect gateway the provider rotates for you. With a list of individual proxies, rotate in order and retire the ones that fail. The retirement rule has to catch exceptions, because that is how a 407 arrives:

import itertools
import threading

import httpx

class ProxyRing:
    def __init__(self, proxies):
        self._live = list(proxies)
        self._cycle = itertools.cycle(self._live)
        self._lock = threading.Lock()

    def next(self):
        with self._lock:
            return next(self._cycle)

    def drop(self, proxy):
        """Retire a proxy that keeps failing, and rebuild the cycle."""
        with self._lock:
            if proxy in self._live and len(self._live) > 1:
                self._live.remove(proxy)
                self._cycle = itertools.cycle(self._live)

ring = ProxyRing([
    "http://user:pass@a.example.net:8000",
    "http://user:pass@b.example.net:8000",
])

def fetch(url):
    proxy = ring.next()
    try:
        with httpx.Client(proxy=proxy, timeout=20) as c:
            return c.get(url)
    except (httpx.ProxyError, httpx.ConnectError, httpx.ConnectTimeout):
        ring.drop(proxy)          # 407, dead port, or no answer from the proxy
        return None

A 403 from the site is not a reason to drop a proxy on its own. The site may be refusing your client rather than that address. Count 403s per proxy and retire one only when it fails where the others succeed.

Keeping one address for a whole flow

Providers that support sticky sessions typically hold the exit while the username carries the same session identifier. The syntax varies by provider; -session- below is an example:

import uuid
import httpx

def flow_client(base_user, password, host, port):
    session = f"s-{uuid.uuid4().hex[:8]}"
    proxy = f"http://{base_user}-session-{session}:{password}@{host}:{port}"
    return httpx.Client(proxy=proxy, timeout=30, follow_redirects=True)

with flow_client("user", "pass", "gw.example.net", 8000) as c:
    c.get("https://example.com/step-one")
    r = c.get("https://example.com/step-two")

The client keeps cookies, and the session identifier keeps the address. A flow needs both; see rotating vs sticky sessions.

Check the exit before you rely on it

import httpx

with httpx.Client(proxy=PROXY, timeout=20) as c:
    me = c.get("http://ip-api.com/json/?fields=status,query,country,city,isp,as").json()
    assert me.get("status") == "success", me
    print(me["query"], me["country"], me.get("isp"))

Log this at the start of a job. When the output looks wrong three hours later, knowing which country you were actually exiting from turns a guess into a fact.

A small fetcher that puts it together

This reuses one client, caps concurrency below the provider’s limit, honours Retry-After with a ceiling, stops on a proxy refusal, and checks content rather than trusting the status:

import asyncio
import random
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

import httpx

PROXY = "http://user:pass@gw.example.net:8000"
BLOCK_MARKERS = ("captcha", "unusual traffic", "access denied")
MAX_WAIT = 120          # seconds; a longer Retry-After ends the attempt

def retry_after_seconds(value, default):
    if value is None:
        return default
    value = value.strip()
    if value.isdigit():
        return int(value)
    try:
        when = parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return default
    return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())

def usable(r, min_bytes=500):
    if r.status_code != 200 or len(r.content) < min_bytes:
        return False
    low = r.text.lower()
    return not any(m in low for m in BLOCK_MARKERS)

async def fetch(client, url, attempts=4):
    for i in range(attempts):
        backoff = (2 ** i) * (0.5 + random.random() * 0.5)
        try:
            r = await client.get(url)
        except httpx.ProxyError:
            raise                          # credentials or proxy policy: stop the job
        except httpx.TransportError:
            await asyncio.sleep(backoff)
            continue
        if r.status_code == 429:
            wait = retry_after_seconds(r.headers.get("Retry-After"), backoff)
            if wait > MAX_WAIT:
                return None
            await asyncio.sleep(wait)
            continue
        if usable(r):
            return r
        if r.status_code == 403:
            return None                    # refused; retrying the same request will not change that
        await asyncio.sleep(backoff)
    return None

async def main(urls, workers=8):
    limits = httpx.Limits(max_connections=workers, max_keepalive_connections=workers)
    async with httpx.AsyncClient(proxy=PROXY, timeout=httpx.Timeout(30, connect=5),
                                 limits=limits, follow_redirects=True) as client:
        sem = asyncio.Semaphore(workers)
        async def one(u):
            async with sem:
                r = await fetch(client, u)
                await asyncio.sleep(0.1)   # pace, do not sprint
                return u, (r.status_code if r else None)
        return await asyncio.gather(*(one(u) for u in urls))

if __name__ == "__main__":
    print(asyncio.run(main(["https://example.com/"] * 3)))

Through the test proxy it printed [('https://example.com/', 200), ('https://example.com/', 200), ('https://example.com/', 200)]. With a wrong password it raised httpx.ProxyError: 407 Proxy Authentication Required on the first request instead of retrying four times. min_bytes=500 suits example.com, whose page is 577 bytes; set it from a real page on your target. Log four fields per request and most incidents explain themselves: status, response size, exit address and elapsed time.

Choosing between the three libraries

requests 2.34.2 httpx 0.28.1 aiohttp 3.14.5
Async No Yes, alongside sync Async only
HTTP/2 No Yes, with the http2 extra No
Where the proxy goes proxies dict, per request or on a Session proxy= on the client; mounts= for per-scheme rules proxy= on the session or per request
SOCKS requests[socks] (PySocks) httpx[socks] (socksio) Separate aiohttp-socks package
Reads proxy environment variables Yes, by default Yes, by default Only with trust_env=True
How a 407 on HTTPS arrives ProxyError httpx.ProxyError ClientHttpProxyError, e.status == 407

httpx is a reasonable default if you want sync and async code with one API. requests is still the simplest choice for a synchronous script. aiohttp suits code that is already built on it. None of them changes your TLS fingerprint, removes the need to validate content, or makes an uncapped retry loop safe. The same setup for the command line is in how to use curl with a proxy, and for JavaScript in proxies in Node.js.

Frequently asked questions

How do I set a proxy in Python requests?
Pass a proxies dict with both keys, {"http": PROXY, "https": PROXY}, ideally on a requests.Session so connections are reused. The key is the scheme of the site you request; the value is the proxy URL, which is usually http:// even for HTTPS sites.
Why does requests ignore my proxy for HTTPS sites?
The proxies dict probably has only an "http" key. requests picks the proxy by the scheme of the target URL, so an HTTPS request with no "https" key goes direct. In our test it did so without raising anything.
Does httpx still accept proxies=?
No. httpx 0.26.0 added proxy= and deprecated proxies=, and 0.28.0 removed it. httpx 0.28.1 raises TypeError: Client.__init__() got an unexpected keyword argument 'proxies'. Use proxy= for one proxy, or mounts= for rules per scheme or host.
Does requests support SOCKS5 proxies?
Yes, after pip install "requests[socks]", which adds PySocks. Use socks5h:// so the proxy resolves the hostname. With socks5:// requests resolved it locally and sent the proxy a bare IP address.
Why does my code never see a 407 status?
For an HTTPS target the proxy refuses the CONNECT tunnel before any response exists, so requests raises ProxyError, httpx raises httpx.ProxyError and aiohttp raises ClientHttpProxyError. Catch the exception; checking status_code only works for plain-HTTP targets.
Does aiohttp use the HTTPS_PROXY environment variable?
Only when the session is created with trust_env=True. With the default, aiohttp 3.14.5 ignored HTTPS_PROXY and connected directly, while requests and httpx both used it.

Sources

  1. requests documentation: proxies and SOCKS
  2. httpx documentation: proxies
  3. httpx changelog: proxy= added in 0.26.0, proxies= removed in 0.28.0
  4. aiohttp client reference: ClientSession proxy and trust_env
  5. aiohttp changelog: session-level proxy in 3.11.0, BasicAuth and proxy_auth deprecated in 3.14.0
  6. aiohttp-socks on PyPI
  7. RFC 9110 section 10.2.3: Retry-After
  8. RFC 1928: SOCKS Protocol Version 5

Read this page as Markdown · Quote it freely under CC BY 4.0 with a link back.