On this page 13 sections
- The one-line version
- 407 Proxy Authentication Required
- How a 407 shows up in your tools
- 407 or 401?
- 403 Forbidden: first find out who sent it
- 429 Too Many Requests
- 5xx answers that came from the proxy
- The block that returns 200
- A decision tree for while it is failing
- Isolating the proxy from the target
- Backoff that does not resynchronise
- What to log so the next incident is faster
- Two worked incidents
- Sudden 407s on a job that ran fine yesterday
- 403 only on some pages
- Proxy checker
HTTP 407 Proxy Authentication Required is sent by your proxy, not by the website. The proxy wanted credentials and got none, or rejected the ones you sent, so your request never left it. A 403 Forbidden or a 429 Too Many Requests normally comes from the destination, which means the proxy did its job and the site said no. The fix follows from the sender: a 407 is fixed in your proxy settings, a 403 in what your client presents, a 429 in how fast you go.
The quickest way to tell them apart is the response headers, because each sender signs its work. Below are real responses for all three: from a Squid proxy, from three sites behind anti-bot systems, and from the GitHub API. Then comes the way each error surfaces in curl and Python. There, the 407 often does not look like a 407 at all.
The one-line version
| Code | Usually sent by | Means | Header that names the sender | First thing to change |
|---|---|---|---|---|
| 407 | Your proxy | Proxy credentials missing or wrong | Proxy-Authenticate, plus a proxy Server such as squid/6.13 |
Your proxy credentials or username parameters |
| 401 | The destination | The site’s own login or API key failed | WWW-Authenticate |
Your API key or login |
| 403 | The destination, sometimes the proxy | Understood and refused | Anti-bot headers such as x-datadome or x-px-blocked; X-Squid-Error if the proxy sent it |
What your client sends |
| 429 | The destination | Too many requests | Retry-After, rate-limit headers |
Your request rate |
The split that matters is the first row against the rest. A 407 means the site was never contacted. Anything else means the proxy connected and the site, or the proxy’s own policy, answered.
407 Proxy Authentication Required
RFC 9110 section 15.5.8 defines it as the proxy’s version of 401: the client must authenticate to the proxy, and the proxy says how in a Proxy-Authenticate header. This is the complete response Squid 6.13 sent for a wrong password, captured with curl -D -:
HTTP/1.1 407 Proxy Authentication Required
Server: squid/6.13
Mime-Version: 1.0
Date: Wed, 21 Oct 2026 19:29:53 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 3615
X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
Vary: Accept-Language
Content-Language: en
Proxy-Authenticate: Basic realm="proxy-test"
Cache-Status: proxy-test
Via: 1.1 proxy-test (squid/6.13)
Connection: keep-alive
Server, X-Squid-Error and Via all name the proxy. A commercial gateway will show its own values, but the shape is the same: the response comes from a machine you configured, not from the site you asked for. Causes to check, cheapest first:
- Wrong username or password, including a stray space or newline from a config file.
- Provider parameters in the username are malformed. Many gateways encode country or session settings in the username, and a bad separator reads as a bad username.
- The right credentials sent to the wrong port or gateway, where a different product with different users listens.
- IP whitelisting is configured and your server’s address changed, so the proxy now expects credentials you are not sending.
- Credentials containing
@or:were not percent-encoded and the client split the proxy URL in the wrong place. - The account is suspended or out of balance, if your provider reports that through the proxy. Its documentation will say whether it does.
How a 407 shows up in your tools
On an HTTPS request the proxy refuses the CONNECT tunnel, before any response from the site exists. Most tools therefore report it as a connection failure rather than a status code. We sent the same wrong password through the same Squid with each tool:
| Tool | HTTPS target | Plain-HTTP target |
|---|---|---|
| curl 8.7.1 | curl: (56) CONNECT tunnel failed, response 407, %{http_code} 000 |
Exit 0, %{http_code} 407 |
| curl 8.23.0 | curl: (7) CONNECT tunnel failed, response 407, %{http_code} 000 |
Exit 0, %{http_code} 407 |
| requests 2.34.2 | Raises ProxyError containing Tunnel connection failed: 407 Proxy Authentication Required |
Returns a response with status_code 407 |
| httpx 0.28.1 | Raises httpx.ProxyError: 407 Proxy Authentication Required |
— |
| aiohttp 3.14.5 | Raises ClientHttpProxyError with status 407, and the message includes the proxy URL with its password |
— |
Three consequences. A monitoring script that reads %{http_code} logs 000 and calls it a network problem. Python code that checks r.status_code == 407 never fires on HTTPS, because there is no response object. And curl changed the exit code from 56 to 7 in version 8.20.0. A script written for one build misreads the other. The full failure table is in how to use curl with a proxy.
Check the proxy in isolation with a command that prints the proxy’s answer, not the site’s:
curl -sS -o /dev/null -x http://gw.example.net:8000 -U user:pass \
-w 'proxy=%{http_connect} site=%{http_code}\n' https://api.ipify.org
proxy=407 site=000 # wrong credentials: fix the proxy settings
proxy=200 site=200 # proxy accepted you and the site answered
In Python, catch the exception and treat it as final. Retrying a 407 cannot succeed:
import httpx
PROXY = "http://user:pass@gw.example.net:8000"
try:
with httpx.Client(proxy=PROXY, timeout=10) as c:
r = c.get("https://api.ipify.org")
print("proxy ok, site said", r.status_code)
except httpx.ProxyError as e:
print("proxy refused the tunnel:", e) # proxy refused the tunnel: 407 Proxy Authentication Required
407 or 401?
Both mean “authenticate”, and they are easy to swap in a log. 407 is the proxy asking for proxy credentials and carries Proxy-Authenticate. 401 is the site asking for its own credentials and carries WWW-Authenticate. If you get a 401 through a proxy, the proxy worked: it delivered your request, and the site wants an API key or a login.
403 Forbidden: first find out who sent it
Proxies send 403 too. When we asked Squid for a port its policy does not allow, it answered itself:
HTTP/1.1 403 Forbidden
Server: squid/6.13
X-Squid-Error: ERR_ACCESS_DENIED 0
Through a tunnel, curl prints that as CONNECT tunnel failed, response 403, which is the giveaway: a 403 inside “CONNECT tunnel failed” came from the proxy. A 403 from the site arrives after the tunnel is up, as an ordinary response. Squid’s default configuration refuses CONNECT to any port except 443, so a target on 8443 can fail this way through a perfectly healthy proxy.
When a site’s 403 comes from an anti-bot system, the headers can say which one. We sent one default curl 8.23.0 request to each of three sites, then one more with a Chrome User-Agent and browser Accept headers:
| Site | Default curl | Headers that identify the blocker | Body | With a Chrome User-Agent |
|---|---|---|---|---|
| www.g2.com | 403 | x-datadome: protected, a datadome cookie, server: cloudflare |
911 bytes | 403, 913 bytes |
| www.indeed.com | 403 | server: cloudflare, a __cf_bm cookie, page title “Blocked – Indeed.com” |
9,714 bytes | 403, 11,198 bytes |
| www.zillow.com | 403 | x-px-blocked: 1, server: CloudFront, page title “Access to this page has been denied” |
5,776 bytes | 200, 51,137 bytes, the real home page |
One of the three was blocking on the User-Agent alone. With curl’s default curl/8.23.0 it refused, and with a browser string on the same curl handshake, from the same address, it served the page. The other two kept refusing, which points past headers to the handshake, JavaScript or the address. Three requests prove nothing about those sites in general, but they do settle the order of operations: try the free change before the expensive one.
- Headers. Send a real browser User-Agent with matching
AcceptandAccept-Language. It costs nothing to try. Copy them from a real browser request rather than inventing them; the default request headers study shows how far library defaults are from a browser’s. - TLS fingerprint. If headers change nothing, the site may be reading the handshake, which no header can alter. See the TLS ClientHello study.
- JavaScript. If the site issues a token from a script, an HTTP client cannot pass, whatever the proxy.
- Address reputation. Only now consider whether the exit itself is the problem, and test that by changing the address and nothing else.
Starting at step four is the expensive mistake. Paying for mobile proxies to fix a User-Agent check, or a TLS check, costs more and changes nothing. The proxy type comparison covers when a different class of address does help. Cloudflare documents one more marker worth grepping for: a cf-mitigated: challenge response header means the answer is a challenge page, whatever the status. None of our three responses carried it.
429 Too Many Requests
RFC 6585 section 4 defines 429 and lets the server say when to come back with Retry-After. RFC 9110 allows two forms, and both appear in the wild:
HTTP/1.1 429 Too Many Requests
Retry-After: 2
HTTP/1.1 429 Too Many Requests
Retry-After: Wed, 21 Oct 2026 23:59:00 GMT
Code that assumes seconds breaks on the second form. float() on it raises ValueError: could not convert string to float: 'Wed, 21 Oct 2026 23:59:00 GMT', and a date can also be hours away, which a naive loop turns into a hang. Parse both forms and cap the wait; the backoff code further down does both.
Not every rate limit uses 429. GitHub’s REST API documentation says that exceeding the primary rate limit returns “a 403 or 429 response”. An unauthenticated request we sent to api.github.com carried its counters in the headers:
HTTP/2 200
x-ratelimit-limit: 60
x-ratelimit-remaining: 50
x-ratelimit-used: 10
x-ratelimit-resource: core
x-ratelimit-reset: 1792612141
So a 403 from an API is worth checking for x-ratelimit-remaining: 0 before you treat it as a block. x-ratelimit-reset is a Unix timestamp; GitHub says to wait until then when no Retry-After is sent.
Rotating to a fresh address and carrying on at the same speed is the instinctive response and often the wrong one. Some limits follow the account rather than the address: GitHub counts authenticated requests against a personal limit of 5,000 per hour, so a new address changes nothing for them. Others follow the address, and rotating through a pool at the same rate spreads the problem across many addresses instead of pausing one. Slow down first. If you use curl, --retry already obeys Retry-After: against a server sending Retry-After: 2, --retry 2 waited two seconds twice and finished in 4.1 seconds on both curl builds.
5xx answers that came from the proxy
A 5xx through a proxy is not automatically the site’s fault. Squid answers for the site when it cannot reach it, and its error header says why:
| What we requested through Squid 6.13 | Status | X-Squid-Error |
|---|---|---|
| A hostname that does not exist | 503 | ERR_DNS_FAIL 0 |
| A port where nothing listens | 503 | ERR_CONNECT_FAIL 61 |
| A port the proxy’s policy does not allow | 403 | ERR_ACCESS_DENIED 0 |
The 61 is the operating system’s error number for “connection refused” on macOS. Other proxies use other codes for the same situations: RFC 9110 reserves 502 for an invalid answer from upstream and 504 for an upstream timeout. Whatever the number, a Server or error header from the proxy means the site never answered.
The block that returns 200
The hardest case is a 200 OK with nothing useful in it: an empty result list, a challenge page, or plausible decoy content. Serving an honest error tells a scraper what to fix; serving an empty 200 wastes its time and bandwidth. Build the check into the fetch layer, not into analysis afterwards:
BLOCK_MARKERS = ("captcha", "unusual traffic", "are you a robot",
"access denied", "verify you are human")
def looks_blocked(r, min_bytes=1_500):
if r.status_code in (403, 429):
return True
if r.headers.get("cf-mitigated") == "challenge":
return True
body = r.text.lower()
if any(m in body for m in BLOCK_MARKERS):
return True
return r.status_code == 200 and len(r.content) < min_bytes # suspiciously small
Set min_bytes from a real page on your target. A response is a success when it contains what you asked for, not when it arrives with a 200. Without this check your success rate looks healthy while the database fills with block pages.
A decision tree for while it is failing
Got an error
|
+- curl says "CONNECT tunnel failed, response 407",
| or Python raises ProxyError ----------------------> Your proxy. Check credentials, username
| parameters, port, whitelist. Stop here.
|
+- "CONNECT tunnel failed, response 403/503" --------> The proxy answered for the site: policy,
| DNS or connection failure. Read X-*-Error.
|
+- 401 -----------------------------------------------> The site's own auth. The proxy worked.
|
+- 429, or 403 with x-ratelimit-remaining: 0 ---------> Slow down. Honour Retry-After. Do not rotate first.
|
+- 403 from the site ---------------------------------> 1. headers 2. TLS fingerprint
| 3. JavaScript 4. address, last
|
+- 200 but wrong or empty ----------------------------> Validate content. It is a block wearing a 200.
Isolating the proxy from the target
Send two requests through the same proxy at the same moment, one to a neutral echo service and one to your target, and print the proxy’s answer separately from the site’s:
PROXY=http://gw.example.net:8000
AUTH=user:pass
W='proxy=%{http_connect} site=%{http_code} exit=%{exitcode}\n'
curl -sS -o /dev/null -x "$PROXY" -U "$AUTH" -w "echo: $W" https://api.ipify.org
curl -sS -o /dev/null -x "$PROXY" -U "$AUTH" -w "target: $W" https://target.example.net/page
The variables are quoted on purpose. An unquoted $P holding several flags splits into words in bash but not in zsh, the default macOS shell. There curl receives the whole string as one argument and stops with curl: (5) Unsupported proxy syntax in ' http://127.0.0.1:3128 -U user:pass'.
| Echo | Target | Conclusion |
|---|---|---|
| proxy=407 | proxy=407 | Proxy credentials. Nothing to do with the target. |
| proxy=200 site=200 | proxy=200 site=403 | The proxy is healthy. The target is refusing your client. |
| proxy=200 site=200 | proxy=200 site=429 | The proxy is healthy. You are going too fast. |
| proxy=200 site=200 | proxy=403 or 503 | The proxy refused or could not reach that one target. |
| exit 5, 7 or 28, proxy=000 | Same | The proxy itself is unreachable or not answering. |
Two commands and that table name the system that failed. That alone stops the common mistake of rewriting scraper logic when the fault is an expired password.
Backoff that does not resynchronise
Honour Retry-After when it is present, in both of its forms, and add jitter when it is not. Without jitter, parallel workers that hit a limit together back off together and retry together, reproducing the burst that caused the limit:
import random
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
def backoff(attempt, retry_after=None, cap=60):
"""Seconds to wait before the next try. Returns None when the server asks for more than cap."""
if retry_after:
value = retry_after.strip()
if value.isdigit():
wait = int(value)
else:
try:
wait = (parsedate_to_datetime(value) - datetime.now(timezone.utc)).total_seconds()
except (TypeError, ValueError):
wait = None
if wait is not None:
return max(0.0, wait) if wait <= cap else None
base = min(cap, 2 ** attempt)
return base * (0.5 + random.random() * 0.5) # 50-100% of base
print(backoff(0, "2")) # 2
print(backoff(0, "Wed, 21 Oct 2026 23:59:00 GMT")) # None once that is more than 60 s away
print(round(backoff(3), 2)) # between 4 and 8
A None means stop and reschedule the job rather than sleep. Cap the attempt count as well as the delay: an uncapped retry loop is an outage generator, and on a per-gigabyte plan it spends money while achieving nothing. The same parser inside a full retry loop for requests and httpx is in Python requests proxy setup.
What to log so the next incident is faster
- The status code and the response size together. Either alone misleads.
- The proxy’s answer separately from the site’s:
%{http_connect}in curl, the exception type in Python. - The
Serverheader and anyX-*-Error,x-datadomeorx-px-*header, which name the sender. - Whether
Retry-Afterorx-ratelimit-*headers were present, and their values. - The exit address in use, and a hash of the body, which makes a repeated block page obvious at a glance.
Two worked incidents
Sudden 407s on a job that ran fine yesterday
Test the credentials alone with the echo command above. If the proxy still answers 407, check whether the job moved: a container rescheduled onto another host gets a different egress address, and IP whitelisting silently stops matching. Then check for a password rotation upstream of you, and the account state with the provider. The address change is the cause people miss, because nothing in the error mentions it.
403 only on some pages
Listing pages work and detail pages fail. A blanket address block would not be that selective. The protected pages more likely expect a cookie or token set on the listing page, a plausible Referer, or sit behind a stricter rule. Fetch the listing page first in the same session, keep its cookies, send the listing URL as Referer, and try again. If that fixes it, the proxy was never the problem.
Going to your provider? Bring evidence: the exact status codes with timestamps, whether the echo service worked through the same proxy at the same moment, the exit address and country you got against what you asked for, and one curl command that reproduces the failure. If you would rather test a proxy from the browser, the checker below opens one real connection through it and reports what came back.
Proxy checker
Live testWe open a real connection through your proxy and report what the exit node presented. Credentials are used for the single request and are never stored.
Frequently asked questions
What does 407 Proxy Authentication Required mean?
What is the difference between 401 and 407?
Why does curl show 000 instead of 407?
Can a proxy send a 403 Forbidden?
Should I rotate my proxy when I get a 429?
Why do I get 403 with a residential proxy?
Sources
- RFC 9110 section 15.5.8: 407 Proxy Authentication Required
- RFC 9110 section 11.7.1: Proxy-Authenticate
- RFC 9110 section 10.2.3: Retry-After
- RFC 6585 section 4: 429 Too Many Requests
- GitHub REST API: rate limits, 403 or 429 on exceeding the primary limit
- Cloudflare: detecting a challenge response (cf-mitigated)
- Squid http_access: default deny CONNECT !SSL_ports
- curl pull request #21084: CONNECT failures return exit code 7 from 8.20.0