403 vs 407 vs 429: reading proxy and block errors

Three status codes account for most of the confusion when a proxied request fails. They look interchangeable in a log and are not. Reading them correctly tells you which system rejected you, and therefore which thing to change.

#The one-line version

The critical split is the first row against the rest. A 407 means your request never left the proxy. Everything else means the proxy worked and the destination answered.

#407 Proxy Authentication Required

Defined in RFC 9110 §15.5.8. The proxy replies with a Proxy-Authenticate header describing the scheme it expects.

HTTP/1.1 407 Proxy Authentication Required
Proxy-Authenticate: Basic realm="proxy"

Causes, in rough order of frequency:

• Wrong username or password.
• Credentials sent to the wrong port — many providers use port to select behaviour.
• Provider-specific parameters malformed. Username fields often encode country or session identifiers, and a bad separator reads as a bad username.
• IP whitelisting configured, but your server’s address changed.
• Account out of balance or suspended. Several providers signal this as 407 rather than something clearer.

Confirm quickly with curl. If this returns 407, the problem is entirely on your side of the connection:

curl -x http://user:[email protected]:8000 -sS -o /dev/null \
  -w "%{http_code}\n" https://api.ipify.org

#403 Forbidden

The destination understood the request and refused it. When it comes from an anti-bot system, the address is rarely the whole story.

Work through the layers in this order, because the cheapest fixes are also the most common causes:

1. TLS fingerprint. Does your client handshake like a browser? TLS fingerprinting happens before any header is read, so nothing you set in headers can compensate.
2. Header coherence. Order, casing and completeness. A browser User-Agent on a Python handshake is an inconsistency, not a disguise.
3. JavaScript. Does the site require execution to issue a token? If so, an HTTP client cannot pass regardless of proxy quality.
4. Address reputation. Only now consider whether the exit itself is the problem.

The expensive mistake is starting at step four. Buying mobile proxies to solve a TLS fingerprint problem costs a great deal and changes nothing.

#429 Too Many Requests

Defined in RFC 6585 §4. A well-behaved server tells you when to return:

HTTP/1.1 429 Too Many Requests
Retry-After: 30

Retry-After is either a number of seconds or an HTTP date. Honour it. Backing off for an interval you invented, when the server has told you the correct one, is how a temporary limit becomes a durable block.

Rotating to a fresh address and continuing at the same speed is the instinctive response and usually the wrong one. It burns pool reputation across many addresses instead of pausing one, and rate limits are frequently applied per account or per fingerprint rather than per address.

#The failure that has no status code

The hardest case returns 200 OK with nothing useful: an empty result array, a challenge page, or plausible-looking decoy content. This is deliberate. Serving an honest error tells a scraper exactly what to fix; serving an empty 200 wastes its time and its bandwidth.

The defence is to validate content rather than status:

ok = (
    r.status_code == 200
    and len(r.content) > 2_000
    and "expected-marker" in r.text
    and "captcha" not in r.text.lower()
)

Without this, your monitoring reports a healthy success rate while your database fills with nothing.

#A diagnostic order that saves time

1. Reproduce with curl. Removes your application from the equation.
2. Check the status. 407 stops here — fix the proxy.
3. Request an echo service through the same proxy. If that succeeds, the proxy is healthy and the target is refusing you specifically.
4. Compare your request against a real browser’s, header by header.
5. Only then change proxy type.

#A decision tree you can follow while it is failing

When something breaks in production, the useful thing is an order of operations rather than a list of possibilities.

Got an error
│
├─ 407 ──────────► Your proxy. Check credentials, port, username parameters,
│                  whitelist, and account balance. Stop here.
│
├─ 401 ──────────► You reached the destination. Its own auth failed.
│                  The proxy is working correctly.
│
├─ 429 ──────────► Slow down. Honour Retry-After. Do not rotate first.
│
├─ 403 ──────────► Reached and refused. Work the layers:
│                    1. TLS fingerprint
│                    2. Header order and completeness
│                    3. JavaScript requirement
│                    4. Address reputation  ← last, not first
│
└─ 200 but empty ► Validate content. This is a block wearing a success code.

#Isolating proxy from target in one step

The single most useful diagnostic is to send two requests through the same proxy at the same moment — one to your target, one to something neutral:

PROXY="http://user:[email protected]:8000"

echo "echo:   $(curl -sS -o /dev/null -w '%{http_code}' -x "$PROXY" https://api.ipify.org)"
echo "target: $(curl -sS -o /dev/null -w '%{http_code}' -x "$PROXY" https://target.example/page)"

That table resolves most incidents in about thirty seconds, and it prevents the common error of rewriting scraper logic when the actual fault is an expired password.

#Backoff that does not resynchronise

The detail people omit is jitter. Without it, parallel workers that hit a limit together back off together and retry together, reproducing exactly the burst that caused the limit:

import random, time

def backoff(attempt, retry_after=None, cap=60):
    if retry_after is not None:
        return float(retry_after)                 # the server told you
    base = min(cap, 2 ** attempt)
    return base * (0.5 + random.random() * 0.5)   # 50-100% of base

for attempt in range(5):
    r = fetch()
    if r.status_code != 429:
        break
    time.sleep(backoff(attempt, r.headers.get("Retry-After")))

Two rules hold generally: honour Retry-After whenever it is present, and cap both the delay and the attempt count. An uncapped retry loop is an outage generator, and it will consume your bandwidth allowance while achieving nothing.

#Detecting the block that returns 200

This is the failure mode that quietly ruins datasets, because every dashboard reports success. Build the check into your fetch layer rather than into analysis:

BLOCK_MARKERS = ("captcha", "unusual traffic", "are you a robot",
                 "access denied", "verify you are human")

def looks_blocked(r):
    if r.status_code in (403, 429):
        return True
    body = r.text.lower()
    if any(m in body for m in BLOCK_MARKERS):
        return True
    if r.status_code == 200 and len(r.content) < 1_500:
        return True                     # suspiciously small for a real page
    return False

Tune the size threshold to your target. The principle is what matters: a response is successful when it contains what you asked for, not when it arrives with a 200.

#Status codes you will also meet

A connection reset during the handshake is worth singling out. There is no status code because no HTTP response was ever sent — the refusal happened at the TLS layer, which is a strong hint that your client fingerprint, not your address, is what was rejected.

#What to log so the next incident is faster

• The status code and the response size together — either alone is misleading.
• The exit address in use at the time.
• Whether Retry-After was present, and its value.
• Time to connect, separately from time to first byte.
• A hash of the response body, which makes repeated identical block pages obvious at a glance.

#Two worked incidents

#Sudden 407s across a job that ran fine yesterday

Everything was working; now every request returns 407. The instinct is to suspect the provider. Work the list instead:

1. Test the credentials in isolation with curl against an echo service. Still 407? The problem is authentication, not your code.
2. Check whether the job moved. A container rescheduled onto a different host gets a different egress address, and IP whitelisting silently stops matching.
3. Check the account balance. Several providers return 407 rather than a clearer signal when a plan is exhausted.
4. Check whether a password rotation happened upstream of you.

The address-change cause is the one people miss, because nothing in the error mentions it.

#403 only on some pages

Listing pages work; detail pages fail. That pattern is informative on its own — a blanket address block would not be selective. Selective refusal usually means the protected pages carry a token issued by JavaScript on the referring page, or they check Referer, or they sit behind a stricter rule in the anti-bot configuration.

Test by fetching the listing page first in the same session, keeping cookies, and sending a plausible Referer. If that fixes it, the problem was never the proxy, and no amount of address quality would have solved it.

#What to check before contacting a provider

Support conversations go faster with evidence. Before opening a ticket, gather:

• The exact status codes, with timestamps.
• Whether an echo service succeeds through the same proxy at the same moment.
• The exit address and country you were given, against what you requested.
• Whether the failure is universal or specific to one destination.
• A curl command that reproduces it.

If an echo service works and only your target fails, that is worth knowing before you ask a provider to investigate — the answer is very likely to be that their network is fine and the destination is refusing your client.