What is rate limiting?
Definition
A cap on how many requests a client may make in a time window. Going over it usually returns 429 Too Many Requests, often with Retry-After.
The limit is counted against something: an address, an account, an API key, a session cookie. Which one it is decides whether a new proxy address helps. Rotating addresses escapes a counter kept per address and nothing else. Against any other counter the burst that caused the limit simply moves with you.
Reading the counter before you hit it
Many APIs publish the counter on every response. One command shows GitHub’s:
$ curl -sI https://api.github.com/repos/curl/curl | grep -i '^x-ratelimit'
x-ratelimit-limit: 60
x-ratelimit-remaining: 46
x-ratelimit-used: 14
x-ratelimit-resource: core
x-ratelimit-reset: 1792612141
Sixty unauthenticated requests an hour, which GitHub’s documentation says are counted against the originating IP address, with the reset given as a Unix timestamp. This is the case where a different address does buy a fresh allowance. The same documentation gives authenticated requests a higher limit.
The same curl 8.7.1 request to Crunchbase’s home page came back with a different family of headers:
ratelimit-limit: 150
ratelimit-policy: 150;w=900
ratelimit-remaining: 149
ratelimit-reset: 900
Those are the field names from earlier drafts of the IETF’s RateLimit header specification: a policy of 150 requests in a 900-second window, and a reset counted in seconds from now rather than as a timestamp. The current draft has since merged them into two structured fields, RateLimit and RateLimit-Policy. So three conventions are live at once, and the same word, reset, means a clock time in one and a countdown in another. Read them defensively and never require them to be present.
Which header means what
| Header | Defined by | What its value means |
|---|---|---|
429 Too Many Requests |
RFC 6585, section 4 | The status itself. The response may include Retry-After |
Retry-After |
RFC 9110, a standard | Seconds to wait, or an HTTP date |
X-RateLimit-Limit, -Remaining, -Reset |
Convention only | Varies by service. GitHub’s reset is a Unix timestamp |
RateLimit-Limit, -Remaining, -Reset, RateLimit-Policy |
Earlier IETF drafts | Reset is seconds until the window restarts |
RateLimit, RateLimit-Policy with q, w, r, t |
Current IETF draft | Quota, window, remaining quota, seconds to reset |
Backing off without making it worse
# Python 3.9.6, requests 2.32.3
import email.utils, random, time
import requests
def retry_after_seconds(value):
"""Retry-After is either delay-seconds or an HTTP-date (RFC 9110)."""
if not value:
return None
value = value.strip()
if value.isdigit():
return float(value)
try:
when = email.utils.parsedate_to_datetime(value)
except (TypeError, ValueError):
return None
return max(0.0, when.timestamp() - time.time())
def get_with_backoff(session, url, attempts=5, base=1.0, ceiling=60.0):
for attempt in range(attempts):
r = session.get(url, timeout=15)
if r.status_code not in (429, 503):
return r
wait = retry_after_seconds(r.headers.get("Retry-After"))
if wait is None:
# No instruction from the server: exponential, capped, with jitter.
wait = min(ceiling, base * 2 ** attempt) * random.uniform(0.5, 1.5)
elif wait > ceiling:
return r # asked to wait longer than we will: stop, never retry early
print(f"attempt {attempt + 1}: {r.status_code}, sleeping {wait:.1f}s")
time.sleep(wait)
return r
Against a local test server that answered 429 with Retry-After: 2, then 429 with an HTTP date three seconds ahead, then a bare 503, then 200, it printed:
attempt 1: 429, sleeping 2.0s
attempt 2: 429, sleeping 2.3s
attempt 3: 503, sleeping 2.3s
final 200 after 6.7s
The second wait is 2.3 seconds rather than 3 because HTTP dates have one-second resolution. Each part of the function does a job. Retry-After wins when present, because it is an instruction rather than a hint. Exponential growth gives a limit time to clear when the server says nothing. The jitter factor stops parallel workers that failed together from retrying together and recreating the burst. The ceiling and the attempt count stop a long outage turning into an unbounded loop. Status handling in more depth is in the 429 section of 403 vs 407 vs 429, and the per-library details in Using proxies in Python.
A limit we hit while measuring
While metering proxy traffic for the bandwidth billing entry, we sent curl 8.7.1 requests to Wikimedia hosts in quick succession. The fourteenth came back 429 with a 2,172-byte body. We cannot see which counter tripped. Wikimedia’s user-agent policy does ask automated clients to send a descriptive user agent with contact details, and names curl among the generic agents to avoid, so the first fix there is to slow down and identify yourself, not to change address. Pacing a crawl per host is covered in Collecting training data for a language model, and how rotation interacts with limits in Rotating vs sticky sessions.
How this was captured. 21 October 2026, curl 8.7.1 from one consumer connection; the GitHub and Crunchbase headers are from single requests. The backoff function was run once with Python 3.9.6 and requests 2.32.3 against a local server scripted to return the sequence described.
Frequently asked questions
What is the difference between 429 and 403?
Should I always honour Retry-After?
Does rotating proxies get around rate limits?
Sources
- RFC 6585: status code 429, Too Many Requests
- RFC 9110: HTTP Semantics, the Retry-After field
- IETF httpapi: RateLimit header fields for HTTP (current draft)
- IETF httpapi: RateLimit header fields, draft 05 with RateLimit-Limit, -Remaining and -Reset
- GitHub Docs: rate limits for the REST API
- Wikimedia Foundation: User-Agent policy