Reverse proxy

A reverse proxy sits in front of one or more origin servers and accepts connections on their behalf. The client believes it is talking to the real service; the reverse proxy decides which backend actually answers.

#What it is used for

• Load balancing across several backends.
• TLS termination, so certificates are managed in one place.
• Caching of static responses.
• Filtering, including the anti-bot systems that scrapers run into.

Nginx, HAProxy, Cloudflare and most CDNs operate as reverse proxies.

#Why it matters when scraping

The thing blocking your request is very often a reverse proxy rather than the application itself. That is why a block can arrive in single-digit milliseconds and why the response frequently looks nothing like the site’s normal error page.

#Commonly confused with

A forward proxy represents the client. A reverse proxy represents the server. The word “proxy” alone almost always means the forward kind.

#What the scraper actually meets

Almost every large site answers from a reverse proxy rather than from the application. That intermediary, not the application, usually issues the block. It explains several behaviours that otherwise look inconsistent.

• The response arrives too quickly to have reached the application. A block served in a few milliseconds came from the edge.
• The error page does not match the site’s design. Edge providers serve their own challenge pages.
• Headers name the intermediary. Server, Via and vendor-specific ray or trace identifiers appear on the response.

This matters when you diagnose a failure. A CAPTCHA or a 403 from the edge means the request never reached the origin, so changing what you ask for will not help. Changing how the request looks might.

#Inspecting the layer

curl -sI https://example.com/ | grep -iE 'server|via|cf-|x-cache|x-served-by'

Named intermediaries and cache-status headers tell you an edge is present. Absence proves nothing, because many operators strip these headers deliberately.