Glossary

Referer header

Referer is a request header naming the resource from which the target URI was obtained.

Referer is a request header naming the resource from which the target URI was obtained. The field name is misspelled, and the specification preserves the misspelling deliberately, noting it in place with a bracketed sic.

#What it may and may not contain

RFC 9110 requires that the fragment and userinfo components be excluded from the value. It permits a user agent to truncate anything beyond the referring origin, so a short value is not evidence of tampering. Where the target URI came from something with no URI of its own, such as typed input or a bookmark, the header must either be omitted or sent as about:blank.

Referer: http://www.example.org/hypertext/Overview.html

#The downgrade rule

A user agent must not send Referer in an unsecured HTTP request when the referring resource was accessed over a secure protocol, and should not send it across origins when the referring resource was secure, unless that resource allows it. This is a hard requirement, not a preference, and it explains a great many missing headers that get blamed on something else.

The W3C Referrer Policy specification defines the tokens that control the rest: no-referrer, same-origin, origin, strict-origin, origin-when-cross-origin, strict-origin-when-cross-origin, unsafe-url and no-referrer-when-downgrade. Which of them a browser applies by default has changed over time and differs between browsers, so observe it rather than assume it.

#Using it when scraping

Sites use the header for analytics, for deep-link restrictions and as one input to request-forgery defences, while the specification notes that not all requests carry it. Fabricating a value that matches the navigation you are claiming to have made is consistent; fabricating one that contradicts your other headers is not. It is a weak signal on its own and we make no claim about its effect on any particular target.

#Commonly confused with

Origin carries only a scheme, host and port and is sent on requests where the full path would be too revealing. Referer may carry a path. See also user agent, cookie jar and anti-bot systems, which read these headers together rather than separately.

Frequently asked questions

Why is Referer spelled that way?

It was misspelled in an early specification and the name became load bearing, so correcting it would break deployed software. RFC 9110 keeps the spelling and marks it with a bracketed sic in the definition, while the surrounding prose spells the English word referrer correctly.

Why does the header disappear on some requests?

Most often the downgrade rule. A user agent must not send Referer in an unsecured HTTP request when the referring page was loaded over a secure protocol, and referrer policy may strip or truncate it in other cases. Requests originating from typed input or bookmarks have no referring resource at all.

Should my scraper send a Referer header?

Send one when your request is claiming to follow a link that would have produced it, and make it consistent with the rest of the request. An inconsistent value is worse than none. Treat it as one small part of coherence rather than as a measure that changes outcomes on its own.