A user agent is a sentence the client writes about itself. Anyone can send Googlebot/2.1, and scanners do, because some sites wave crawlers through rules that stop everyone else. The address cannot be faked the same way: a request has to come from it for the connection to work. So the question “is this really Googlebot?” is a question about the address, and the operators publish what is needed to answer it.
The two checks operators document
The published IP list. Google, Microsoft, OpenAI, Anthropic, Perplexity, Apple, Amazon and DuckDuckGo each publish the address ranges their crawlers use. If the address is inside one of them, the crawler is genuine. The table under the form links every file we read.
Forward-confirmed reverse DNS. Google, Microsoft and Apple also document a DNS check. Look up the address’s PTR record; the name must end in the operator’s domain; then look up that name and confirm it points back to the same address. The second step is what makes it proof. Whoever controls an address block can set its PTR record to any name, but only the operator controls what googlebot.com names resolve to.
$ dig -x 66.249.66.1 +short
crawl-66-249-66-1.googlebot.com.
$ dig crawl-66-249-66-1.googlebot.com A +short
66.249.66.1How the verdict is decided
| Result | When we say it |
|---|---|
| Genuine | The address is in the operator’s published list, or it passes forward-confirmed reverse DNS under the operator’s documented domain. |
| Spoofed | The operator’s own documented check says no. For OpenAI, Anthropic, Perplexity and DuckDuckGo, which document only a list, that means the address is outside the list. For the three that also document DNS, the address must fail the DNS check too. |
| Could not verify | A list could not be fetched or a DNS lookup did not finish. Nothing is concluded from a check that did not run. |
One domain is deliberately not accepted for Googlebot: googleusercontent.com. Google’s documentation lists it for user-triggered fetchers, and it is also where rentable Google Cloud machines get their default reverse DNS names. A scanner on a cloud VM therefore passes a naive “does it end in a Google domain” check. It does not pass this one.
Check a list yourself
Most lists are JSON files of CIDR prefixes; Amazon prints its list on a web page instead. Python’s standard library is enough to test an address against one:
import ipaddress, json, urllib.request
url = "https://developers.google.com/static/crawling/ipranges/common-crawlers.json"
data = json.load(urllib.request.urlopen(url, timeout=10))
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in data["prefixes"]]
ip = ipaddress.ip_address("66.249.66.1")
print(any(ip in n for n in nets if n.version == ip.version))What we do with the address you enter
We check it and discard it. It travels in the body of the request rather than the URL, so it does not appear in our access logs, and it is not stored. The range files are fetched from the operators by our server and refreshed every 12 hours. If a refresh fails, the previous copy is used for up to 14 days; the result labels that evidence stale and includes when our copy was fetched.