---
title: "What is crawl budget?"
url: https://proxy.wiki/glossary/crawl-budget/
type: Glossary Term
author: "proxy.wiki editorial"
published: 2026-09-06
updated: 2026-10-21
site: proxy.wiki
topics: ["Web scraping"]
license: CC BY 4.0; quote freely with attribution to https://proxy.wiki/
---

# What is crawl budget?

> Crawl budget is Google's term for the set of URLs Googlebot can and wants to crawl on a site, set by server capacity and crawl demand.

Most sites never need to think about it. Google’s own guide is written for sites with roughly a million or more unique pages that change about weekly, or 10,000 or more that change daily, and calls those figures a rough estimate rather than thresholds. Below that, Google says keeping the sitemap current and checking the indexing report is enough.

## The two halves

Google splits it into a capacity limit and demand. The capacity limit is how hard Googlebot can crawl without hurting the server: it goes up while the site answers quickly and consistently, and down when the site slows or returns 5xx errors or 429. Demand is how much Google wants to crawl: popular URLs attract more, a site move can raise it, and duplicate or unneeded URLs waste it. Google calls that last factor the one you can positively control the most.

The capacity side is now controlled from the server only. Google retired Search Console’s crawl rate limiter on 8 January 2024, saying that Googlebot slows down almost immediately when a server persistently returns HTTP 500 for a range of URLs, or when responses get much slower. The documented way to reduce crawling is the server’s own responses.

## Count real Googlebot before you count anything

Any budget you estimate from logs is only as good as your filter for Googlebot, because the user agent is free to copy. We checked this site’s own log for the 45 days from 6 September to 21 October 2026. It records only requests that reach WordPress, so static files such as robots.txt are missing from it.

- 424 requests carried a Googlebot user agent, from 45 distinct addresses.

- 27 addresses, sending 286 of the requests, passed a reverse lookup ending in googlebot.com or google.com and a forward lookup returning the same address. All 27 are inside Google’s published `common-crawlers.json` ranges.

- 18 addresses, sending the other 138, failed. Sixteen of them reverse-resolved to names under bc.googleusercontent.com, the pattern Google Cloud gives its customers’ virtual machines. None of the 18 appear in any of Google’s published crawler or fetcher lists.

- The impostors asked for paths like `/config/env/aws_credentials.env`, `/sa.json`, `/serverless.yaml` and `/actuator/mappings`. That is a hunt for leaked credentials, not indexing.

The googleusercontent.com result is the trap. Google’s verification steps accept googleusercontent.com as a valid domain, because some of its user-triggered fetchers run there, so a check that accepts the whole domain lets any Google Cloud tenant through. For a request claiming to be Googlebot, expect `crawl-*.googlebot.com` or `geo-crawl-*.geo.googlebot.com`, or test the address against the published list:

```
$ host 192.178.4.161
161.4.178.192.in-addr.arpa domain name pointer crawl-192-178-4-161.googlebot.com.
$ host crawl-192-178-4-161.googlebot.com
crawl-192-178-4-161.googlebot.com has address 192.178.4.161
```

```
import ipaddress, json, sys, urllib.request

URL = "https://developers.google.com/static/crawling/ipranges/common-crawlers.json"
prefixes = json.load(urllib.request.urlopen(URL))["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
for ip in sys.argv[1:]:
    print(ip, any(ipaddress.ip_address(ip) in n for n in nets))
```

```
$ python3 is_googlebot.py 192.178.4.161 203.0.113.7     # Python 3.15.0
192.178.4.161 True
203.0.113.7 False
```

## Where 286 verified fetches went

Same log, verified requests only, grouped by the kind of URL:

_Verified Googlebot requests to proxy.wiki by URL type, 6 September to 21 October 2026_

| URL type | Requests |
| --- | --- |
| Glossary entries | 54 |
| Tag and topic archives | 53 |
| /ads.txt | 46 |
| Sitemaps | 41 |
| Legal and about pages | 27 |
| REST API under /wp-json/ | 18 |
| Guides | 15 |
| Section index pages | 15 |
| Home page | 11 |
| Feeds and other | 5 |
| Research articles | 1 |

Seventy of the 286, about a quarter, reached an article. The rest went to archives, a text file for ad networks, sitemaps, legal pages and API endpoints. On a site this small that is a demand question, not a capacity one: the server answered 280 of the 286 with a 200. It is one small site over 45 days, so treat it as an illustration of the method rather than a benchmark.

## Spending it badly

The waste Google describes is URLs that should not exist: faceted filters in every combination, sort parameters, session identifiers in paths and calendars with no end. Each produces distinct URLs for the same content. Canonical URLs, disallow rules and fewer generated links reduce it. Do not count on `Crawl-delay`: RFC 9309 does not define it, and Google’s robots.txt documentation says Google supports only user-agent, allow, disallow and sitemap. RFC 9309 does set two limits worth knowing. A crawler must parse at least 500 kibibytes of the file, and should not use a cached copy for more than 24 hours without caching directives.

## Your own crawler has one too

If you run the crawler, the same idea applies per host: a fixed allowance of requests and bytes, spent on URLs worth having. Normalising and deduplicating URLs before they are queued saves requests before any are made, which is the argument of [deduplication is most of the work](/guides/collecting-llm-training-data/#deduplication-is-most-of-the-work). An agent that fetches a few pages to answer one question is not crawling at all, as [fetching to answer a question is not crawling](/guides/proxies-for-ai-agents/#fetching-to-answer-a-question-is-not-crawling) explains, and the publishers in [our survey of AI crawler policies](/research/ai-crawler-robots-policies/) draw the same line between training and answering in robots.txt. [Honeypot links](/glossary/honeypot-link/) exist to waste exactly this allowance.

## Frequently asked questions

### Is crawl budget a published number?

No. Google describes how it is set but does not publish a figure for any site. What you can measure is your own log: which URLs verified Googlebot requested, how often, and what each returned.

### Does Crawl-delay in robots.txt control Googlebot?

No. RFC 9309 does not define it and Google's documentation says Google does not support it. Google points instead to the server's own responses: persistent 5xx errors or 429 make Googlebot slow down.

### How do I tell real Googlebot from a fake?

Reverse-resolve the address, check that the name ends in googlebot.com or google.com, resolve that name forward and compare, or test the address against Google's common-crawlers.json. In our log, 18 of 45 addresses claiming to be Googlebot failed.

## Sources

1. [Google Search Central: crawl budget management for large sites](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget)
2. [Google Search Central: verifying Googlebot and other Google crawlers](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot)
3. [Google: published IP ranges of common crawlers, including Googlebot](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)
4. [Google Search Central blog: deprecation of the crawl rate limiter tool](https://developers.google.com/search/blog/2023/11/sc-crawl-limiter-byebye)
5. [Google Search Central: how Google interprets robots.txt, supported fields](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
6. [RFC 9309: the Robots Exclusion Protocol, parse limit and caching](https://www.rfc-editor.org/rfc/rfc9309.html)
