---
title: "Crawl-delay in robots.txt: what it means and who honors it"
url: https://proxy.wiki/glossary/crawl-delay/
type: Glossary Term
author: "proxy.wiki editorial"
published: 2026-10-21
updated: 2026-10-21
site: proxy.wiki
topics: ["Proxy fundamentals"]
license: CC BY 4.0; quote freely with attribution to https://proxy.wiki/
---

# Crawl-delay in robots.txt: what it means and who honors it

> Crawl-delay asks a crawler to wait between requests, but it is not standard and Google ignores it. Learn the safer controls.

## Key takeaways

- Crawl-delay is a crawler convention, not an RFC 9309 rule.
- Google says it ignores Crawl-delay.
- Enforce request rates with queueing, Retry-After and server controls.

**`Crawl-delay` is a convention that asks a crawler to wait between requests.** It is not part of the standard Robots Exclusion Protocol, and Google says its crawlers do not support it. A crawler that chooses to implement the convention may read the number as seconds, but the meaning is not guaranteed across bots.

## Where the line came from

The current standard is [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html). Its robots file grammar defines `User-agent`, `Allow` and `Disallow` rules, with `Sitemap` supported by major search engines. It does not define `Crawl-delay`. The line is still accepted as ordinary text, so a file can contain it without becoming invalid, but a client has no standards-based obligation to act on it.

Google’s specification page is explicit: `crawl-delay` and other fields outside its supported list are ignored. That means adding `Crawl-delay: 10` will not reduce Googlebot’s rate. If Googlebot is using too much capacity, use the Search Console crawl-rate controls where available, fix the slow response path, or return useful caching headers. A robots file is access policy, not a bandwidth governor.

## What the line looks like

```
User-agent: *
Disallow: /private
Crawl-delay: 7
```

We served that exact file from a local Python HTTP server and fetched it with curl. The command ran on Python 3.15.0 and curl 8.7.1 on 21 October 2026. The capture proves delivery, not that a crawler will wait:

```
$ curl -sS -i http://127.0.0.1:57006/robots.txt
HTTP/1.0 200 OK
Server: BaseHTTP Python/3.15.0
Content-Type: text/plain
Content-Length: 48

User-agent: *
Disallow: /private
Crawl-delay: 7
exit=0
```

The port is assigned by the operating system and changes between runs. The important observation is that an HTTP client retrieves the line as text. Interpretation happens later, inside the crawler. The site’s own [robots.txt reference](/glossary/robots-txt/) covers matching rules and scope.

## Which clients may use it

| Client or control | What happens to Crawl-delay | Use when |
| --- | --- | --- |
| Google crawlers | Ignored, according to Google’s supported-field list | You need Google crawl-rate changes |
| A crawler that documents the extension | May wait the documented number of seconds | You control that crawler’s policy |
| Your own worker queue | Can enforce a precise delay and concurrency limit | You need predictable load |
| HTTP status and response headers | 429 and Retry-After can signal backoff | The server needs to slow a client now |

Do not use a delay as an access-control mechanism. A client can ignore it, and a malicious client can ignore the whole robots file. Authentication, rate limiting and network controls belong at the application or edge. The [rate-limiting entry](/glossary/rate-limiting/) explains the enforcement side, while [403, 407 and 429 troubleshooting](/guides/403-vs-407-vs-429/) shows what a caller sees.

## How to make a polite crawler

If you operate the crawler, parse the line only as an opt-in extension and document your unit. A simple implementation can keep the next-allowed timestamp per host, add a bounded worker count, and honour `Retry-After` when a server sends it. Do not sleep after every request globally: that makes unrelated hosts wait for one another. Store the delay per origin and keep a timeout around the fetch. The [sticky-session guide](/guides/rotating-vs-sticky-sessions/) is useful when the crawler also needs one proxy exit for a sequence.

If you operate the site, publish the line only if the crawlers you care about document support for it. Keep the authoritative rules in `Allow` and `Disallow`, link the sitemap, and measure origin load. A delay that looks conservative can still be too fast for a small backend or too slow for a large one.

Measure the delay from the time a response is available, not from the start of a batch. A retry after a timeout should use explicit backoff rather than waiting only for the next crawl slot. Keep per-host state keyed by the normalized origin, and do not share one timer across unrelated domains. Record the parsed value, chosen unit, and next permitted request so an operator can explain why a fetch was held.

## Frequently asked questions

### Does Googlebot honor Crawl-delay?

No. Google's robots.txt specification says that crawl-delay is not supported. Use Search Console controls and server-side rate limiting when Googlebot needs a different pace.

### Is Crawl-delay part of robots.txt?

It is a widely used extension, but RFC 9309 does not define it. A crawler may implement it, so follow that crawler's own documentation.

### Can Crawl-delay stop a scraper?

No. It is advisory text. A scraper can ignore it. Enforce limits with authentication, rate limiting, response controls and monitoring.

## Sources

1. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)
2. [Google Search Central: How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)
3. [RFC 9110: Retry-After response header](https://www.rfc-editor.org/rfc/rfc9110.html#section-10.2.3)
