---
title: "robots.txt"
url: https://proxy.wiki/glossary/robots-txt/
type: Glossary Term
author: "proxy.wiki editorial"
published: 2026-08-19
updated: 2026-08-19
site: proxy.wiki
topics: ["Proxy fundamentals"]
license: CC BY 4.0 — quote freely with attribution to https://proxy.wiki/
---

# robots.txt

> A file telling crawlers which paths a site prefers they avoid. A convention, not an access control.

**`robots.txt` is a plain-text file at the root of a domain that states which paths automated clients are asked not to fetch.** It is advisory: it expresses a preference and enforces nothing.

## How it reads

```
User-agent: *
Disallow: /admin/
Allow: /

Sitemap: https://example.com/sitemap.xml
```

Directives are grouped by user agent, with `*` as the fallback. The `Sitemap` line is how crawlers discover a site’s URL index.

## What it is not

- **Not security.** Listing a path here advertises it. Anything sensitive needs authentication.

- **Not enforcement.** Compliance is voluntary. Major search engines honour it; many other clients do not.

- **Not a licence.** Being permitted to fetch a page says nothing about your right to reuse its contents.

## Why it matters when scraping

Ignoring it is technically trivial and reputationally expensive. It is frequently cited in disputes as evidence of intent, and it is often the first thing examined when a site owner objects.

## Sources

1. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)