Skip to content
Glossary

robots.txt

A file telling crawlers which paths a site prefers they avoid. A convention, not an access control.

robots.txt is a plain-text file at the root of a domain that states which paths automated clients are asked not to fetch. It is advisory: it expresses a preference and enforces nothing.

#How it reads

User-agent: *
Disallow: /admin/
Allow: /

Sitemap: https://example.com/sitemap.xml

Directives are grouped by user agent, with * as the fallback. The Sitemap line is how crawlers discover a site’s URL index.

#What it is not

  • Not security. Listing a path here advertises it. Anything sensitive needs authentication.
  • Not enforcement. Compliance is voluntary. Major search engines honour it; many other clients do not.
  • Not a licence. Being permitted to fetch a page says nothing about your right to reuse its contents.

#Why it matters when scraping

Ignoring it is technically trivial and reputationally expensive. It is frequently cited in disputes as evidence of intent, and it is often the first thing examined when a site owner objects.

Sources

  1. RFC 9309: Robots Exclusion Protocol

Related terms