---
title: "What is web scraping?"
url: https://proxy.wiki/glossary/web-scraping/
type: Glossary Term
author: "proxy.wiki editorial"
published: 2026-08-19
updated: 2026-10-21
site: proxy.wiki
topics: ["Web scraping"]
license: CC BY 4.0; quote freely with attribution to https://proxy.wiki/
---

# What is web scraping?

> Using software to fetch web pages and turn them into structured records. The hard part is knowing the page you got is the page you wanted.

A scraper does four things: fetch a page, pull out the fields you want, check that the result is real, and store it with the time it was captured. Fetching is the step people worry about. Checking is the step that fails without anyone noticing, so it is where most of the thought should go.

## A minimal scraper that checks what it got

```
# Python 3.9.6, requests 2.32.3. Fetch, parse, validate, then store.
import sys, json, hashlib, datetime
from html.parser import HTMLParser
import requests

PROXIES = None  # e.g. {"https": "http://user:pass@gw.example.net:8000"}

class Title(HTMLParser):
    def __init__(self):
        super().__init__(); self.inside = False; self.text = ""
    def handle_starttag(self, tag, attrs): self.inside = tag == "title"
    def handle_endtag(self, tag): self.inside = False
    def handle_data(self, data):
        if self.inside: self.text += data

def scrape(url, must_contain):
    r = requests.get(url, proxies=PROXIES, timeout=15)
    p = Title(); p.feed(r.text)
    record = {"url": url, "status": r.status_code, "bytes": len(r.content),
              "title": p.text.strip(),
              "sha256": hashlib.sha256(r.content).hexdigest()[:16],
              "fetched": datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds")}
    # Validate content, not transport: a 200 can still be a block page.
    record["accepted"] = r.status_code == 200 and must_contain in r.text
    return record

for url, marker in [("https://example.com/", "Example Domain"),
                    ("https://www.etsy.com/", "Etsy")]:
    print(json.dumps(scrape(url, marker)))
```

Run from one consumer connection, it printed:

```
{"url": "https://example.com/", "status": 200, "bytes": 577, "title": "Example Domain", "sha256": "25ddf2c883e0d195", "fetched": "2026-10-21T19:26:09+00:00", "accepted": true}
{"url": "https://www.etsy.com/", "status": 403, "bytes": 776, "title": "etsy.com", "sha256": "1953037b1621a7e4", "fetched": "2026-10-21T19:26:09+00:00", "accepted": false}
```

The second line is the point. Etsy answered with a 776-byte page titled with its own domain name, served by an [anti-bot system](/glossary/anti-bot-system/). A scraper that stored every response with a title would have filed it as Etsy’s home page. Here the status code would have caught it too. The marker check is for the cases where it would not: a challenge served with 200, a consent wall where the content should be, a layout change that leaves every field empty. The `fetched` timestamp and the content hash make the record traceable later, which matters the first time someone asks where a number came from.

## Where a pipeline breaks without an error

| Step | How it fails silently | The check that catches it |
| --- | --- | --- |
| Fetch | A block or challenge page arrives instead of content, sometimes with status 200 | Require a marker only the real page contains |
| Parse | A changed selector returns empty fields, not an exception | Reject records missing a field you need |
| Validate | Skipped, so empty records are stored as successes | Count accepted records, not responses |
| Store | Today’s empty run overwrites yesterday’s good data | Compare each run’s record count with the last before replacing anything |

Every row produces a job that reports success. That is why a [success rate](/glossary/success-rate/) measured on status codes can look healthy while the pipeline collects nothing. [Proxies for RAG pipelines](/guides/proxies-for-rag-pipelines/) shows the same failure in a retrieval corpus, with per-source checks that catch a source changing shape.

## Before the first request

- **Look for an API or a JSON endpoint.** Many pages fetch their data from one. It is smaller, steadier and often permitted outright.

- **Read [robots.txt](/glossary/robots-txt/) and the terms.** They state the operator’s position, and ignoring them is easy to prove later.

- **Decide what you will not collect.** Personal data brings data-protection law into scope however public the page is.

- **Pace it per host.** [Rate limits](/glossary/rate-limiting/) are counted per host and per client, so spread the work across hosts and keep each one slow.

## Where proxies fit

A [proxy](/glossary/proxy-server/) changes the address the site sees and nothing else. It helps with geography and with limits counted per address. It does not change your client’s [TLS handshake](/glossary/tls-fingerprinting/) or its headers, which is why a scraper blocked for its client stays blocked through any proxy. The fetch side for Python is in [Using proxies in Python](/guides/proxies-in-python-requests-httpx/), choosing a proxy type in [Residential vs datacenter vs ISP vs mobile](/guides/residential-vs-datacenter-proxies/), and crawling at scale, with provenance and deduplication, in [Collecting training data for a language model](/guides/collecting-llm-training-data/).

## Is it legal?

It depends on the jurisdiction, on what is collected, on whether access controls are bypassed and on what is done with the data afterwards. The United States case cited most often is _hiQ Labs v. LinkedIn_, in which the Ninth Circuit affirmed a preliminary injunction in hiQ’s favour, finding serious questions about whether scraping publicly available profiles was access “without authorization” under the Computer Fraud and Abuse Act. That was one appeals court in one country, ruling at an early stage of the case, and it did not settle contract, copyright or data-protection questions. Take advice for anything at scale.

## Frequently asked questions

### Is web scraping legal?

It depends on the jurisdiction, what you collect and how you get access. Public pages, personal data and anything behind a login are treated differently. In the US the case cited most often on public pages is hiQ Labs v. LinkedIn in the Ninth Circuit. Take advice for anything at scale.

### How do I stop a scraper silently collecting nothing?

Validate the content, not the status code. Require a marker or field you need before accepting a record, and compare each run's record count with the previous run before you replace stored data.

### Do I need a proxy to scrape?

Not always. Many targets work from one address at a modest rate. A proxy helps with geography and with limits counted per address; it does not fix a client that is blocked for its handshake or headers.

## Sources

1. [hiQ Labs v. LinkedIn, US Court of Appeals for the Ninth Circuit, opinion on remand](https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/17-16783.pdf)
2. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)
3. [RFC 9110: HTTP Semantics, 403 Forbidden](https://www.rfc-editor.org/rfc/rfc9110.html#name-403-forbidden)
