What is web scraping?
Definition
Using software to fetch web pages and turn them into structured records. The hard part is knowing the page you got is the page you wanted.
A scraper does four things: fetch a page, pull out the fields you want, check that the result is real, and store it with the time it was captured. Fetching is the step people worry about. Checking is the step that fails without anyone noticing, so it is where most of the thought should go.
A minimal scraper that checks what it got
# Python 3.9.6, requests 2.32.3. Fetch, parse, validate, then store.
import sys, json, hashlib, datetime
from html.parser import HTMLParser
import requests
PROXIES = None # e.g. {"https": "http://user:pass@gw.example.net:8000"}
class Title(HTMLParser):
def __init__(self):
super().__init__(); self.inside = False; self.text = ""
def handle_starttag(self, tag, attrs): self.inside = tag == "title"
def handle_endtag(self, tag): self.inside = False
def handle_data(self, data):
if self.inside: self.text += data
def scrape(url, must_contain):
r = requests.get(url, proxies=PROXIES, timeout=15)
p = Title(); p.feed(r.text)
record = {"url": url, "status": r.status_code, "bytes": len(r.content),
"title": p.text.strip(),
"sha256": hashlib.sha256(r.content).hexdigest()[:16],
"fetched": datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds")}
# Validate content, not transport: a 200 can still be a block page.
record["accepted"] = r.status_code == 200 and must_contain in r.text
return record
for url, marker in [("https://example.com/", "Example Domain"),
("https://www.etsy.com/", "Etsy")]:
print(json.dumps(scrape(url, marker)))
Run from one consumer connection, it printed:
{"url": "https://example.com/", "status": 200, "bytes": 577, "title": "Example Domain", "sha256": "25ddf2c883e0d195", "fetched": "2026-10-21T19:26:09+00:00", "accepted": true}
{"url": "https://www.etsy.com/", "status": 403, "bytes": 776, "title": "etsy.com", "sha256": "1953037b1621a7e4", "fetched": "2026-10-21T19:26:09+00:00", "accepted": false}
The second line is the point. Etsy answered with a 776-byte page titled with its own domain name, served by an anti-bot system. A scraper that stored every response with a title would have filed it as Etsy’s home page. Here the status code would have caught it too. The marker check is for the cases where it would not: a challenge served with 200, a consent wall where the content should be, a layout change that leaves every field empty. The fetched timestamp and the content hash make the record traceable later, which matters the first time someone asks where a number came from.
Where a pipeline breaks without an error
| Step | How it fails silently | The check that catches it |
|---|---|---|
| Fetch | A block or challenge page arrives instead of content, sometimes with status 200 | Require a marker only the real page contains |
| Parse | A changed selector returns empty fields, not an exception | Reject records missing a field you need |
| Validate | Skipped, so empty records are stored as successes | Count accepted records, not responses |
| Store | Today’s empty run overwrites yesterday’s good data | Compare each run’s record count with the last before replacing anything |
Every row produces a job that reports success. That is why a success rate measured on status codes can look healthy while the pipeline collects nothing. Proxies for RAG pipelines shows the same failure in a retrieval corpus, with per-source checks that catch a source changing shape.
Before the first request
- Look for an API or a JSON endpoint. Many pages fetch their data from one. It is smaller, steadier and often permitted outright.
- Read robots.txt and the terms. They state the operator’s position, and ignoring them is easy to prove later.
- Decide what you will not collect. Personal data brings data-protection law into scope however public the page is.
- Pace it per host. Rate limits are counted per host and per client, so spread the work across hosts and keep each one slow.
Where proxies fit
A proxy changes the address the site sees and nothing else. It helps with geography and with limits counted per address. It does not change your client’s TLS handshake or its headers, which is why a scraper blocked for its client stays blocked through any proxy. The fetch side for Python is in Using proxies in Python, choosing a proxy type in Residential vs datacenter vs ISP vs mobile, and crawling at scale, with provenance and deduplication, in Collecting training data for a language model.
Is it legal?
It depends on the jurisdiction, on what is collected, on whether access controls are bypassed and on what is done with the data afterwards. The United States case cited most often is hiQ Labs v. LinkedIn, in which the Ninth Circuit affirmed a preliminary injunction in hiQ’s favour, finding serious questions about whether scraping publicly available profiles was access “without authorization” under the Computer Fraud and Abuse Act. That was one appeals court in one country, ruling at an early stage of the case, and it did not settle contract, copyright or data-protection questions. Take advice for anything at scale.