---
title: "Web scraping"
url: https://proxy.wiki/glossary/web-scraping/
type: Glossary Term
author: "proxy.wiki editorial"
published: 2026-08-19
updated: 2026-08-19
site: proxy.wiki
topics: ["Proxy fundamentals"]
license: CC BY 4.0 — quote freely with attribution to https://proxy.wiki/
---

# Web scraping

> Extracting structured data from web pages automatically. Lawful in many contexts and restricted in others.

**Web scraping is the automated extraction of data from websites.** A scraper fetches pages and parses the response into structured records.

## The usual pipeline

- **Fetch** — retrieve the page, often through a [proxy](/glossary/proxy-server/).

- **Render** — only if the content requires JavaScript; a [headless browser](/glossary/headless-browser/) is expensive and often unnecessary.

- **Parse** — extract fields from the markup or an underlying JSON response.

- **Validate** — confirm you received real content rather than a block page.

- **Store** — persist, deduplicate, and record when each record was captured.

## The step most often skipped

Validation. Many [anti-bot systems](/glossary/anti-bot-system/) answer with 200 and an empty or decoy payload. A pipeline that checks only the status code will happily record thousands of empty results and report a high [success rate](/glossary/success-rate/) while collecting nothing.

## Legal note

Legality depends on jurisdiction, on what is collected, on whether access controls are bypassed, and on what is done with the result afterwards. Personal data brings data-protection law into scope regardless of how public the page is. This is not legal advice.