Skip to content
Glossary

Web scraping

Extracting structured data from web pages automatically. Lawful in many contexts and restricted in others.

Web scraping is the automated extraction of data from websites. A scraper fetches pages and parses the response into structured records.

#The usual pipeline

  1. Fetch — retrieve the page, often through a proxy.
  2. Render — only if the content requires JavaScript; a headless browser is expensive and often unnecessary.
  3. Parse — extract fields from the markup or an underlying JSON response.
  4. Validate — confirm you received real content rather than a block page.
  5. Store — persist, deduplicate, and record when each record was captured.

#The step most often skipped

Validation. Many anti-bot systems answer with 200 and an empty or decoy payload. A pipeline that checks only the status code will happily record thousands of empty results and report a high success rate while collecting nothing.

Legality depends on jurisdiction, on what is collected, on whether access controls are bypassed, and on what is done with the result afterwards. Personal data brings data-protection law into scope regardless of how public the page is. This is not legal advice.

Related terms