Most scraping code written by an AI agent works exactly once. It launches a browser for a page that didn't need one, sleeps for five seconds, zips a list of titles against a list of prices, and silently drops whatever failed. Next week the site renames a class and the dataset quietly fills with nulls.
Web Scraping & Data Extraction teaches your agent to build extractors the way a senior data engineer would: the cheapest reliable source first, a schema before the first request, and a quality report at the end.
Why agent scrapers break
- They reach for a headless browser on static pages.
- They use brittle selectors like
div.flex.mt-2:nth-child(3) > span. - They retry forever, or not at all.
- They treat a 200 response as success even when it is a login page or a bot challenge.
- They lose track of where each value came from.
How the skill works
- Define the data contract — what one record is, every field's type, and how it is validated.
- Pick the source from an eight-level ladder: official API, downloadable file, embedded structured data, feeds and sitemaps, static HTML, client-side network data, rendered DOM, and OCR only as a last resort.
- Crawl politely — per-site rate limits, backoff on 429 and 5xx responses,
Retry-After, conditional requests and caching. - Extract resiliently — semantic selectors with fallbacks, and a warning when a fallback starts doing the work.
- Normalise and validate — dates, currencies, phones and addresses without guessing; failed records go to quarantine instead of disappearing.
- Report — coverage, duplicates, failures, confidence and known gaps, with a source URL and timestamp on every record.
Before a large crawl it samples 10 records, then 100, and checks the distributions — because 98% of prices suddenly reading 0 is a parser failure, not a sale.
Ready for real pipelines
Resumable crawls, incremental re-crawls, change detection, schema-drift alerts and canary checks, plus guidance for Requests, BeautifulSoup, Scrapy and Playwright — including when not to use them. Field recipes cover ecommerce, jobs, business directories, news, property listings, company websites and contacts.
Who it's for
Developers and analysts who use Claude Code or Codex to build datasets, monitor prices or listings, or turn websites and PDFs into clean tables.
What it won't do
It works only with public, authorised or user-provided content. No CAPTCHA solving, no proxy rotation to get around blocks, no bypassing logins, and no reproducing copyrighted articles in full. When a source is blocked, it reports it as unavailable.
