A catalog scraper almost never fails loudly. It fails silently, and that's what makes it dangerous.
When you pull product data from retail sources you build for the happy path: fetch the page, match the selectors, extract price, stock, title, images. It works, so you move on. Then the source ships a redesign, or splits one field into two, or starts A/B testing a new layout on half its traffic. Your scraper keeps returning 200. It keeps writing rows. Nothing throws. But the price is now the crossed-out price, or stock reads "in stock" from a cached banner, and your catalog is quietly wrong for days before a human notices a listing that makes no sense.
After years running parsers across many retail sources, I stopped treating extraction as the hard part. Fetching is easy. The real work is trusting what you fetched.
So I build the parser to distrust itself: shape checks on every extraction, ranges each field must fall in, and a diff against the last known-good snapshot per source. A sudden jump in null rates or a price that moved an order of magnitude doesn't get written — it gets quarantined and flagged. I'd rather serve stale data I know is stale than fresh data that's silently corrupt.
For those running scrapers at scale: what's your first line of defense against a source that changes shape without telling you?