Overview
How an Engineering Team Reduced Scrape Breakages from Weekly to Rare
RelayHQ's platform team was fixing a broken selector almost every week. Downstream teams stopped trusting the dataset because it silently went stale between fixes.
The old workflow
A cron job ran a scraper against a handful of sites. When a site redesigned its markup, the job kept running and kept returning empty or garbage fields—nobody noticed until a downstream report looked wrong.
What changed
Scheduled runs now retry with a real browser when a product page comes back without commerce signals, and fail explicitly instead of storing junk when a soft-block or bot-fallback page is detected. Failures show up as a job status, not a silent empty row.
Stack
- Scheduled crawls with automatic Playwright fallback for JS-rendered pages
- Soft-block / bot-fallback detection on every product-detail fetch
- Slack alerts on job failures instead of silent empty exports
Ready to try Scrapliy?
Turn competitor research into live catalogs, change alerts, and export-ready feeds.