companies of all sizes
Detecting soft-blocks and bot-served fallback content
Some sites serve generic fallback content to anything that looks like a bot instead of a real product page—here's how that gets caught.
Updated 2026-04-20 · Reliability · 6 min read
Step-by-step walkthrough
The failure mode
A product URL returns HTTP 200 with a full page—but it's the same generic collection/placeholder content Shopify or Cloudflare serves to suspected bots, not the real product. A naive scraper stores this as if it were valid data.
How Scrapliy catches it
Product-detail URLs that come back without commerce signals (price, SKU, GTIN, JSON-LD Product type) trigger a forced re-fetch with a real browser. If the content still lacks commerce signals, or a hash-collision check shows identical content served under a different product slug, the job fails explicitly instead of silently storing junk.
- Layer 1: PDP without commerce signals -> retry with Playwright
- Layer 2: hash collision across different slugs -> retry then fail
- Layer 3: still no signals after retry -> explicit failure, not silent bad data
Ready to set this up for your store?
Add a site, pick a crawl schedule, and Scrapliy handles discovery, extraction, and change detection from there.
Related: Reliability → · Compare tools · Blog · All guides