Snag Watcher Guides
Sites and robots.txt
| Site | where.sites | match | Gives |
|---|---|---|---|
| Any Shopify store | allbirds.com | url, line, category | price, stock, quantity, every option |
| Any page with schema.org product data | sweetwater.com | url | price, stock |
| GitHub releases | github.com/nodejs/node | url, line | version (major), status, date |
| Nike | nike.com | url, sku | price, stock per size, low stock, release date |
| Amazon | amazon.com | url, sku, exact | price, stock |
| Running Warehouse, Zappos, Brooks, Saucony, On, adidas, New Balance | the store's host | url, some sku or exact | price, stock per size |
| Recreation.gov | a campground URL | listing | open nights. Not available yet |
If a store isn't listed, preview a product URL from it. Many stores run on Shopify or publish schema.org data, and those work without anything from us.
Blocks
We fetch directly, and through a residential proxy only after a site blocks us. We don't solve captchas. A site that blocks 3 runs in a row sends a blocked event, and its conditions stay unknown until it answers again.
robots.txt
We read each host's robots.txt once a day and don't fetch paths it disallows. A check that hits one fails with the issue robots_disallowed.
Some sites disallow the paths that hold the data. We list one of those only after a person on our side has read its terms. Watchers on it must then set:
"fetch": { "accept_site_terms": true }By setting it, you confirm you're allowed to fetch that site under its terms, and that you, not Snag, are responsible for doing so.
| Error | Means |
|---|---|
site_not_reviewed | Nobody has reviewed the site's terms yet |
site_terms_not_accepted | Reviewed, but this watcher didn't set accept_site_terms |
Recreation.gov is waiting on a review. Its robots.txt disallows /api, which is where availability lives.