How we built the data engine behind PriceWho.
We built the distributed crawling and data platform behind PriceWho's product comparison engine — covering 50+ retailers. PriceWho has since been acquired.
- Client
- PriceWho
- Industry
- E-commerce / price comparison
- Services
- Data engineering, scraping, analytics
- Status
- Delivered · product since acquired
- Stack
- Scrapy, Scrapyd, Zyte, BigQuery, AWS
Fresh prices, at scale, without breakage.
PriceWho's promise is simple: search one product, see every store's price. Delivering it means crawling a large and constantly changing set of retailers, each with its own layout, anti-bot measures and product naming.
A crawling platform, not a pile of scripts.
- 01
One spider framework for every retailer
Shared base spiders and item pipelines, so adding a store is configuration and selectors rather than a new codebase.
- 02
Distributed scheduling on Scrapyd
Crawls are spread across workers and scheduled by priority, so high-demand products refresh most often.
- 03
Managed access with Zyte
Proxy rotation and ban handling keep success rates stable without hand-tuning each site.
- 04
Validated data into BigQuery
Normalised prices, currencies and product matches land in BigQuery, powering the comparison engine and internal dashboards.
The outcome.
“We've worked with hundreds of vendors, and ThundermindTech stands out among the best. Exceptional communication and top-tier professionalism — highly recommended.”