How we collect data from the public web, and what goes wrong when you do it badly.
Modern sites build their pages in your browser, so a plain download comes back empty. How to tell whether your target site works that way, the five ways to get the data out, what each one costs you, and what a custom build actually includes.
Seven firms priced the web scraping market in 2025, from USD 501.9 million to USD 3.2 billion. Three more priced the AI slice higher than the whole thing. Why they disagree, what the vendor lists give away, and how to pick a figure you can defend.
Customers asked to run the collector themselves, so we built them a tool. Most are on a scheduled extract now. What we got wrong about the control they wanted.
Most "best tools" pages are written by a company in their own ranking, and three sites calling themselves independent benchmarks are owned by the provider they place first. Here are ten tools by the job you'd hire them for, and a test you can run yourself.