Most Pages Are Easy to Scrape. So What’s the Catch?

Is the web really closing up? We measured it. Everyone in scraping says the same thing: sites are getting harder to access programmatically. Nobody says how much harder, or where. So we ran the scan. https://www.zyte.com/blog/sowa-interview-theresia-tanzil/ Two numbers came back that we didn't expect. Anti-bot adoption sits at 18.5%. And 87.9% of the landing pages we recorded were simple or easy to access. The web is not uniformly locked down. But the pages that are hard cluster in the sectors where the data is worth the most, which changes what the question should be. Not "is scraping getting harder," but "can I tell how hard this project is before I pick an architecture?" Neha talks to Theresia, who worked on the research behind State of Web Access 2026. She selected and scanned the sites, read the results, and did the unglamorous work of turning a messy web into a data set the team was willing to make claims from. This conversation is about the method, not the charts. How you pick sites that stand in for the whole web, how you decide a barrier is genuinely present rather than a fluke, and what you do when the data comes back a mess. If you build scrapers, the method is the part you can steal. 📊 Read the full report: https://www.zyte.com/sowa/2026/ 🔔 Subscribe for more on web data infrastructure #webscraping #dataengineering #antibot #webdata 0:00 The headline numbers 1:12 Meet Theresia and the State of Web Access 2026 report 2:17 Why build this report at all 3:25 Deciding what "the web" even means 5:35 When one domain shows up in five categories 6:47 Why the scan stopped at the landing page 7:32 The seven checks, from TLS fingerprinting to robots.txt 9:02 One spider or many? Architecting the scanner 10:30 Turning a CLI tool into a batched Scrapy job 12:05 The check that got cut 13:06 Why we deliberately did not use Zyte API 14:41 Mapping every page to a difficulty tier 15:51 When is a barrier actually a barrier? 18:11 CAPTCHA, JavaScript and rate limiting are not equal signals 19:04 Testing rate limits without distorting the result 21:03 The finding nobody had a hypothesis for 22:06 What this means if you scrape for a living 23:38 Scraping at scale is a system, not a script 24:19 What AI crawlers will do to access 26:27 The belief this project broke 28:12 TL;DR: the web is being priced up