Latest AI and tech news
Rate limiting sounds like a speed limit, but 93.7% of rate-limited sites block on the very first request. It is an identity check, not a throttle.
Zyte MCP brings live web access, structured extraction and Scrapy Cloud operations into coding agents such as Claude Code, Cursor and Codex.
The famously awkward security mechanism has quietly become an invisible behavioral scoring system. Two vendors enable 89% of CAPTCHAs, and the smallest sites use it most.
Recordings, slides, and a rundown of Zyte's second Developer Community Meetup with Humanbound: a live prompt injection attack on a price agent, Zyte CDP managed browsers, and the launch of harness-run.
Recordings, slides, and a rundown of Zyte's second Developer Community Meetup with Humanbound: a live prompt injection attack on a price agent, Zyte CDP managed browsers, and the launch of harness-run.
Scrapy MCP connects an AI agent to a running Scrapy crawl, using the new Remote Control extension in Scrapy 2.19 to let the agent discover jobs, check status and run Python against the live crawler, all without restarting it.
One Scrapy spider, three browser setups: stock Playwright, Patchright via the new PLAYWRIGHT_BROWSER_PROVIDER hook, and a remote browser on Zyte over CDP. Same spider and selectors throughout, only the browser changes.
A Scrapy pipeline that asks a fast, calibrated AI model whether each scraped field still looks real, and stops the crawl when too many don't. What it caught, what it misses, and whether building it was worth it.
Jev by TypeFace AI cannot generate a string, so it cannot extract a field. What it is, how it differs from an LLM, and the one job it earns in a scraping pipeline.
Only 18.5% of top sites run dedicated antibot, but every one of them chose to. Why it's the most intentional barrier in the stack, and the strongest signal of a hardened site.
uv 0.12 changed several defaults that Python developers rely on—from how `uv init` structures projects to how hash checking and lockfiles behave. This practical cheatsheet tests the differences across versions and highlights the command-line traps most likely to surprise you.
Web Application Firewalls run on 92.4% of the top websites, but most arrived bundled with a CDN and were never tuned to resist automated access.
What if your AI agent could use a real browser to search, click, compare products, and collect data, without you writing a selector for every step? By connecting it to a remote Zyte CDP browser, you can turn plain-language instructions into practical browser automation.
It was just a silly game side project. But, as I grew my sim to 3,000 matches, I levelled up on edge routing, flattening memory spikes and hardcore monitoring.
Real web is becoming hostile for AI Agents. The page your agent scrapes now could be a potential attack surface. Read more and join Zyte's virtual meet-up to see it in action.
Nearly four in 10 of the world's top sites are now closed to well-behaved AI crawlers. Here's what operators actually do with robots.txt, and which AI agents they block or welcome.
Nearly four in 10 of the world's top sites are now closed to well-behaved AI crawlers. Here's what operators actually do with robots.txt, and which AI agents they block or welcome.
A personal take on Claude Fable 5.1 and GLM-5.3-Flash: real benchmarks, a live extraction test, and the model I guessed before Zhipu confirmed it.
A personal take on Claude Fable 5.1 and GLM-5.3-Flash: real benchmarks, a live extraction test, and the model I guessed before Zhipu confirmed it.
A personal take on Claude Fable 5.1 and GLM-5.3-Flash: real benchmarks, a live extraction test, and the model I guessed before Zhipu confirmed it.
Want to use traditional browser automation frameworks without infrastructure headaches? Tap our new CDP support to run scripts on Zyte’s powerful infrastructure.
New guidelines on generative AI scraping aim to protect user privacy. But turning a rudimentary, 30-year-old web server standard into a legal barrier will disenfranchise users and create accidental monopolies.
New guidelines on generative AI scraping aim to protect user privacy. But turning a rudimentary, 30-year-old web server standard into a legal barrier will disenfranchise users and create accidental monopolies.
New guidelines on generative AI scraping aim to protect user privacy. But turning a rudimentary, 30-year-old web server standard into a legal barrier will disenfranchise users and create accidental monopolies.
Zyte's State of Web Access research finds new barriers making web scraping more difficult. We interview the researcher who says that doesn't mean the opportunity is over.
Domagoj Marić explores how AI, web scraping, and OSINT turn scattered personal data into profiles, scams, and security risks at Extract Summit.