I Asked Claude and Codex to Build a Web Scraper

I asked Claude and Codex to build the same Scrapy project. Both spiders ran and successfully extracted the data—but both introduced architectural and maintenance problems that would become painful in production. In this video, I compare the generated projects and examine their overloaded spiders, inconsistent settings, outdated item structures, mixed extraction logic, and lack of reliable guardrails. I then compare them with a project generated using purpose-built agent skills. This version uses Scrapy-Poet page objects, modern data classes, reusable item schemas, saved HTML fixtures, and automated selector tests. The important lesson isn’t that AI can’t build web scrapers. It can. The challenge is getting AI to build them consistently, maintainably, and in a way that can scale beyond a single website. With the right architecture, fixtures, tests, and agent instructions, we can even begin building agentic systems that detect and repair broken selectors automatically. Links: Install the Zyte agent skills: npx skills add zytedata/skills Zyte agent skills repository: https://github.com/zytedata/skills Zyte Community MCP: https://github.com/zytelabs/zyte-community-mcp Learn more about Scrapy-Poet: https://www.scrapy.org/resources/extensions Deploy with Scrapy Cloud: https://www.scrapy.org/deploy Chapters: 00:00 Claude vs Codex: comparing the generated projects 00:18 The problem with overloaded spiders 01:13 Better alternatives to traditional Scrapy Items 02:04 Inconsistent AI-generated settings 02:54 How agents access websites when building spiders 04:05 Building with purpose-made agent skills 05:03 Reviewing the improved project 05:17 Separating logic with Scrapy-Poet 06:07 Page objects and maintainable selectors 07:28 Modern items and data classes 08:05 Automatically generated fixtures and tests 08:58 Building agentic selector repair 09:24 Why AI-generated spiders need guardrails #Scrapy #ClaudeCode #Codex #WebScraping #AIAgents #Python #Zyte