I Used Claude to Build an Entire Scrapy Project

Here’s my complete end-to-end process for using AI coding agents to build a production-ready Scrapy project. Starting with an empty folder, I use Claude and multiple parallel agents to scaffold the project, define a shared item schema, generate page objects and selectors for two different domains, build monitoring with Spidermon, and create a universal spider that saves the extracted data to SQLite. I also explain how my Scrapy Open Spec captures my preferred project structure, how Scrapy-Poet separates extraction logic from crawling logic, and why I retrieve fixtures through the actual Scrapy project instead of using an unrelated browser or proxy setup. Finally, we look at how field-level extraction functions, automated tests, and monitoring can become the foundation of a self-healing scraping system—allowing an agent to identify and repair broken selectors with minimal intervention. Links: Scrapy Open Spec: https://github.com/zytelabs/scrapy-openspec Scrapy agent skills: https://github.com/zytedata/skills Try Zyte API with free credit: https://www.zyte.com/zyte-api/ Join the community: https://www.zyte.com/join-community/ Chapters: 00:00 Building a complete Scrapy project with AI 00:14 Setting up the Python environment 00:42 Introducing the Scrapy Open Spec 01:19 Scaffolding the project with Claude 01:53 Managing the multi-agent workspace 02:20 Reviewing the generated project structure 03:16 Creating the shared item schema 04:03 Separating crawling and extraction with Scrapy-Poet 04:57 Generating the page objects 06:03 Adding data-quality monitoring with Spidermon 07:30 Adding a second website in parallel 09:20 Reviewing the generated selectors 10:28 How I divide work between agents 10:59 Stopping and repairing a failing spider 11:50 Why fixtures must be fetched through Scrapy 13:40 Building the universal spider 15:14 Designing for self-healing selectors 17:18 How the domain-agnostic spider works 18:42 Running the completed crawl 19:30 Results, monitoring, and recap 20:10 Final thoughts and resources #Scrapy #WebScraping #ClaudeCode #AIAgents #Python #Zyte #BrowserAutomation