How to Build Self-Healing Scrapers with Claude
Can an AI agent detect and repair a broken web scraper automatically? In this video, I build a complete Scrapy project with Claude Code, intentionally break one of its selectors, detect the failure using Spidermon, and create a rough self-healing agent that finds a replacement selector and updates the page object. The project begins in an empty folder and is generated using my personal Scrapy specification. I explain how a living spec differs from an agent skill, why I separate extraction logic from crawling logic with Scrapy-Poet, and how saved HTML fixtures make selector generation repeatable and testable. Once the spider is running, we add field-coverage monitoring for the product name. I then deliberately break its selector, use the resulting monitor output to identify the failed field, download fresh HTML, and ask Claude to generate and test a replacement. The first attempt isn’t flawless—it introduces a formatting error—but it successfully repairs the selector. This demonstrates the architecture needed to build more reliable, production-ready self-healing scrapers. Links: Scrapy agent skills: https://www.scrapy.org/build-with-ai My Scrapy specification: https://github.com/zytelabs/scrapy-openspec Scrapy Extensions: https://www.scrapy.org/resources/extensions Try Zyte API: https://www.zyte.com/zyte-api/ Chapters: 00:00 Building an agentic scraping workflow 00:15 Setting up the empty project 01:09 Adding the Scrapy specification 01:33 Specifications versus agent skills 02:34 Separating extraction with page objects 04:36 Claude begins generating the project 05:48 Reviewing the generated architecture 07:04 How the page objects are structured 08:03 Saving and cleaning HTML fixtures 09:20 Generating selectors with subagents 10:32 Automatically testing the selectors 11:08 The completed Scrapy project 12:02 Running the generated spider 13:06 What does “self-healing” mean? 14:00 Preparing the project for automatic repairs 15:11 Adding field-coverage monitoring 18:21 Deliberately breaking a selector 19:39 Spidermon detects the failure 20:17 Building the self-healing agent 22:05 Why periodic monitoring matters 23:36 Running the repair agent 24:24 The first repair attempt 24:47 Claude fixes the broken selector 25:18 Results and final recap #Scrapy #ClaudeCode #WebScraping #AIAgents #Python #Automation #Zyte