Zyte demos self-healing Scrapy spiders that repair crawls after site redesigns
Zyte published a blog post detailing two workshop demos from Extract Summit Austin where an agent fixes a Scrapy spider after a site redesign and heals a live crawl mid-run using Spidermon and the Scrapy MCP, including cost measurements and design mistakes.
Briefing
- Zyte: Zyte published a blog post detailing two workshop demos from Extract Summit Austin where an agent fixes a Scrapy spider after a site redesign and heals a live crawl mid-run using Spidermon and the Scrapy MCP, including cost measurements and design mistakes.
Why it matters
This is a practical demonstration of agent-driven resilience in web scraping, showing how AI can reduce the manual maintenance burden of production crawlers. By quantifying costs and sharing design failures, Zyte provides a realistic benchmark for teams considering similar self-healing approaches, though the post is a tutorial writeup, not a product launch.
Sources
- Primary source · Zyte · 2026-10-22 — Self-healing spiders: letting an agent fix your Scrapy crawl, with a lid on it
Watch next
Will Zyte productize this self-healing agent pattern into a managed service or keep it as a reference implementation?