The hardest test suite I ever built - a pytest case study - Grzegorz Kocjan

[EuroPython 2026 - S2 on 2026-07-15] 🎤 *The hardest test suite I ever built - a pytest case study by Grzegorz Kocjan* 🔗 https://ep2026.europython.eu/session/the-hardest-test-suite-i-ever-built-a-pytest-case-study 📝 Abstract: For years, this real-time video system had no tests. Every change produced unpredictable side effects. Accuracy drifted. Production incidents and escalations followed. The only “verification” was manual inspection and hope. When I joined the project, this was the reality - and building a proper integration test suite became my first priority. In this talk, I’ll share how I designed and evolved the hardest integration test suite of my career using pytest - and kept it readable. The system processed live streams in production. It was non-deterministic. Individual detections were only 80–90% accurate. For testing, we replayed recorded scenarios to make system behavior observable and comparable across runs. But binary assertions were not enough. A single failed event did not mean the whole system was broken - but we needed a way to measure when it actually was. Instead of writing one massive test, I built a layered architecture: - dual parametrization - recording scope and event scope - orchestration in fixtures - assertions in tiny, single-purpose tests - statistics collection during execution - end-of-run aggregation that summarizes system accuracy The result was a suite that could detect regressions in model changes and produce reproducible evidence - HTML reports, structured dumps, and a summary statistics file. This talk explores how far pytest can be stretched beyond unit tests - into a framework for architecting complex integration systems. --- This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License: https://creativecommons.org/licenses/by-nc-sa/4.0/