"Ship Reliable RAG: Evals That Grow with Your Retrieval Complexity" - Ravi Vats (PyCon AU 2026)

(Ravi Vats) RAG applications come in many shapes. Some start as simple keyword searches over historical records. Others layer on semantic embeddings, hybrid retrieval (semantic + keyword/lexical searches), reranking, and increasingly sophisticated prompting. But regardless of where your system sits on that spectrum, the question is the same: how do you know it actually works? This talk presents a level-by-level framework for building and evaluating RAG systems in Python, using a real AI-assisted ticket resolution application as the running example. Starting from the simplest possible retrieval setup (lexical matching with BM25 over historical tickets), the talk introduces evaluations early, explains why they matter, and then walks through growing both the RAG application and its eval suite in lockstep through four levels of complexity: lexical retrieval, semantic embedding-based retrieval, hybrid retrieval with log filtering using Drain3, and hybrid retrieval with reranking and context window tuning. At each level, the talk introduces only the evaluation metrics that become necessary at that stage, so the audience builds intuition for which evals matter and when. Along the way, the talk covers practical topics including how to compose simple metrics like faithfulness, context precision, and recall into an experiment harness for tuning reranking strategies (comparing Weighted RRF, plain RRF, Z-score normalization, and Min-Max normalization), and for detecting context rot. At the end, the talk covers how to avoid common pitfalls in LLM-based evaluation such as positional bias and numerical scoring scales. This talk is for Python developers and data scientists who are building or planning to build reliable and robust RAG applications and want a practical mental model for when to introduce which evaluations as their system grows in complexity. https://2026.pycon.org.au/schedule/XWLM8L/ python, pycon, australia, programming, conference, technical, developers, panel, sessions, libraries, frameworks, community, sysadmins, students, education, data, science Videos licensed as CC-BY-NC-SA 4.0 PyCon AU is the national conference for the Python programming community, bringing together professional, student and enthusiast developers, sysadmins and operations folk, students, educators, scientists, statisticians, and many others besides, all with a love for working with Python. Licensed as CC BY-NC-SA - http://creativecommons.org/licenses/by-nc-sa/4.0/ Produced by Next Day Video Australia: https://nextdayvideo.com.au Thu Aug 27 11:30:00 2026 AEST (UTC+10) at Ballroom 3