​Building Trustworthy, High-Quality AI Agents with MLflow

Building trustworthy, high-quality agents remains one of the hardest problems in AI today. Even as coding assistants automate parts of the development workflow, evaluating, observing, and improving agent quality is still manual, subjective, and time-consuming. Teams often spend hours “vibe checking” agents, labeling outputs, and debugging failures. In this session, Corey Zumar, Staff Software Engineer at Databricks, demonstrates how to use MLflow to automate and accelerate agent observability. Learn how to apply proven patterns to deliver agents that behave reliably in real-world conditions. Key Takeaways and Learnings: ​🔹 Understand the development lifecycle of Agent development for better observability ​🔹 Use MLflow key components along the development lifecycle to enhance general observability: tracking and debugging, evaluation with MLflow judges, and a prompt registry for versioning ​🔹 Select appropriately from a suite of over 60+ built-in and custom MLflow judges for evaluation, and use Judge Builder for automatic evaluation. ​🔹 Use MLflow UI to compare and comprehend evaluation scores and metrics 🗓️ Date: February 17, 2026 0:00 Introduction to MLflow and Agent Development Challenges 4:01 The MLflow Agent Development Lifecycle and Components 7:05 Observability, Feedback, and Issue Identification 13:18 Automated Evaluation with LLM Judges 18:21 Prompt Optimization and Fix Verification 23:39 Production Deployment and Continuous Monitoring 27:00 MLflow AI Gateway and Future Roadmap 30:19 Q&A