MLflow Agent Evaluation: Judges, Scorers & Multi-Turn Sessions (Notebook 1.7)

In the seventh tutorial of the Mastering MLflow for GenAI series, Jules Damji goes beyond traces to evaluate GenAI agents, using LLM-as-a-judge patterns, built-in and deterministic scorers, custom judges, third-party metrics, and multi-turn session checks, so quality is measurable and visible in the MLflow UI. What You’ll Learn: 🔹 Why evaluation matters beyond traces—and what breaks with non-determinism, long chains, and multi-turn chats. 🔹 LLM-as-a-judge: rubrics, scores, rationales, and how that maps cleanly into MLflow workflows. 🔹 Built-in scorers: fast coverage for relevance, correctness, guidelines, safety, and related checks. 🔹 Datasets + predict-style evaluation: systematic inputs/expectations and when outputs aren’t precomputed. 🔹 Deterministic checks vs LLM judges: cheap guardrails plus nuanced judging where it counts. 🔹 Custom judges (make_judge) + deep eval-style integrations for advanced / session-level metrics. 🔹 Multi-turn session evaluation: history, context, and judging the whole conversation—not one turn at a time. Resources: 📕 Notebook 1.7: https://github.com/dmatrix/mlflow-genai-tutorials/blob/main/07_evaluating_agents.ipynb 🔗 Series playlist: https://www.youtube.com/playlist?list=PLaoPu6xpLk9EI99TuOjSgy-UuDWowJ_mR