Multi-Agent Supervisor Pattern: LangGraph Routing, Tracing & Evaluation (Notebook 1.10)

In the tenth and final tutorial of the Mastering MLflow for GenAI series, Jules Damji (Databricks) builds a multi-agent supervisor pattern with LangGraph and MLflow, routing queries to specialized agents, synthesizing results, and evaluating the full workflow with granular traces and scorers. This session simulates a Databricks Agent framework–style supervisor using LangGraph’s StateGraph for acyclic, state-machine-like execution. The use case is a FEMA disaster-response assistant over roughly simulated records. What You’ll Learn: 🔹 Multi-agent supervisor pattern: a supervisor fans out queries to specialized agents, then returns one cohesive answer 🔹 Supervisor router (Node 1): classify natural-language queries and route to a Genie-like agent, a knowledge assistant, or both 🔹 Genie-like agent (structured data): Natural Language → SQL with an LLM chain, executed against FEMA data via Python Pandas + SQLite 🔹 Knowledge-based assistant (unstructured data): document retrievers, OpenAI embeddings, and cosine similarity over in-memory files like PDFs 🔹 Synthesizer (Node 3): combine one or both agent outputs into a single response 🔹 Granular MLflow tracing: define agent tools: query, tool, embedding, and retriever functions with MLflow span markers 🔹 Built-in scorers: use MLflow’s 60+ scorers for answer relevance, safety, and guideline adherence on an evaluation dataset 🔹 Custom routing accuracy: implement a customer MLflow @scorer metric that compares supervisor fan-out against ground truth 🔹 Extra practice: supplementary notebooks for Deep Agents and CrewAI multi-agent frameworks Resources: 🔗 Notebook 1.10: https://github.com/dmatrix/mlflow-genai-tutorials/blob/main/10_multi_agent_supervisor.ipynb 🎥 Full Series Playlist: https://youtube.com/playlist?list=PLaoPu6xpLk9EI99TuOjSgy-UuDWowJ_mR