Part 2: Align a Custom LLM Judge with SME Feedback using MLflow | MemAlign & the Alignment Lifecycle

Generic LLM judges work well for broad tasks. In domain-specific settings, they often miss the nuance that subject matter experts care about. This is Part 2 of a two-part series on evaluating a RAG agent with MLflow. In Part 1, we built the evaluation pipeline: traces, ground truth, multi-framework scorers, and a custom ChildAppropriateness judge. In Part 2, we close the loop with the alignment lifecycle: aligning that judge to SME standards using MemAlign. What this video covers: • Use case: The same RAG agent from Part 1 (assumes you already registered the judge and ran an initial evaluation) • Why generic LLM-as-a-judge scorers drift from expert standards in domain-specific evaluation • The three-stage alignment lifecycle: judge assessment → SME feedback → MemAlign alignment • Collecting SME feedback on traces in MLflow (UI and programmatically with mlflow.log_feedback) • Aligning the ChildAppropriateness judge with human feedback using MemAlign (semantic + episodic memory) • Re-running evaluation with the aligned judge to improve domain-specific scoring • How the evaluation-driven development loop and alignment lifecycle work together Watch Part 1 first: https://www.youtube.com/watch?v=4wqkHroNGFQ 🎤 Speaker: Joana Mesquita, MLflow Ambassador 🔗 Repo with the code: https://github.com/joanacmesquitaf/rag-agent-mlflow-evaluation 📖 Related blog (series context): https://medium.com/@joana.c.mesquita.f/from-generic-evaluators-to-domain-experts-judge-alignment-with-mlflow-memalign-8fc34d0ad717?postPublishedType=initial 📚 MemAlign docs: https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/memalign/