Why Your LLM Evals Are Missing Critical Failures
In this episode, Scott Clark, co-founder and CEO of Distributional, joins us to explore how teams can reliably operate and improve complex LLM systems and agents in production. Scott introduces a Maslow’s hierarchy of observability: telemetry for logging, monitoring for known signals, and post-production or online analytics to surface unknown unknowns. We dig into examples of real-world failures Scott’s team has seen in production systems, such as “lazy” tool-use hallucinations that standard evals miss, and how mapping traces into vector fingerprints enables clustering and topic discovery to uncover emergent behaviors. Scott explains how analytics can feed the data flywheel by generating evals, guardrails, and training data, and why online, adaptive approaches are essential for non-stationary models. We also touch on practical how-to’s such as instrumentation with OpenTelemetry, the GenAI semantic conventions, and the role of dedicated analytics tools. 🗒️ For the full list of resources for this episode, visit the show notes page: https://twimlai.com/go/767. 🔔 Subscribe to our channel for more great content just like this: https://youtube.com/twimlai?sub_confirmation=1 🗣️ CONNECT WITH US! =============================== Subscribe to the TWIML AI Podcast: https://twimlai.com/podcast/twimlai/ Follow us on Twitter: https://twitter.com/twimlai Follow us on LinkedIn: https://www.linkedin.com/company/twimlai/ Join our Slack Community: https://twimlai.com/community/ Subscribe to our newsletter: https://twimlai.com/newsletter/ Want to get in touch? Send us a message: https://twimlai.com/contact/ 📖 CHAPTERS =============================== 00:00 - Introduction 01:32 - What is Distributional? 03:54 - Bayesian statistics and optimization in multiagents 08:14 - Anti-patterns 10:11 - Hierarchy of observability 16:12 - Applying analytics in the lifecycle 21:58 - Trace clustering and vector mapping 26:42 - Evals 31:04 - OpenTelemetry (OTEL) and the Gen AI semantic convention 35:47 - Non-stationarity and “model weather” reports 41:30 - Examples of distribution shifts 46:24 - Distributional is open distribution 47:05 - Metrics for applying analytics 48:54 - Academic benchmark 51:07 - Future directions 🔗 LINKS & RESOURCES =============================== Distributional - https://www.distributional.com/ Distributional App - http://app.dbnl.com/ Distributional Docs - http://docs.dbnl.com/ Supporting Rapid Model Development at Two Sigma with Scott Clark & Matthew Adereth - 273 - https://twimlai.com/podcast/twimlai/supporting-rapid-model-development-at-two-sigma Bayesian Optimization for Hyperparameter Tuning with Scott Clark - 50 - https://twimlai.com/podcast/twimlai/bayesian-optimization-for-hyperparameter-tuning Democast: Automated Model Tuning with Scott Clark - https://twimlai.com/podcast/twimlai/automated-model-tuning 📸 Camera: https://amzn.to/3TQ3zsg 🎙️Microphone: https://amzn.to/3t5zXeV 🚦Lights: https://amzn.to/3TQlX49 🎛️ Audio Interface: https://amzn.to/3TVFAIq 🎚️ Stream Deck: https://amzn.to/3zzm7F5