Episode 5: AI Model & Agent Monitoring

AI model monitoring is the continuous tracking of an AI system's performance, inputs and behavior in production, so teams catch degradation, drift and failure before they reach users. It starts by capturing an expected baseline at deployment, then scoring live behavior against it. It's what turns "we deployed it" into "we know it's still working." The gap between your last monitoring check and right now? That's exactly where production failures hide. For ML engineers, monitoring used to mean watching a handful of model metrics on a dashboard. Agents changed the job. A model that degrades returns worse predictions. An agent that degrades takes worse actions — calling tools it shouldn't, looping without terminating, reaching data it was never granted access to. And the silent failures almost never start with the model breaking. They start upstream, with data shifting, weeks before accuracy visibly drops. In this video, Michael Rahm, Director of Product Marketing at Collibra, breaks down the four families of signal every team should watch: 1️⃣ Performance — accuracy, latency and error rate, the core health check 2️⃣ Data — whether input distributions have shifted from your training baseline, usually the first sign something is wrong 3️⃣ Output — prediction drift, hallucination rate, and for agents, whether responses are grounded 4️⃣ Behavior (unique to agents) — tool-call patterns, loop detection, scope adherence, and policy-trigger rate Plus, the four parts of effective alerting: define thresholds per signal, attach context to every alert, route by ownership, and tier severity — so a minor drift and an agent breaching policy don't arrive looking identical. The question isn't whether your models and agents will drift. It's whether you'll catch it in your monitoring — or in your customer complaints. 📖 Get the full breakdown, including the three types of model drift, how agent monitoring differs from model monitoring, and how monitoring relates to AI observability: https://www.collibra.com/blog/ai-model-and-agent-monitoring-metrics-drift-detection-and-runtime-alerting-in-production #AIModelMonitoring #AIObservability #AIGovernance #Collibra #ModelDrift #ResponsibleAI #AICommandCenter #MLOps