Scale and Improve AI Agent Evaluations with Dynamic Model Escalation in Fiddler Centor

As agentic AI applications scale, so must the evaluations that keep them in check. In this product video, we show how Fiddler Centor evaluates unstructured inputs and outputs for safety, faithfulness, and sensitive information, including PII, in real time across thousands of agent conversations, without relying solely on costly general-purpose models. Centor routes evaluations to efficient specialist models. Dynamic Model Escalation sends inputs outside their training domain to Fiddler-hosted general-purpose LLMs, keeping costs low and accuracy high. In the Fiddler platform, custom evaluators can be configured for use-case specific evaluations, and dashboards show agent performance and outcomes over time. Zoom in on any data point, inspect the underlying conversation, and investigate deviations to tighten agent safeguards. Learn how Fiddler Centor scales evaluations for your agentic AI deployments: https://www.fiddler.ai/blog/dynamic-model-escalation-ai-evals What You'll Learn: - Specialist model evaluations: Evaluate agent conversations for safety, faithfulness, and PII at scale, efficiently and at a lower cost. - Dynamic Model Escalation : Route inputs that fall outside a specialist model's training domain to general-purpose LLMs to improve accuracy while keeping costs low. - Custom evaluators: Define use case-specific evaluations in the Fiddler AI Control Plane using Fiddler-hosted models. - Dashboards and investigation: Track agent performance over time, inspect underlying conversations, and investigate deviations to strengthen safeguards. Timestamps: 0:00 Why AI Agent Evaluations Need to Scale 0:27 The Cost of General-Purpose Models 0:49 Specialist Models for Efficient Evaluation 1:02 Introducing Dynamic Model Escalation 1:21 Balancing Evaluation Cost and Accuracy 1:46 Custom Evaluators in the Fiddler AI Control Plane 1:58 Dashboards for Agent Performance 2:09 Investigating Deviations and Tightening Safeguards 2:33 Wrap-Up