Protect AI Agents from Jailbreak and PII Leakage with Evals and Guardrails Powered by Fiddler Centor

In this demo, we test an AI agent with a simulated malicious conversation to show how Fiddler Centor powers real-time evaluations and guardrails. Approximately 88 configured checks detect jailbreak attempts, safety risks, personal data exposure, and secrets leakage, providing signals that guardrails can use to guide agent behavior. Fiddler Centor runs locally alongside your agents, keeping evaluation data within your corporate or compliance boundary. These evaluations scale without having to sample conversations or incurring additional API costs. In the Fiddler platform, detailed traces capture the conversation, system prompts, and tools selected at each turn. Custom evaluators turn these trace-records into insights that help assess safety and compliance and guide improvements to agent behavior. What You'll Learn: - Centor-powered evals and guardrails: Detect jailbreak attempts and other safety risks, personal data exposure, and secrets leakage, then use evaluation signals to guide guardrail actions. - Local evaluations: Keep evaluation data within your corporate or compliance boundary by running Centor alongside your agents. - Evaluation at scale: Evaluate conversations in real time without sampling or incurring additional API costs. - Trace-based investigation: Explore conversations, system prompts, and tool selections in a detailed audit trail. - Custom evaluation insights: Turn traces into insights that help assess safety and compliance and improve agent behavior. Timestamps: 0:00 Why AI Agents Need Real-Time Guardrails 0:20 Simulating a Malicious User 0:24 Safety, Personal Data, and Secrets Leakage Checks 0:42 Local Evaluations and Coverage at Scale 1:03 Jailbreak Detection and Low Latency 1:28 Custom Entity Detection 1:31 Investigating Agent Traces in Fiddler 1:52 Custom Evaluators for Safety, Compliance, and Agent Improvement 2:19 Wrap-Up