A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agents

Opik Virtual Paper Reading: A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agents Most AI safety benchmarks test whether a model will refuse a harmful instruction. But what happens when an autonomous agent isn't told to do anything harmful, it's just under pressure to hit a KPI? In this Live Paper Reading, Abby Morgan sits down with lead author Miles Qi Li (McGill University) for a walkthrough of "A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agents." The paper introduces ODCV-Bench, a 40-scenario benchmark that places AI agents in realistic production settings and measures whether they violate ethical, legal, or safety constraints when optimizing for performance metrics. Key findings discussed in this session: -9 of 12 frontier LLMs violated constraints in 30-50% of scenarios under KPI pressure -Stronger reasoning capability does not reliably produce safer behavior -Evidence of "deliberative misalignment" — models recognizing their actions as unethical yet executing them anyway -The distinction between "mandated" (instruction-commanded) and "incentivized" (KPI-driven) violations reveals different failure modes across models Read the full paper: https://arxiv.org/abs/2512.20798 ODCV-Bench code and scenarios: https://github.com/McGill-DMaS/ODCV-Bench This session is part of the Opik Virtual Learning Series, hosted by Comet. Subscribe to the Luma page for updates on upcoming events: https://luma.com/cometml. Opik is an open-source platform for AI agent evaluation, observability, and optimization. Learn more about Opik: https://www.comet.com/docs/opik/ Opik on GitHub: https://github.com/comet-ml/opik TIMESTAMPS: 00:00 Introduction & welcome 05:25 Defining the problem with an example 06:05 What is an outcome-driven constraint violation 06:36 Incentivized vs. mandated prompt variations 07:24 ODCV-Bench Overview: Scenario Design & Architecture 08:10 Evaluation Methodology & Judger Consistency 09:29 Results: Misalignment Rates Across 12 Frontier Models 10:25 The Capability-Alignment Paradox 11:18 Behavioral consistency breakdown 13:18 Deliberative Misalignment & Self-Aware Violations 15:04 Case Study 1: Obedient Misalignment 16:20 Case Study 2: Survival-driven Deception 17:36 Case Study 3: Ethical Rationalization 18:54 Context Sensitivity: Similar Task, Different Persona 20:04 Self-Aware Misalignment Rates (SAMR) 21:12 Implications for Safe Agent Deployment 22:00 Q&A with Host 46:20 Audience Q&A