The Capability-Reliability Gap: Why AI Agents Still Fail in the Real World | Alignment Workshop
Why AI agents top benchmarks but fail in practice: a panel with Stefan Rabanser (Princeton), Adam Gleave (FAR.AI), and Dylan Hadfield-Menell (MIT). Rabanser opens with the measurement problem. Aviation, nuclear, and automotive engineering never certify safety on a single accuracy number, so he proposes four dimensions for AI agents: consistency (same answer on repeated identical inputs), robustness (graceful degradation under perturbation), predictability (does the model know when it does not know), and operational safety (bounded severity when failure happens). His data shows accuracy climbing across model generations while reliability stays comparatively flat. Gleave asks why reliability remains unsolved when billions of dollars ride on it. His answer is two-part: reliability is hard to measure, since capability is something you can feel in a single afternoon while a system that drops a production database once a year looks fine in testing, and there is no simple recipe for it, no equivalent of scaling laws. He argues for an engineering approach, modularity, safety predictions made ahead of time, and rigorous white-box evaluation, in contrast to mechanistic interpretability as a scientific route and guaranteed safe AI as a mathematical one, on the grounds that engineering can arrive in time. Hadfield-Menell locates the problem in training. Systems overfit to verifiable proxy metrics at the cost of unverifiable ones, a brittleness he expects reinforcement learning pressure to worsen, and they meet a last-mile specification problem where real-world complexity exceeds what anyone specified upfront. The discussion turns to what a clean pre-deployment result actually means, tampering evaluations, machine unlearning failures, and the case for treating agent security infrastructure as a first-order design problem. Chapters 0:00 Introductions: the capability-reliability gap 2:07 Stefan Rabanser: reliability as the bottleneck 2:41 Adoption lags what models can already do 3:31 Real-world agent failures 4:04 What aviation, nuclear, and automotive do differently 4:47 Four dimensions: consistency, robustness, predictability, safety 5:42 Capability rises, reliability stays flat 7:04 Autonomy raises the reliability bar 8:00 Adam Gleave: why has reliability not been solved? 8:39 Reliability is hard to measure 9:56 No simple recipe, unlike scaling laws 10:30 The pragmatic goal, and why we are flying blind 12:17 The case for an engineering approach to safety 13:28 Dylan Hadfield-Menell: overfitting to proxy metrics 15:35 The last-mile specification problem 17:03 What technical solutions could help 18:22 Discussion: what created the gap? 20:47 Open-world evaluation and construct validity 21:50 What a clean pre-deployment result means 25:03 Tampering evaluations and stress-testing 27:04 Post-training, shortcut learning, and unlearning 28:02 Models asking clarifying questions 29:24 Audience Q: can we salvage compositional safety? 30:37 Agent security as an unsolved engineering problem 33:37 Building safety into agent infrastructure 35:21 What to watch for in the next 12 months More AI safety research: https://far.ai Alignment Workshop playlist: https://youtube.com/playlist?list=PLBY5kyt_LfFg&si=bNxmUxbr_otXFoqj FAR.AI is a research nonprofit working to ensure the safe development of advanced AI. We host the Alignment Workshop series and publish frontier alignment research.