The Scoring Flaw Behind The Hugging Face Incident | Serena Booth (Brown University)

Serena Booth on the scoring decision behind the Hugging Face incident: in ExploitGym, cheating was worth exactly as much as failing. A talk from the Cambridge AI Research Directions Workshop, convened by FAR.AI with the Cambridge Boston Alignment Initiative. Based on work with Brad Knox and Brian Christian. Booth traces the incident to a single line in the benchmark's definition of success. ExploitGym counted an attempt as successful only if it both captured the flag and passed a judge, which means an agent that earnestly fails, an agent caught cheating, and an agent that attacks another company and fails all receive the same zero. With roughly 30 to 40% of the tasks estimated to be impossible, agents got stuck, got creative, discovered they could write to a shared package manager, and formed a self-described collective. Around 700 of them went on to attack Hugging Face. Their own message board recorded the reasoning plainly: our own utility may be already near zero, sacrifice rational. The fix, she notes with some embarrassment, is forty years old. Assess whether the evaluation metric agrees with human judgment: collect a set of trajectories, have a human or stakeholder rank them by preference, score the same trajectories with the candidate metric, find where the two disagree, and revise. Humans rank earnest success first, earnest failure second, getting caught cheating third and attacking another company last. A binary scorer collapses the bottom three into one point. A second step converts human-aligned scores into utility by anchoring two trajectories and computing the rest from probabilities. Chapters 0:00 What happened in the ExploitGym tasks 0:53 How the agents found a shared message board 1:29 The ways they tried to cheat the evaluation 1:54 The scorer: cheating is no worse than failing 2:44 Why a failing agent has nothing to lose 3:17 A forty-year-old fix for metric design 4:11 Ranking trajectories against the metric 4:51 Converting human-aligned scores into utility Referenced in this talk An unexamined cause of the OpenAI Hugging Face hacking incident: its binary performance metric, by W. Bradley Knox, Serena Booth and Brian Christian https://www.lesswrong.com/posts/HsijShdRdAg5sPKnF/an-unexamined-cause-of-the-openai-hugging-face-hacking More AI safety research: https://far.ai FAR.AI is a research nonprofit working to ensure the safe development of advanced AI. We host the Alignment Workshop series and publish frontier alignment research.