Long-Horizon AI Agents: How to Evaluate Them (Without Reward Hacking)
Frontier Data Summit 2026: Four builders of long-horizon agents on how to define long-horizon work, where training environments fall short, and how to design rewards that can't be hacked. Sudip Roy (Adaption), Yu Su (Ohio State), Niko Grupen (Harvey), and Huaxiu Yao (UNC-Chapel Hill) discuss what makes a task long-horizon, why errors compound over extended episodes, and how memory and continual learning enter the picture. They cover the data and verification constraints in knowledge work like law, how domain experts fit into environment and verifier design, partial rewards and rubric-based grading, and why reward hacking may be inevitable in RL and what to do about it. IN THIS VIDEO - How the panelists define long-horizon agents - Why errors compound over long episodes, and the role of memory - The data and verification gap in knowledge work like legal - How domain experts shape environments, verifiers, and tasks - Process rewards, rubrics, and the limits of intermediate signals - Why reward hacking is hard to avoid, and ways to mitigate it - Simulating real organizations to evaluate agents on the job SPEAKERS Sudip Roy, Adaption Yu Su, Ohio State University Niko Grupen, Harvey Huaxiu Yao, UNC-Chapel Hill CHAPTERS 00:00 Why long-horizon agents matter 01:02 Defining long horizon 04:19 Why long-horizon tasks fail 08:00 Training environments for knowledge work 09:17 Data constraints and verification 12:11 The role of domain experts 16:07 Environment design with human expertise 19:25 Experts as task proposers and verifiers 21:15 Rewards for long-horizon tasks 26:37 Partial rewards and process supervision 28:57 Unit tests and rubrics as graders 33:43 Mitigating reward hacking 38:50 Evaluating against real-world reward 41:11 Simulating organizations and the job RESOURCES Harvey Legal Agent Bench: https://harvey.ai/blog/introducing-harveys-legal-agent-benchmark Senior SWE-Bench: https://snorkel.ai/blog/senior-swe-bench-evaluating-coding-agents-like-senior-engineers/ Vending-Bench paper: https://arxiv.org/abs/2502.15840 OSWorld 2: https://osworld-v2.xlang.ai/ Snorkel AI Open Benchmarks Grants: https://benchmarks.snorkel.ai/