Building Open AI Benchmarks: Terminal-Bench-Science, JudgmentBench & SlopCodeBench

Frontier Data Summit 2026: Three open-benchmark project leads discuss what it takes to build and maintain benchmarks for frontier AI, moderated by Vincent Sunn Chen. At Snorkel AI’s Frontier Data Summit, the project leads behind Terminal-Bench-Science, JudgmentBench, and SlopCodeBench join Vincent Sunn Chen for a panel on building open benchmarks. They cover the surprises of building their benchmarks, balancing realism with reliable grading, using domain experts and AI agents to create and review tasks, benchmaxing, and advice for the next wave of benchmark builders, followed by audience Q&A. All three projects received support through Snorkel AI’s Open Benchmarks Grants. IN THIS VIDEO • The biggest surprises in building each benchmark • Balancing realistic work with reliable, verifiable grading • Using domain experts and AI agents to create and review tasks • Designing verifiers and graders that accept every valid solution • Benchmaxing, held-out test sets, and the community’s responsibility • Audience Q&A on onboarding non-technical experts, keeping up with the frontier, and automating benchmark creation SPEAKERS Vincent Sunn Chen (moderator), Founding Team, Snorkel AI Steven Dillmann, PhD Student, Stanford; Project Lead, Terminal-Bench-Science Russell Yang, Applied Scientist, Microsoft; Project Lead, JudgmentBench Gabriel Orlanski, PhD Student, UW-Madison; Project Lead, SlopCodeBench CHAPTERS 00:00 Welcome and panel introduction 00:22 The biggest surprises in building the benchmarks 01:57 Realistic work versus reliable measurement 04:41 Using domain experts and AI agents to build tasks 07:37 Designing grading and verifiers 11:52 Benchmaxing and the community’s responsibility 19:01 Lessons for the next wave of benchmark builders 21:46 Audience Q&A: onboarding non-technical experts 23:50 Audience Q&A: choosing benchmarks and keeping up with the frontier 26:13 Audience Q&A: can benchmark building be automated? 30:39 Where benchmark builders spend their time RESOURCES Terminal-Bench-Science: https://www.terminal-bench-science.ai/ JudgmentBench paper: https://arxiv.org/abs/2605.25240 SlopCodeBench paper: https://arxiv.org/abs/2603.24755 Snorkel AI Open Benchmarks Grants: https://benchmarks.snorkel.ai/