How Do Lawyers Actually Judge AI Output? Rubrics vs. Preferences — Russell Yang, Microsoft

Frontier Data Summit 2026: Russell Yang introduces JudgmentBench, a benchmark built with Harvey and expert lawyers to test how legal AI should be evaluated. Russell Yang introduces JudgmentBench, a benchmark built with Harvey and more than 50 expert lawyers. The study uses 3,000 lawyer annotations across 30 BigLawBench tasks to compare pairwise preferences with rubric-based scoring and ask how to evaluate AI in legal work, where quality can be subjective and ground truth is hard to establish. The project received support through Snorkel AI’s Open Benchmarks Grants. IN THIS VIDEO • Why evaluating legal work is part science and part art • Pairwise preferences versus rubric-based scoring • How 3,000 expert lawyer annotations across 30 BigLawBench tasks were collected • How well different evaluations rank output quality • Results and implications for legal AI evaluation SPEAKER Russell Yang, Applied Scientist, Microsoft; Project Lead, JudgmentBench CHAPTERS 00:00 JudgmentBench and the science-versus-art question 01:55 Evaluating legal work: preferences or rubrics? 02:52 Expert lawyers, annotations, and BigLawBench 04:30 How the study compares model outputs 06:09 Measuring how well evaluations rank quality 07:50 Results and implications for legal AI evaluation RESOURCES JudgmentBench paper: https://arxiv.org/abs/2605.25240 Snorkel AI Open Benchmarks Grants: https://benchmarks.snorkel.ai/