Don't Trust Your AI Agents. Test Them Like a Scientist — Stephanie Jarmak, Omni Analytics

Stephanie Jarmak, a former astronomer who now builds evals at Omni Analytics after building a large coding benchmark at Sourcegraph, argues that most agent evaluations are easy to misread. She shows how swapping only the harness moved results by 30%, why public benchmarks are for shortlisting rather than decisions, and how to design your own evals like an experiment: realistic tasks, repeated runs, full traces and a measurement that matches what you care about. She also covers quality gates that actually stop work, the human review bottleneck, agent records you can't trust, and why production is the final test. Chapters: - 00:00 Intro - 01:11 When the agent ignores your tool - 02:47 The harness changes the results - 03:51 Public benchmarks vs. your own evals - 05:34 Designing evals like a scientist - 07:05 Trace the whole pipeline - 08:50 Choose what you measure - 09:48 Quality gates that actually stop work - 11:20 The human review bottleneck - 12:20 When the agent's record can't be trusted - 13:21 Production is truth - 15:05 When to escalate to a human Stephanie Jarmak, Omni Analytics: https://omni.co Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!