Meet SAO Autotune: From 79% to 95% Eval Accuracy in One Click

Watch SAO’s Autotune automatically improve eval accuracy from 79% F1 to 95% F1 with just a handful of human corrections, no manual prompt engineering required. This demo shows the complete observe-correct-tune-verify loop that compounds with every iteration. Out-of-the-box evals are 70% accurate at best. Even custom evals drift as your application evolves. Closing the accuracy gap traditionally takes hours of expert feedback and manual prompt engineering. Autotune automates this entire process. In this demo, you'll see: Starting with stock context adherence eval (79% F1 baseline) Providing natural language corrections on mislabeled traces One-click tuning cycle that rewrites evaluation criteria Prompt transformation: tightened rubrics and domain-specific language Validation: 95% F1 score on 2,000-trace benchmark 15.5% accuracy improvement from minutes of feedback Re-evaluating production log streams with tuned metrics The compounding improvement loop: observe, correct, tune, verify Key Results: F1 Score: 0.79 → 0.95 (+20% improvement) Accuracy: 80.9% → 96.4% (+15.5 percentage points) From: 2 out of 10 eval scores wrong To: Fewer than 1 out of 20 eval scores wrong Scale this to thousands of daily production traces and the impact compounds, better metrics, fewer hallucinations slipping through, and more confidence in every release. Try Splunk Agent Observability: https://www.splunk.com/en_us/download/observability-cloud-free-edition.html Docs: https://agent-observability-docs.splunk.com/what-is-splunk-agent-observability 0:00 - The Problem: 70% Baseline Eval Accuracy 0:22 - Stock Context Adherence Eval Testing 0:40 - Baseline Results: 79% F1 Score (2 of 10 Wrong) 0:50 - Reviewing Traces in Galileo Console 1:03 - Identifying Mislabeled Trace: Property Values Question 1:15 - Human Correction with Natural Language Explanation 1:28 - One-Click Autotune Activation 1:35 - Under the Hood: Prompt Rewriting Process 1:45 - Validation: Testing Tuned Metric on Same Traces 2:05 - Benchmark Results: 79% → 95% F1 Score 2:14 - Accuracy Improvement: 80.9% → 96.4% 2:40 - Re-Evaluating Production Log Stream 2:52 - Corrected Results Across the Board 3:10 - The Compounding Loop: Better Metrics, Fewer Hallucinations