Autotune: 81% to 96% Eval Accuracy in One Click | Splunk Agent Observability

Watch Autotune in Splunk Agent Observability automatically improve eval accuracy from 71% to 96% with just a handful of human corrections, no manual prompt engineering required. This demo shows the complete observe-correct-tune-verify loop that compounds with every iteration. Out-of-the-box evals are 70% accurate at best. Even custom evals drift as your application evolves. Closing the accuracy gap traditionally takes hours of expert feedback and manual prompt engineering. Autotune automates this entire process. In this demo, you'll see: • Starting with stock context adherence eval (79% F1 baseline) • Providing natural language corrections on mislabeled traces • One-click tuning cycle that rewrites evaluation criteria • Prompt transformation: tightened rubrics and domain-specific language • Validation: 95% F1 score on 2,000-trace benchmark • 15.5% accuracy improvement from minutes of feedback • Re-evaluating production agent streams with tuned evals • The compounding improvement loop: observe, correct, tune, verify Key Results: • F1 Score: 0.79 → 0.95 (+20% improvement) • Accuracy: 80.9% → 96.4% (+15.5 percentage points) • From: 2 out of 10 eval scores wrong • To: Fewer than 1 out of 20 eval scores wrong Scale this to thousands of daily production traces and the impact compounds, better evals, fewer hallucinations slipping through, and more confidence in every release. Try Splunk Agent Observability: https://www.splunk.com/en_us/download/observability-cloud-free-edition.html Docs: https://agent-observability-docs.splunk.com/what-is-splunk-agent-observability 0:00 - The Problem: 70% Baseline Eval Accuracy 0:22 - Stock Context Adherence Eval Testing 0:40 - Baseline Results: 79% F1 Score (2 of 10 Wrong) 0:50 - Reviewing Traces in Splunk Agent Observability Console 1:03 - Identifying Mislabeled Trace: Property Values Question 1:15 - Human Correction with Natural Language Explanation 1:28 - One-Click Autotune Activation 1:35 - Under the Hood: Prompt Rewriting Process 1:45 - Validation: Testing Tuned Evals on Same Traces 2:05 - Benchmark Results: 79% → 95% F1 Score 2:14 - Accuracy Improvement: 80.9% → 96.4% 2:40 - Re-Evaluating Production Agent Stream 2:52 - Corrected Results Across the Board 3:10 - The Compounding Loop: Better Evals, Fewer Hallucinations