Why You Shouldn't Be Sampling Your AI Agent Evals | SAO Production Monitoring Demo

The agent is in production. Real users, real consequences. But most teams evaluate only a fraction of their traffic because running evals and guardrails at scale means seconds of latency and costs that spiral fast. So the one hallucination that actually matters slips through unsampled. SAO closes that gap with 100% coverage. Every trace, every user, every session is evaluated against your configured metrics at sub-150ms, at a fraction of the cost, powered by Luna, Splunk’s family of custom small language models running as judges in the background. In this walkthrough, we tour SAO’s production monitoring end-to-end: Live Log Stream with every trace scored, not sampled, and per-trace cost and latency visible at a glance Filtering for Context Adherence drops to surface a hallucination where the agent quoted S&P at $1,486, nearly 2x the real price, caught on the first occurrence Signals automatically detecting anomalies with no metric pre-configured, including root-cause analysis and suggested remedies Creating a new metric directly from a Signal and applying it to incoming traffic Retroactive recomputation scoring historical logs against the new metric to separate new regressions from long-running silent issues Full execution graphs exposing every LLM call, tool invocation, and decision point inside a single trace Agent Graph showing aggregate execution paths, stall points, and loop patterns across all sessions Dashboards rolling up metric trends, volume, cost, and latency drift over time This is what AI observability looks like when cost and latency stop being the bottleneck. Is cost or latency holding back your evals? See 100% coverage in action. Try Splunk Agent Observability: https://www.splunk.com/en_us/download/observability-cloud-free-edition.html Docs: https://agent-observability-docs.splunk.com/what-is-splunk-agent-observability 0:00 The coverage problem: evals cost too much to run on everything 0:18 100% coverage in the logstream — every trace scored by Luna 0:55 Catching a hallucination on first occurrence (S&P at $1,439) 1:22 Signals: surfacing issues nobody configured a metric for 1:35 Turning a Signal into a new metric instantly 1:52 Recomputing over historical logs 2:10 Inside a trace: the full execution graph 2:20 Agent graph: aggregate paths, stalls and loops 2:25 Dashboard: metric trends, volume, cost, latency 2:36 Recap and next steps