Catching a Silent AI Hallucination in Under 2 Minutes | SAO Trace View Demo

The most dangerous AI failure is a confident hallucination. Perfectly formatted. Delivered with certainty. Completely wrong. And without the right observability in place, no one ever knows it happened. In this walkthrough, we reproduce a real hallucination in a Stock Agent workflow and use SAO to isolate the exact step where the model went off the rails, without touching the API, the data pipeline, or any upstream code. You'll see: A healthy baseline run where the agent correctly fetches Apple's live price A second identical query that returns a completely different (and wrong) number, delivered with the same confidence Side-by-side metric comparison in SAO Console — Context Adherence dipping, Correctness flagging a false response A full Trace View breakdown showing the first LLM call (ticker ID correct), the tool call (API returned the right price), and the final LLM call (hallucinated despite having the correct data in context) The Correctness judge rationale explaining why the output contradicted the retrieved tool data The entire investigation, from symptom to root cause, takes under two minutes. No log-diving. No guessing. No debugging the wrong layer. This is the difference between hoping your agent is reliable and knowing exactly where and why it fails. See it yourself: Try Splunk Agent Observability: https://www.splunk.com/en_us/download/observability-cloud-free-edition.html Docs: https://agent-observability-docs.splunk.com/what-is-splunk-agent-observability 0:00 The silent failure: confident hallucinations 0:27 The healthy run: agent fetches Apple's real price 0:42 LLMs are non-deterministic: same question, different answer 1:08 Spotting the difference in Galileo Console 1:15 Context Adherence dips, Correctness flags the false response 1:23 Opening the Trace View 1:26 First LLM call — ticker correctly identified 1:35 Tool call — API returned the right price 1:46 Final LLM call — hallucination despite correct context 1:53 Correctness judge rationale: output contradicts tool data 2:08 From trace to root cause in under two minutes