The Retry Loop That's Burning Your AI Agent Budget | SAO Trace Walkthrough

One of the most expensive failure modes in AI agents is a silent retry loop. An agent hits a rate limit, blindly retries until it succeeds, the user gets a response, and no ticket ever gets filed. Under the hood, tokens are burning, latency is compounding, and a transient error today becomes a sustained outage tomorrow. In this walkthrough, we run a Multi-Agent Stock Analysis app: a Growth Analyst building the bull case and a Risk Analyst building the bear case, converging on a consensus score, and show how Galileo surfaces a fragile retry pattern that's invisible from the outside. You'll see: A healthy baseline run analyzing Apple stock across bull/bear agents The same request with a flaky tool call, output still looks perfect, user never notices SAO’s Log Stream flagging a spike in Tool Error Rate on the second trace The trace view isolates two failed tool calls before a successful retry Root cause identified in seconds: fragile retry logic in the get_stock_info tool call not the analysts, not the consensus logic Why retry patterns without circuit breakers or timeout guards are a ticking cost-and-latency bomb What used to take hours of log-diving, Galileo collapses into span-level visibility in seconds. Try Splunk Agent Observability: https://www.splunk.com/en_us/download/observability-cloud-free-edition.html Docs: https://agent-observability-docs.splunk.com/what-is-splunk-agent-observability 0:00 The hidden cost of silent retry loops 0:23 Meet the app: Growth Analyst + Risk Analyst consensus pipeline 0:38 Healthy baseline run — analyzing Apple stock 0:51 Same request, flaky tool call — user sees no difference 1:10 Why silent failures never get a ticket 1:25 Galileo Log Stream: clean trace vs. problem trace 1:30 Tool Error Rate spikes on the second run 1:42 Trace view: two failed tool calls before success 1:53 Why "just two retries" is the real red flag 2:08 Root cause isolated to the get_stock_info tool call 2:20 From hours of log-diving to seconds of span-level visibility