Fix Your Al Voice Agent Before Users Break It

Most voice agents get tested by the person who built them, which means they only ever see the paths that person thought of. LiveKit Simulations runs your real agent against LLM-driven simulated callers, in text or in audio, and grades every conversation against expectations you write in plain English. Nothing gets deployed. The CLI starts your agent from your working tree, runs the conversations, and hands you a pass rate and a dashboard link. Sign up for LiveKit Cloud: https://cloud.livekit.io/signup?utm_source=youtube&utm_medium=video&utm_campaign=devrel&utm_content=ycSxhQTKX7Y In this walkthrough I scaffold the Python agent starter, run its three checked-in scenarios in text mode, generate five more from the agent's own source, and then run the same scenarios in audio with background noise and packet loss. You'll see a passing run that still reports a real issue, a generated scenario that catches a pacing bug the hand-written ones never would (and how to turn it into a checked-in test from the dashboard), and the one line of code an agent needs before audio-mode simulation can run at all. Then I cover the scenario file format, grading on final state with on_simulation_end, wiring simulations into CI, and when to reach for text versus audio. Text or audio? Text mode disables STT, TTS, and VAD and exercises your LLM, your tools, and your conversation logic, so it's fast enough to run on every merge. Audio mode runs your full speech pipeline in real time and measures what a transcript can't show you: the end-to-end latency the caller heard at p50, p95 and p99, turn-taking, awkward silences, unanswered turns, and word and character error rates in both directions. Keep pull request runs in text and run audio nightly or before a release. Simulations are free through October. Your agent still pays for its own models. 📚 Resources 📚 Agent Simulations docs: https://docs.livekit.io/agents/start/testing/simulations/?utm_source=youtube&utm_medium=video&utm_campaign=devrel&utm_content=ycSxhQTKX7Y ⏰ Timestamps ⏰ 0:00 - Intro 0:30 - What a simulation runs 1:11 - Scaffold the Python agent starter 1:35 - The scenarios file 2:06 - Text mode: Run a text simulation 2:46 - Read the run in the dashboard 3:27 - Generate scenarios from your source 4:38 - Audio mode: your agent speaks first 6:19 - Generated audio scenarios catch 30s of silence 7:11 - Latency and turn-taking metrics 7:38 - Background noise and packet loss 8:46 - Turn any scenario into a test 9:10 - Grade on final state with on_simulation_end 9:31 - Run simulations in CI 9:42 - When to run text vs audio 10:06 - Tips for reliable scenarios 🤝 Join the Community: https://community.livekit.io/?utm_source=youtube&utm_medium=video&utm_campaign=devrel&utm_content=ycSxhQTKX7Y #livekit #ai #voiceai