See What Breaks When You Build a Voice AI Agent & How To Fix It | Jesse Hall

Every voice agent has the same 8-stage pipeline, and each stage quietly eats your second 🤖 Here's how to claw it back - and why the hard parts aren't the ones you'd expect. What's inside: ⚡️ Where your latency budget actually goes - and why you're measuring the wrong metric (TTFT vs TTFA). ⚡️ The production traps that kill demos: why voice AI isn't serverless, and why silence reads as failure. ⚡️ What belongs in your code vs. the model - and why the LLM is the wrong place for workflow logic. 🎙️ Jesse Hall (@codeSTACKr) is a Staff Developer Advocate at LiveKit — the company whose open-source real-time stack powers ChatGPT's voice mode and voice products at 100,000+ developers. A self-taught developer turned educator, Jesse is known for turning the hardest parts of modern web dev into content you can actually ship. Because he learned it the hard way, he doesn't hand-wave the complexity — he shows you exactly where things break. Connect with Jesse: 👉 https://twitter.com/codestackr 👉 https://github.com/codeSTACKr 👉 https://linkedin.com/in/codeSTACKr Explore his YouTube Channel: 👉 https://youtube.com/codestackr 📝 Talk: Open Source Voice AI: How We Built ChatGPT's Voice Mode Infrastructure | Jesse Hall This talk was recorded at JSNation Amsterdam 2026: 🔗 https://jsnation.com/?utm_source=youtube&utm_medium=JesseHall 📌 Follow the link to watch the full version of all the conference talks, QnA’s with speakers and hands-on workshop recordings: 🔗 https://gitnation.com/events/jsnation-2026?utm_source=youtube&utm_medium=JesseHall 🕐 Timestamps: 00:00 - The Keyboard Era Is Ending 00:57 - Why Voice Agents Make You Want to Hang Up 02:37 - The Weekend Hack Behind ChatGPT's Voice Mode 04:31 - The 8 Stages Quietly Eating Your Second 06:26 - Why Your Dashboard Lies: TTFT vs TTFA 07:29 - The Art of Not Cutting Users Off 10:07 - Function Calls: Where Your Agent Feels Broken 12:33 - Stop Making the LLM Your Workflow Engine 14:07 - Voice AI Is Not Serverless (and Other Hard Truths) 15:50 - Reach for the Transcript, Not the Prompt 18:21 - 10 Lines to Your First Voice Agent 🚀 The most honest conversations happen after the talk. Join the next conferences by GitNation to sync with the engineers actually shipping the future: 💫 JSNation US 2026 Nov 16 & 19, 2026 (NewYork & Online) 🔗 https://jsnation.us/?utm_source=youtube&utm_medium=JesseHall 💫 React Summit US 2026 Nov 17 & 20, 2026 (NewYork & Online) 🔗 https://reactsummit.us/?utm_source=youtube&utm_medium=JesseHall 💫 AI Coding Summit Nov 16 & 17, 2026 (New York & Online) https://aicodingsummit.com/nyc/?utm_source=youtube&utm_medium=JesseHall and more! 🚀 Check out all the upcoming events from GitNation: 🔗 https://gitnation.com/events?utm_source=youtube&utm_medium=JesseHall #JSNation #GitNation #VoiceAI #ChatGPT #OpenSource #LiveKit #WebRTC #AIAgents #RealTime #TypeScript #ConversationalAI #SpeechToText #TextToSpeech #LLM #AIEngineering #DevTools #VoiceAssistant