How to Engineer AI Inference Systems [Philip Kiely] - 766

In this episode, Philip Kiely, head of AI education at Baseten, joins us to unpack the fast-evolving discipline of inference engineering. We explore why inference has become the stickiest and most critical workload in AI, how it blends GPU programming, applied research, and large-scale distributed systems, and where the line sits between inference and model serving. Philip shares how research-to-production can move in hours, not months, and why understanding “the knobs” of inference—batching, quantization, speculation, and KV cache reuse—lets teams design better products and SLAs. We trace the inference maturity journey from closed APIs to dedicated deployments and in-house platforms, discuss GPU lifecycles, and survey today’s runtime landscape, including vLLM, SGLang, and TensorRT LLM. Finally, we look ahead to agents and multimodality, making the case for specialized, workload-specific runtimes when performance and efficiency matter most. 🗒️ For the full list of resources for this episode, visit the show notes page: https://twimlai.com/go/766. 🔔 Subscribe to our channel for more great content just like this: https://youtube.com/twimlai?sub_confirmation=1 🗣️ CONNECT WITH US! =============================== Subscribe to the TWIML AI Podcast: https://twimlai.com/podcast/twimlai/ Follow us on Twitter: https://twitter.com/twimlai Follow us on LinkedIn: https://www.linkedin.com/company/twimlai/ Join our Slack Community: https://twimlai.com/community/ Subscribe to our newsletter: https://twimlai.com/newsletter/ Want to get in touch? Send us a message: https://twimlai.com/contact/ 📖 CHAPTERS =============================== 00:00 - Introduction 03:40 - Why inference is the most important AI workload? 06:21 - Inference vs model serving 07:18 - Inference challenges 09:57 - Pace of inference research to production timeline 13:41 - Reasons to care about inference engineering 15:49 - Considerations in build vs buy decisions 22:08 - Product maturity cycle 27:14 - GPU lifecycles in inference maturity 32:14 - LLM-assisted inference 36:46 - Agents and multimodal models in specialized inference optimization 47:21 - Open source runtimes: vLLM, SGLang, and TensorRT LLM 49:50 - Specialized AI hardware 51:24 - Future trends and predictions 52:36 - Where to find the inference engineering book 🔗 LINKS & RESOURCES =============================== Inference Engineering Book - https://www.baseten.co/inference-engineering/ Baseten - https://www.baseten.co/ 📸 Camera: https://amzn.to/3TQ3zsg 🎙️Microphone: https://amzn.to/3t5zXeV 🚦Lights: https://amzn.to/3TQlX49 🎛️ Audio Interface: https://amzn.to/3TVFAIq 🎚️ Stream Deck: https://amzn.to/3zzm7F5