Kubernetes GPU Autoscaling: Why Scale to Zero Costs You 10 Minutes Per Request

This video takes a rigorous, measured look at what it actually costs to scale AI inference workloads to zero on Kubernetes — and what it costs to bring them back. Using a real GPU cluster with a running language model, the video walks through the full autoscaling stack: KEDA's HTTP interceptor catching requests when no pods exist, the Cluster Autoscaler reclaiming idle GPU nodes, and the cold start wall that emerges when someone finally asks a question. That wall — over ten minutes for a single request that previously answered in four seconds — gets broken down phase by phase, revealing that the container image pull and model initialization are harder to attack than most people expect, and that a persistent volume cache buys back far less time than intuition suggests. The second half confronts the scaling-out problem, where a reactive signal like queue depth is already too late when a new replica takes twelve minutes to become ready. Along the way, the video uncovers a subtle but critical bug: the placeholder response that makes cold starts survivable can silently prevent the model from ever waking up, depending on which scaling metric you choose. The honest conclusion is that autoscaling inference is less a technical solution than a set of tradeoffs — between idle cost and cold start latency, between reacting to traffic and forecasting it, and between what works for a batch job and what a customer-facing product can actually tolerate. ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Sponsor: master.dev 🔗 https://master.dev/sale/?utm_campaign=DevOpsToolkit ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ #KubernetesAutoscaling #GPUInference #KEDA Consider joining the channel: https://www.youtube.com/c/devopstoolkit/join ▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬ ➡ Transcript and commands: https://devopstoolkit.live/ai/kubernetes-gpu-autoscaling-why-scale-to-zero-costs-you-10-minutes-per-request ▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please visit https://devopstoolkit.live/sponsor for more information. Alternatively, feel free to contact me over Twitter or LinkedIn (see below). ▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/ ▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox ▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬ 00:00 Autoscaling Inference (AI Models) 01:00 master.dev (sponsor) 02:22 The Inference Cost Of Doing Nothing 05:02 Scaling Inference Pod To Zero 08:45 The Inference Bill Did Not Move 12:19 The Inference Wall 17:10 Cutting The Wall Down 22:35 While You Wait for Inference 26:52 The Second Inference Replica