LLM Inference Explained: 12 Concepts You Actually Need to Know
Inference engines are full of jargon — continuous batching, paged attention, prefix caching, speculative decoding — and if you've ever nodded along while understanding very little, this video is for you. In one focused pass, it breaks down the 12 core ideas behind how language models actually run in production: what an engine does, what it's holding on that GPU, and exactly how each optimization works under the hood. More importantly, the video answers the question that actually matters: *when* does each of these things start to matter to *you*? Some bite the moment you deploy anything at all. Some wait until fifty people are using your service simultaneously. And some, honestly, you may never need. Rather than handing you a list to memorize, this is a practical framework for figuring out where you are right now — and what to focus on next. ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Sponsor: Infisical 🔗 https://infisical.com/?utm_source=youtube&utm_medium=paid&utm_campaign=devops_toolkit_review_ai_written_code ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ #LLMInference #AIEngineering #MachineLearning Consider joining the channel: https://www.youtube.com/c/devopstoolkit/join ▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬ ➡ Transcript and commands: https://devopstoolkit.live/ai/llm-inference-explained-12-concepts-you-actually-need-to-know ▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please visit https://devopstoolkit.live/sponsor for more information. Alternatively, feel free to contact me over Twitter or LinkedIn (see below). ▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/ ▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox ▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬ 00:00 LLM Inference Jargon, Decoded 01:10 Sponsor (Infisical) 02:20 What An Inference Engine Does (vLLM, Ollama, etc.) 05:15 GPU Memory: Model Weights vs KV Cache 07:39 Continuous Batching And Paged Attention 09:47 Quantization: More KV Cache, Same GPU 10:58 Prefix Caching: The Big Lever For Agents 12:28 Prefill vs Decode, And Speculative Decoding 14:48 Control Planes And LLM Gateways 16:30 Sharding, Autoscaling, And Cold Starts 18:38 When Each One Starts To Matter