How LLM Serving Works: An Introduction to vLLM and SGLang

How LLM Serving Works: An Introduction to vLLM and SGLang, a lecture by Dr. Sreedath Panat (Co-founder, Vizuara AI Labs; PhD, MIT). Every AI product pays for every token it generates, and over a model's life serving costs far more than training. This lecture follows one running example the whole way, Llama 3.1 8B on a single H100, and builds up why serving is limited by memory rather than maths, what the KV cache is and why only keys and values are cached, how prefill and decode differ, and why batching many users onto one GPU changes everything. We then look at the ideas behind vLLM and SGLang: PagedAttention, static versus continuous batching, cache-aware routing across replicas, the layout of an inference engine, and RadixAttention for prefix caching. The lecture ends with the full arithmetic: serving 1 million daily users takes about 90 H100s and $2.0M a year with a simple server, and about 15 H100s and $0.33M a year with vLLM or SGLang, about 6 times cheaper. Join the live bootcamp, vLLM & SGLang Engineering: End-to-End (12 live lectures, 11 Nov to 26 Dec, Wednesdays and Saturdays 7:30 to 9:30 AM IST, which is Tuesdays and Fridays 6 to 8 PM PT; all recorded): https://maven.com/rajat-dandekar/vllm-sglang-engineering?promoCode=VLLMSGLANG Use code VLLMSGLANG for 10% off. Part 2, Journey of a token in vLLM and SGLang Video: https://youtu.be/IDjyrXKsC5U Slides: https://vllm-lightning-2.vercel.app Slides for this lecture: https://vllm-lightning-1.vercel.app Interactive websites Animations (VRAM to SRAM, prefill vs decode, KV cache, continuous batching, PagedAttention, RadixAttention): https://llm-serving-animations.vercel.app Serving simulator (simple server vs vLLM/SGLang on one H100): https://vllm-serving-demo.vercel.app Follow one request through vLLM, step by step: https://vllm-request-journey.vercel.app Follow the same request through SGLang: https://vllm-request-journey-sglang.vercel.app Chapters 0:00 Welcome and audio check 0:55 Who is teaching: Sreedath and Vizuara 2:54 Why this lecture exists: the vLLM and SGLang bootcamp 5:16 Roadmap for today's five part lecture 8:31 Audience poll: who already knows transformers 9:31 Example setup: Llama 3.1 8B and its memory footprint 12:34 Inside a GPU: VRAM, SRAM, and compute cores 16:59 Memory bandwidth, TFLOPS, and H100 rental cost 17:49 A user request, tokens, and the first KV cache numbers 21:10 Why serving costs eventually dwarf training costs 22:46 Back of envelope cost to serve 1 million users 24:40 What vLLM and SGLang actually are 31:22 Attention math recap and the prefill versus decode split 45:56 Batching requests to push GPU throughput higher 51:12 KV cache deep dive: why keys and values, not queries 1:02:32 Routing requests and cache aware load balancing 1:04:56 Inside an inference engine: API server to model runner 1:06:49 Paged attention explained 1:08:56 Static batching versus continuous batching, live animation 1:16:34 Paged attention memory math and GPU utilization 1:23:08 Radix attention and prefix caching 1:28:12 Full cost comparison: naive serving versus vLLM and SGLang 1:37:56 Beyond today: quantization, speculative decoding, and course scope 1:40:16 Course logistics, prerequisites, and pricing 1:48:04 Closing, demo site, and wrap up