One GPU. 30 People. What Runs Out First? (vLLM)

At nine in the morning, one person asks the office's local AI model to summarize a meeting, and the answer streams back faster than they can read. By ten, the whole team logs on. Same GPU, same model—and suddenly everyone's replies stutter, freeze, and crawl. Something ran out. But was it compute, memory, or scheduling? 🔔 Subscribe: https://www.youtube.com/channel/UC0DZj1PNa_Fp0MU6uPSKv5w?sub_confirmation=1 💙 Become a Member: https://www.youtube.com/channel/UC0DZj1PNa_Fp0MU6uPSKv5w/join 🐦 Twitter/X: https://x.com/cloud_codes 💬 Discord: https://discord.gg/mhgXXmhrh In this deep dive, Cloud Codes investigates how to serve an entire company from a single GPU using vLLM, continuous batching, and PagedAttention—and reveals the hidden bottleneck that actually kills your server under load. We break down the entire multi-user inference pipeline: • Prefill vs. Decode: Why processing prompts is compute-heavy, while generating text is memory-I/O bound—and why single-user setups leave most of the GPU's math units sitting idle. • Continuous Batching (Orca): How moving from static batches to iteration-level scheduling lets new requests take a seat the millisecond another user finishes, eliminating empty slots. • The 20-Page Document Freeze: What happens when one coworker pastes a massive prompt, and how chunked prefill (Sarathi-Serve) slices long documents so ongoing conversations never stall. • The Real Monster (KV Cache): Why model weights are only the entry fee, while conversation history eats your VRAM. We do the math on Qwen2.5-7B (~57 KB per token, ballooning to ~1.9 GB for a single 32k chat) and show how PagedAttention cuts memory waste from 80% down to under 4%. • The 3-Way Tuning Framework: Concrete settings and latency targets for a solo developer assistant, a 5-to-30 person internal office tool, and a public customer-facing app. If this breakdown helped you optimize your local AI infrastructure and understand production LLM serving, subscribe to Cloud Codes for a new deep dive every week. Build, solve, deploy. 🔗 Resources Mentioned: • vLLM Official Project Home & Documentation: https://docs.vllm.ai/ • Anyscale Guide to Continuous Batching: https://www.anyscale.com/blog/continuous-batching-llm-inference • Orca: Iteration-Level Scheduling (OSDI '22): https://www.usenix.org/conference/osdi22/presentation/yu • Sarathi-Serve: Stall-Free Chunked Prefill (arXiv:2403.02310): https://arxiv.org/abs/2403.02310 • PagedAttention Paper (vLLM, arXiv:2309.06180): https://arxiv.org/abs/2309.06180 • Qwen2.5-7B-Instruct Architecture & Config: https://huggingface.co/Qwen/Qwen2.5-7B-Instruct ⏱️ Chapters: 0:00 - 9 AM vs 10 AM: Why Shared AI Stutters 0:34 - Prefill vs Decode: The Two Phases of Inference 0:57 - TTFT vs ITL: Measuring How AI Feels 1:23 - Why AI Is Memory-Bound, Not Compute-Bound 1:57 - Static Batching: The Empty Chair Problem 2:23 - Continuous Batching: How Orca & vLLM Refill Slots 3:33 - Bob’s 20-Page Document: The Whole Office Freezes 3:58 - Chunked Prefill: Slicing Long Prompts to Save Decodes 4:58 - Tuning max_num_batched_tokens 5:26 - The Real Monster: The Growing KV Cache 5:34 - KV Cache Math: 57 KB Per Token on Qwen 7B 6:48 - Chat History: Why Long Threads Eat Your GPU 7:21 - PagedAttention: Slashing 80% Memory Waste to 4% 8:18 - Preemption: Why AI Pauses Mid-Sentence 8:59 - The 3-Way Tuning Framework: Solo, Office, or Public App 10:13 - Final Verdict: Watch the Cache, Not the User Count #vllm #localllm #gpu #aiengineering #machinelearning #sysadmin #devops #selfhosted #cloudcodes #hardware #datascience User Queries: how to share one gpu for multiple users llm vllm continuous batching tutorial how does chunked prefill work vllm why does local llm freeze with multiple users pagedattention explained simply how to calculate kv cache vram usage vllm max_num_batched_tokens explained vllm preemption warning fix how many users can run on rtx 3090 vllm difference between static and continuous batching orca iteration level scheduling llm ttft vs itl latency metrics qwen 2.5 7b vllm benchmark best settings for vllm office server cloud codes vllm