Improving LLM Throughput via Data Center-Scale Inference Optimizations

Speaker: Maksim Khadkevich, Sr. Software Engineering Manager, Dynamo, NVIDIA Khadkevich discusses data center scale inference optimizations for Large Language Models (LLMs). This talk walks through Dynamo architecture, focusing on techniques like disaggregated serving to handle the distinct prefill and decode workloads, a KV-aware router for KV cache reuse, and an SLO-based planner for autoscaling and resource rebalancing. Learn how Dynamo achieves significant throughput improvements for LLM serving. ➡️ Learn more: https://developer.nvidia.com/dynamo 00:00 -- Introduction: Data Center Scale Inference Optimizations 01:54 -- Understanding Inference Workloads: Prefill vs. Decode 03:26 -- Disaggregated Inference Serving 06:01 -- KV-Aware Router for Reusing KV Cache 08:00 -- SLO-Based Planner for Autoscaling and Resource Rebalancing 09:44 -- KV Block Manager for Offloading KV Cache 11:09 -- Solving Kubernetes Deployment Challenges 12:46 -- AIConfigurator for Optimal Disaggregated Serving Configuration 14:49 -- Dynamo Architecture and Ecosystem 16:33 -- Get Started with NVIDIA Dynamo #DataCenterInference #LLMServing #DisaggregatedServing