Optimize KV cache with llm-d and vLLM

Discover how llm-d improves vLLM inference efficiency by preventing KV cache duplication across multi-GPU deployments. Traditional L7 load balancers use naive round-robin routing, causing up to 50% KV cache duplication and higher latency in multi-turn LLM serving. Greg Pereira, Senior Machine Learning Engineer at Red Hat, explains how llm-d introduces a prefix-based prefill and decode (PD) decider. Learn how independent routing decisions for prefill and decode optimize prefix cache affinity and token load scoring across GPUs. 00:00 The challenge of scaling Models-as-a-Service 00:22 Why traditional L7 load balancing fails vLLM 00:45 KV cache duplication and latency issues 01:16 Selective PD disaggregation with llm-d 01:56 How the prefix-based PD decider works 02:18 Prefill vs. decode routing concerns 03:02 Calculating token load score for prefill From vLLM Office Hours #55: Mooncake + vLLM/llm-d Deep Dive. Full session: https://youtu.be/RFyeBEy1AP8 Join us live every other Thursday: https://red.ht/office-hours #vLLM #llm-d #KVCache #AIInference #RedHat #MLOps