Beyond Model Sharding: Atomic Scheduling and Disaggregated LLM Serving With ... - K. Yan & C. Zicong

LLM serving on Kubernetes is moving beyond model sharding. In production, one inference replica may be a coordinated group of pods across nodes, GPUs, and network topology. If only part of the group is scheduled, accelerators can be reserved without serving traffic. With disaggregated prefill and decode, rollout, scaling, and placement mismatches can also hurt latency and GPU utilization. LeaderWorkerSet provides a Kubernetes-native API for managing a group of pods as one workload unit for multi-host inference. This talk focuses on two recent areas of LWS evolution: gang scheduling and DisaggregatedSet. We will show how PodGroup integration with Volcano, scheduler-plugins, and YuniKorn schedules leader and worker pods together, and how DisaggregatedSet coordinates multiple LWS resources for prefill and decode. Attendees will learn when Deployment is no longer enough, when LWS is the right abstraction, and when multiple LWS resources should be managed as one logical service.