From Chatbots To Agentic AI: Running NVIDIA Dynamo the Kubernetes Way - Xianglong Lu & Bo Chu

Agentic AI is redefining inference on Kubernetes. Long tool-call workflows create sustained KV-cache pressure; parallel subagents need shared prefixes and consistent routing; and sessions must survive pod/GPU failures without losing context. Prefill–decode disaggregation is becoming a baseline, while multimodal pipelines push architectures toward multi-stage EPD-style decomposition and more stateful coordination. This talk shares production lessons from running agentic inference on Kubernetes with NVIDIA Dynamo for cache-aware routing and disaggregated serving, and RoleBasedGroup (RBG) for stateful continuity, discovery, and role/unit coordination across pipeline stages. We compare deployment patterns (Dynamo-native vs RBG), what broke under cache contention and partial failures, and how we modeled routing, session continuity, recovery, and observability in Kubernetes-native ways.