Tidal Auto-scaling for Training and Inference Based on Kubernetes + KEDA - Jun Zheng & Tan Pei Xiang

CMB AI assistants exhibit distinct peak and off-peak traffic patterns. During off-peak hours, the platform allocates idle GPUs to LoRA incremental training. As peak traffic arrives, KEDA scales out vLLM inference services based on request load, prompting training tasks to save progress at secure checkpoints and proactively release capacity. Once traffic subsides, training automatically resumes from the saved checkpoints. This solution prioritizes online services to meet latency targets while continuously leveraging off-peak resources for model iteration, achieving coordinated optimization between inference SLOs and training efficiency.