Flyte K8s Agent: Scalable Data Services for GNN Workflow Training

00:20 Intro and community updates 06:27 LinkedIn presentation In modern ML workflows, efficient data handling is essential for maximizing GPU utilization and accelerating training. Join us for the next community presentation where Shuying Liang will introduce the Flyte K8s agent LinkedIn is developing to orchestrate services that offload dataset loading and transformation tasks from individual training pipelines. This setup supports consistent, reusable data transformations across runs and dynamically scales to meet varying training workload demands. Built on the open-source Flyte agent framework, the Flyte K8s agent enables a decoupled, modular system architecture, allowing rapid development iterations and seamless transitions to production. At LinkedIn, this agent serves as a cornerstone in supporting scalable, adaptable data services across various training workflows, with Graph Neural Networks (GNN) as a prominent use case. This presentation delves into how the Flyte K8s agent’s general design drives productivity and operational excellence, making it an ideal solution for LinkedIn’s deep learning needs such as GNN. Everyone is welcome to join, learn, and ask questions!