How to Create Scalable and Distributed Workflows with DVC and Ray

Training models at the scale of the Gemini or GPT-4 models requires advanced tools that manage complexity while ensuring efficiency. In this workshop, Mikhail Rozhkov shows us how Data Version Control (DVC) can be a game-changer for ambitious projects. DVC simplifies AI development by automating pipelines, managing versions, and tracking experiments while embracing GitOps for reproducibility. It excels in both local and cloud environments for traditional ML workflows. However, the rise of Generative AI and complex deep learning projects demands scalable, distributed training solutions. This is where the combination with Ray distributed computing possibilities steps in to supercharge your workflow. Mikhail will show how to set up the solution and how to extend it to a Ray cluster on AWS. Link to slides: https://drive.google.com/file/d/1IPheoFdG47_dGzQqYfdUAjo2hMgnodHz/view?usp=drive_link Link to blog post blog post, part 1: https://dvc.ai/blog/dvc-ray Link to blog post, part 2: https://dvc.ai/blog/dvc-ray-part-2 Link to repo: https://github.com/iterative/tutorial-mnist-dvc-ray To learn more about Iterative's open-source and SaaS tools please visit: 🧑🏽‍💻 *Our free online course:* [https://learn.iterative.ai](https://learn.iterative.ai/) ✍🏼 *Our docs:*https://dvc.org/doc (Data Version Control, Pipelines, Experiments) [https://studio.iterative.ai](https://studio.iterative.ai/) (Team Collaboration, Experiments, Model Registry) *Try out the DVC Extension for VS Code here:* https://marketplace.visualstudio.com/items?itemName=Iterative.dvc *Join the Community on our Discord server:* https://discord.gg/W49xzNmycw #dvc #machinelearning #datascience #generativeai