Building self-service Dags with GitHub Actions at Lyft

At Lyft, a small data engineering team supports dozens of scientists and analysts who tune driver pay configurations on demand, all under SOX controls. In this episode, Ken Obata (https://www.linkedin.com/in/ken-obata/), Senior Software Engineer at Lyft (https://www.lyft.com/), joins Kenten Danas to walk through how his team built self-service Dags with an event-driven integration between GitHub Actions and Airflow, and how they solved the data modeling and consistency problems that came with it. Key Takeaways: 00:00 Introduction. 02:49 How Airflow responsibility is split at Lyft between the infrastructure org (scaling, version control, Astronomer partnership) and product teams that write Dags, maintain metadata, and orchestrate ETL and MLOps. 04:00 Contrasting data problems at large ownership-first orgs (data reconciliation) versus mid-size lean-data orgs like Lyft (data engineers as the bottleneck). 07:04 Why Lyft treats Airflow as a service layer rather than a scheduled ETL system, including productionizing Jupyter notebooks from science teams. 10:14 The experience running Airflow on Astro, including the 2 to 3 upgrade and consolidated observability in one place. 12:13 The two problems driving self-service Dags: supporting 30+ users with a small data engineering team while meeting SOX compliance on driver pay changes. 14:38 How the GitHub Actions and Airflow integration works bidirectionally, blocking PRs until validation completes. 18:28 Why Lyft chose event-driven over polling: easier upstream-to-downstream context passing and avoiding wasted compute on on-demand work. 20:23 The two hardest production challenges: GitHub's tight coupling of check IDs to commit SHAs, and multi-hour simulations racing user rebases. How Lyft built an external context store to solve both. 25:15 Why Airflow 3.3's pluggable retry policy and task state store would unlock efficiency gains for transient GitHub failures and long-running Spark simulations. 26:39 Looking ahead to task loops and agentic operators for optimization problems with multiple correct answers. Resources Mentioned: - Lyft (https://www.lyft.com/) - Apache Airflow (https://airflow.apache.org/) - Astro by Astronomer (https://www.astronomer.io/product/) - GitHub Actions (https://github.com/features/actions) Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #orchestration #Airflow