Scaling to 1,000+ Airflow Dags at Descartes / Idelic

Scaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure. In this episode, Kenten Danas talks with [Matt Stavinga](linkedin.com/in/matthew-stavinga), Data Engineer at Idelic (now part of Descartes), about how his team standardized DAG creation and logging, built a Jenkins-driven testing workflow with per-PR Airflow environments, and is opening up DAG configuration and triggering to their own customers through the Safety Suite UI. Key Takeaways: 00:00 Introduction. 01:40 Idelic's mission: collecting data from North American trucking fleets and using it to identify risky drivers so fleets can intervene before accidents happen. 02:59 Where Airflow fits: Idelic runs on Astro, with each customer/integration combination as its own DAG. 03:56 The reality of vendor integrations: APIs, SFTP, email, screen scraping, and even executables installed on customer servers to push data out. 06:11 How the DAG count grew and where things started to hurt as a small team tried to keep up. 08:49 The standardized framework: abstract methods that wrap common Airflow functions and generate task groups for each ETL phase, plus standardized logging across vendors. 11:15 CI/CD with Jenkins: per-PR Airflow deployments that QA can spin up on demand through a Jira hook and tear back down when done. 13:31 Letting customers create their own DAGs through a wizard in Safety Suite, with plans for customer-triggered runs, filtered result review, and human-in-the-loop approval. 15:37 Using the Airflow API to trigger customer DAGs, and why Idelic pushes run data into its own database instead of relying on the Airflow logs API. 18:58 What Matt wants next from Airflow: task and DAG level trend tracking, and more from Otto on self-healing DAGs. Resources Mentioned: Airflow Survey: astronomer.typeform.com/airflowsurvey26 Astro by Astronomer: astronomer.io Apache Airflow: airflow.apache.org Otto (Astronomer's data engineering agent): https://www.astronomer.io/product/otto/ Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #ai #orchestration #airflow