Orchestrating strictly sequential ETL pipelines at Synechron

Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections. [Ivana Isailovic](linkedin.com/in/ivana-isailovic), Senior Big Data Engineer at [Synechron](synechron.com), joins Marc Lamberti to walk through the three-layer DAG architecture her team built to solve exactly that, plus how they generate 200-task DAGs and how they rebuilt subdag-style group retries in Airflow 3. Key Takeaways: (00:00) Introduction. (02:16) The stack: Snowflake source, MongoDB with a medallion (bronze, silver, gold) layout, Spark for processing, Airflow for orchestration, Elasticsearch for reports. (05:29) Why standard Airflow options (max active runs, pools, dependency setups) each solved only part of the problem. (06:40) Data consistency across bronze, silver, and gold layers is what forced strict sequential execution. (09:11) Scheduled DAGs versus event-driven DAGs triggered at any moment from the application side. (10:04) The three-layer architecture: trigger DAGs, a single proxy DAG that controls the queue, and main ETL DAGs. (12:00) The queue is literally another DAG. The proxy DAG allows only one active run and serializes everything behind it. (15:13) 200-task DAGs generated from nested task groups and YAML configuration files, with DAG versions tied to release numbers. (17:38) How the layers talk to each other: sensors and TriggerDagRunOperator. (19:44) Migrating from subdags to task groups without losing the ability to retry a whole group. (21:49) Airflow 2.9 approach: reset task instance state via the metadata DB, keyed off the task group identifier. (23:13) Airflow 3 approach: move the retry logic onto the official REST API for stability, security, and maintainability. Resources Mentioned: - [Orchestrate Everything](https://astronomer.link/data-flowcast-oe) - [Apache Airflow](airflow.apache.org) - [Snowflake](snowflake.com) - [MongoDB](mongodb.com) - [Apache Spark](spark.apache.org) - [Elasticsearch](elastic.co/elasticsearch) Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #ai #automation #airflow