What's New in Apache Airflow® 3.3

Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at [Astronomer](astronomer.io), joins Kenten Danas to walk through what's new in the release and where each feature actually pays off. Key Takeaways: - 00:00 Introduction. - 01:46 The new task state store (AIP-103) lets tasks persist state across retries, so a long-running Spark job can be reattached after a worker failure instead of being duplicated on retry. - 03:46 The asset state store enables watermarking patterns: persist the last processed date or offset to an asset and resume from there on the next run. - 05:33 Why this matters for agentic workflows: resume an agent from where it left off rather than replaying every action. - 06:58 Why XComs don't solve this problem: they get reinitialized on every retry. - 09:27 Pluggable retries let you attach a retry policy to a task that branches on the exception type. Retry on transient errors, stop immediately on a 403. - 11:42 Subclassing the retry rule for more complex logic, including dynamic retry counts that used to require hacking the metadatabase. - 15:52 Updates to asset partitions in 3.3: segment-based partitioning with fan-out and roll-up mappers for downstream DAGs. - 21:42 Running tasks in Java and Go, moving Airflow toward a multi-language orchestrator. - 23:33 DAG versioning improvement: choose whether a manual rerun uses the most recent DAG version or the original version from that run. - 25:03 Advice for teams still on Airflow 2: use the upgrade ebook and Astro's AI migration tooling to handle the undifferentiated heavy lifting. Resources Mentioned: - Astronomer (astronomer.io) - Airflow 3.3 Release Notes (https://airflow.apache.org/docs/apache-airflow/3.3.0/release_notes.html) - The Task State Store (https://www.astronomer.io/docs/learn/airflow-task-state-store) - Retry policies (https://www.astronomer.io/docs/learn/rerunning-dags#retry-policies) - Multi-language support (https://www.astronomer.io/docs/learn/airflow-multilanguage) - 3.3 Webinar (https://www.astronomer.io/events/webinars/whats-new-airflow-3-3-video/) - Upgrading from Airflow 2 to 3 ebook (https://www.astronomer.io/ebooks/upgrade-from-airflow-2-to-3/) Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #aigenerated #automation #airflow