Managing travel platform data with Airflow at Headout

Running a travel platform means dealing with fast-moving inventory, real-time fraud detection, and heavy performance marketing attribution, all while keeping data trustworthy for a data-driven org. In this episode, Mrinalini Singh, Data Platform Engineer at [Headout](headout.com), walks through how her team uses Airflow as the nervous system of their stack: orchestrating dbt with a write-audit-publish pattern, running ML training and inference, and wiring up alerting that points to the exact commit that broke a DAG. Key Takeaways: - 00:00 Introduction. - 01:05 What a data platform engineer does at Headout, and the hub-and-spokes model where analysts and scientists write their own dbt models. - 04:11 The specific data challenges of a travel platform: fast-changing inventory, real-time fraud analytics, and performance marketing attribution. - 05:40 Where Airflow sits in the stack, from ingestion to transformation to serving. - 07:01 The write-audit-publish dbt pattern and why slightly stale data beats wrong data. - 09:55 Why Headout uses a custom Python operator instead of the dbt provider or Cosmos, reading the dbt manifest to build task groups per model. - 13:45 ML use cases on Airflow: Feast feature store, model training, inference, and data/feature drift tracking. - 17:00 Custom Slack failure hooks that stitch together Airflow logs, GitHub commit URLs, and teammate Slack IDs. - 20:04 A zombie task incident that filled the metadata DB, caused locking issues, and drove the move to Grafana-based monitoring. - 22:26 Using AI to generate Airflow code, encoding internal patterns as a skill file, and running an AI reviewer bot on every PR. Resources Mentioned: - [Orchestrate Everything](https://astronomer.link/data-flowcast-oe) - [Headout](headout.com) - [dbt](getdbt.com) - [Cosmos](github.com/astronomer/astronomer-cosmos) - [Feast](feast.dev) - [Apache Flink](flink.apache.org) - [Grafana](grafana.com) Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #ai #automation #apacheairflow