Orchestrating cost-efficient data pipelines at Insignis

Running Airflow with a one-person data engineering team means every decision about cost, tooling, and best practices has to earn its keep. In this episode, Kenten sits down with Aniruddha Sengupta (https://www.linkedin.com/in/aniruddha-sengupta-2ab207125/), Staff Data Engineer at Insignis (https://www.insigniscash.com/), to talk through how the UK cash-savings fintech runs a low-cost, scalable Airflow platform. The conversation covers observability, DAG-level best practices, the internal tools Insignis built to control costs, and where AI fits into the roadmap. Key Takeaways: 00:00 Introduction. 02:22 Insignis data team as a "newspaper" for the business. Aniruddha describes the data team's role: consuming data from operational systems and external sources like Salesforce and Ringover, and delivering insight without owning business logic. 08:07 Airflow as the orchestrator, not the executor. Airflow sits at the heart of the platform, orchestrating SQL pipelines, API ingestion, and even acting as an observability layer over real-time pipelines built elsewhere. 12:19 Observability as definition of done. Why a green Airflow UI is not observability, and how Insignis builds independent dashboards to catch silently dropped data. 14:38 Choosing managed Airflow. How abstracting away the scheduler, workers, and database frees a small team to focus on code and documentation. 20:04 DAG patterns that keep costs down. CI/CD enforcement, failure callbacks, mindful DAG boundaries, and periodic purging of logs and metadata. 22:06 Tariff and Maestro. Two internal tools Insignis built: Tariff to surface SQL pipeline costs, and an AI-powered tool called Maestro to recommend worker, scheduler, and database sizing. 25:40 Exploring AI with Airflow. MLOps use cases around client behavior and the appeal of self-healing pipelines. 28:20 Wishlist for Airflow. Event-based DAG parsing so code changes surface in the UI closer to instantaneously. Resources Mentioned: Annual Airflow survey (https://astronomer.typeform.com/airflowsurvey26) - Insignis (https://www.insigniscash.com/) - Apache Airflow (https://airflow.apache.org/) - Astronomer (https://www.astronomer.io/) - Building a Low-Cost Scalable Airflow Platform for Small Teams (https://airflowsummit.org/sessions/2026/building-a-low-cost-scalable-airflow-platform-for-small-teams/) - Google BigQuery (https://cloud.google.com/bigquery) Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow