Why One Data Pipeline Can't Serve Everyone | Modern Data Engineering Pipelines

https://confluent.io | What do you do when three teams show up in the same week, all wanting the same data in three completely different shapes? You run the data pipeline at an investment bank. Employees report their personal trades so insider trading can be detected, and the pipeline is pristine — external APIs into Apache Kafka®, through Apache Flink® for enrichment, into Apache Iceberg® tables on S3. Then compliance asks for two years of trades in Tableau, the risk team needs history plus live features for an insider-trading model, and the trading platform wants pre-trade checks in fifty milliseconds. Same data, three completely different needs — and no single physical output can serve all three. In this video, Afzal from Confluent breaks down the last mile of the data engineering lifecycle: serving data. You'll learn the three consumer shapes that cover almost every client you'll ever meet — BI, ML, and operational — the two questions that characterize each one, and the single design move that solves all three: shifting compute upstream so you pay at write time, not at query time. Then we wrap the serving layer in observability and contracts — and turn a pipeline into a platform. Topics covered: - The three consumer shapes: BI (pull), ML (hybrid), operational (push) - SQL serving and pre-aggregation for BI at scale - Training-serving skew and ""define once, execute consistently"" - The latency ladder: 500 ms → 50 ms → 5 ms - Reverse ETL and the loop-back on a streaming backbone - Multi-tenant observability, schema contracts, SLAs, and failure semantics Technologies covered: Apache Kafka®, Apache Flink®, Apache Iceberg®, Apache Pinot®, ClickHouse, Redis, Tableau, PyTorch, Schema Registry 📚 CHAPTERS: 00:00 - Introduction 01:07 - The Pipeline at the Bank 02:00 - Consumer 1: Compliance Audit (BI) 03:14 - Consumer 2: Insider Trading Model (ML) 04:34 - Consumer 3: Pre-Trade Check (Operational) 06:18 - Same Data, Three Shapes 07:46 - Observability 08:46 - Contracts Questions or opinions? Drop a comment or find Afzal on the Confluent Community Slack. LEARN MORE ► Confluent Developer: https://developer.confluent.io CONNECT Subscribe, if you dare: https://www.youtube.com/@ConfluentDeveloper?sub_confirmation=1 Community Slack: https://confluentcommunity.slack.com X: https://x.com/confluentinc Linkedin: https://www.linkedin.com/company/confluent GitHub: https://github.com/confluentinc Site: https://developer.confluent.io ABOUT CONFLUENT DEVELOPER Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io. #apachekafka #apacheflink #confluent