Apache Fluss and the Seven Deadly Sins of Streaming Analytics by Giannis Polyzos

00:00 Intro 00:38 Kafka is designed for streaming Events NOT for analytics and AI 02:37 Streaming data pipeline with Kafka 08:12 Introducing streams as queryable tables with Apache Fluss 10:17 Sin 1: Query data streams 10:45 Sin 2: Shema enforcement 11:11 Sin 3: Lack of changelogs 12:52 Sin 4: Separate log and cache 14:50 Sin 5: Network data transfer cost 17:47 Sin 6: The curse of streaming joins 20:51 Sin 7: Separation of stream/lakehouse 22:52 Key takeaways 24:15 Q&A For the last decade, Apache Kafka has been at the core of streaming data pipelines. After years in the trenches, we've learned some hard lessons and taken a fresh look at what true streaming analytics really needs. We refer to existing painpoints as the seven deadly sins of streaming analytics: 1. Complex architectures, that require multiple data copies, resulting in redundant storage costs. 2. Inability to query data streams directly 3. Complex Multi-Joins with Large States 4. Separate links for streaming and Lakehouses (think Lambda-style architectures) 5. Lakehouse data freshness - latencies can vary from multiple double-digit minutes - to hourly - to daily intervals. 6. Network Data Transmission and Costs 7. Changelog Generation, for keeping track of all the changes that happen at your events. In response to these challenges, while seeking better and modern alternatives to address more efficiently the rising demand for real-time Data & AI, Apache Fluss was created, a columnar streaming storage layer. In this presentation, we will examine modern streaming data architectures and discuss concrete use cases it empowers, like real-time dashboards, streaming ETL and customer 360. We will see how architecting around Apache Fluss helps address these ""seven deadly sins"", by introducing streams as tables that unlock new capabilities for delivering the next generation of real-time analytics, in a single unified streaming/lakehouse solution. The goal? A unified architecture that helps reduce complexity and delivers better analytics to data streams and better data freshness to data lakehouses.