Real Time Analytics Has an Update Problem And the Architecture That Fixes It

🎟 Join us for NEXT QUERY 2026 | Free | Virtual | Nov 4 👉 20+ practical talks from teams building and operating data + AI infrastructure in production: https://bit.ly/4hBgFGR PhoenixAI PM Sida Shen breaks down what an update actually costs on columnar storage, why copy-on-write and merge-on-read both fall short, and the delete-and-insert architecture that keeps data freshness under 5 seconds on object storage without impacting query performance — plus live Q&A. If you serve data that keeps changing — orders, payments, inventory, risk scores — this is a practical, engineer-to-engineer look at making high-frequency updates work in production. In this video: 🌟 Why almost nothing is append-only — state transitions, corrections and backfills, recomputed derivatives. 🌟 Why staleness hurts — customer-facing analytics and AI agents that trust the data blindly. 🌟 What an update costs on columnar storage — and why in-place updates don't exist. 🌟 Copy-on-write vs. merge-on-read on Apache Iceberg — where each one pays, and why merge-on-read breaks as delete files pile up. 🌟 The delete-and-insert architecture — a primary key index that resolves updates at write time, so zone maps and predicate pushdown keep working. 🌟 A persistent primary key index on S3 — hot layer in memory, index files on object storage. 🌟 Production hardening — P99-first benchmarking across 1,000+ tables; ~5-second P99 writes with PhoenixAI 4.1. 🌟 Real production numbers — an e-commerce seller's P99 writes down to a stable 5 seconds; Fanatics running 6 PB and ~1B events a day after retiring Apache Druid and cutting Snowflake usage 95%. 🌟 Where PhoenixAI fits — one SQL layer over real-time and lakehouse data, BYOC on AWS, Azure, and GCP. ------------------------------------------------------------------------------------------------------------------- Timestamps 00:00 Welcome & agenda 02:04 Data changes: almost nothing is append-only — payments, orders, inventory, corrections, backfills, recomputed scores 03:32 Most "real-time" engines only do append-only, and that's a small subset of real workloads 04:26 Why staleness hurts: customer-facing analytics and AI agents that trust data blindly (Coinbase example) 05:57 What an update actually costs on columnar storage: there's no such thing as an in-place update 08:21 Copy-on-write vs. merge-on-read on Apache Iceberg: pay on the write side or pay on the read side 11:16 Why merge-on-read breaks in production: delete files pile up and compaction never catches up 12:15 The architecture that works: under-5s freshness, pure columnar, persisted on object storage 14:25 PhoenixAI overview: fresh, fast, multi-table joins at scale, shared-data, BYOC 16:47 Primary Key table: delete-and-insert, paying for updates at write time with an O(1) index lookup 19:08 A persistent primary key index on S3: hot layer in memory, index files prebuilt during compaction 21:24 Streaming integrations: Kafka, Flink CDC, partial & conditional updates 22:30 What this architecture costs: index lookups, deletion vectors, compaction 23:01 Production hardening: P99-first testing, a 1,000-table benchmark suite, ~5-second P99 writes in 4.1 25:32 Case study: e-commerce backfills & month-end corrections at a stable 5-second P99 26:49 Case study: Fanatics — 6 PB, ~1B events a day, Apache Druid retired, Snowflake usage cut 95% 28:03 Key takeaways: four things to remember about real-time updates 29:13 Try PhoenixAI free: 30-day trial, no credit card required 29:29 Live Q&A begins 29:34 Q1: Where do deletion vectors live, and are they expensive to maintain? (A: a bitmap — cheap to read and update) 30:18 Q2: How do you control query parallelism in shared-data vs. shared-nothing? (A: node-to-data mapping stays the same, plus a cache layer) 31:33 Wrap-up & how to reach Sida ------------------------------------------------------------------------------------------------------------------- Learn more at https://www.phoenixdata.ai/ Connect with us: LinkedIn: https://www.linkedin.com/company/phoenixai-data/ Twitter: https://x.com/phoenixdataai CelerData Website: https://www.phoenixdata.ai/ StarRocks GitHub: https://github.com/StarRocks/StarRocks StarRocks Website: https://www.starrocks.io/ Slack: https://starrocks.io/redirecting-to-slack #RealTimeAnalytics #RealTimeData #DataEngineering #DataInfrastructure #DataArchitecture #OLAP #Database #ApacheIceberg #Lakehouse #StarRocks #OpenSource #StreamingData #ApacheKafka #CDC #DataEngineer #AIAgents