Announcing StarRocks 4.0
StarRocks 4.0 is our most ambitious release yet—Get a clear look at what’s new and improved in StarRocks 4.0 — from 60% faster query performance and seamless Iceberg integration to smarter governance and lower real-time costs — and see where we’re headed next on the StarRocks roadmap. Timestamps 00:00 Intro and Agenda 02:10 StarRocks High-Level Overview: Real-Time Analytics & Data Lakehouse (Apache Iceberg, Delta Lake, and Hudi) 04:26 StarRocks 4.0 Performance 04:37 1.6× Faster on TPC-DS 1TB YoY 05:19 Core Operator Improvements: Fast Joins, Efficient Aggregations, Responsible Spill Handling 06:36 Solving the JSON Problem: JSON Evolution of StarRocks — Query JSON up to 15× Faster (No Flattening Required) 10:19 Architectural Leaps: How We’re Leveraging JSON to Be More Columnar-Based 12:00 Dictionary Encoding 12:32 ZoneMap and SortKey Indexing 14:37 Late Materialization 15:56 Data Lake Analytics 16:27 Apache Iceberg Ingestion & Maintenance in StarRocks 19:01 Partition Shuffle & Local Sort 21:16 Spill Operator to Avoid Forced Early Flush 22:11 Compaction API 22:47 StarRocks 4.0 Security and Access Control 23:13 REST Catalog Security – Traditional Limits 24:54 Catalog Layer Governance and Security 25:32 Catalog-Enforced Permissions 27:14 StarRocks 4.0 Storage and QOL 27:28 Real-Time Analytics on S3 – File Bundling, Metadata Caching, Smarter Compaction 28:75 High-Precision Decimal256 30:59 Extended Support for Multi-Statement Transactions 33:40 ASOF Join for Time-Series 35:49 Ease of Use – Node Blacklisting, Case-Insensitive Identifiers, Global Connection IDs 37:57 StarRocks 4.x Roadmap 38:13 4.x Real-Time Intelligence on the Lakehouse: Real-Time Analytics, Lakehouse, Agents, and AI Analytics 42:52 Open-Sourcing StarOS and Multi-Warehouse 44:06 Q&A 44:41 Do you have plans for supporting other Iceberg maintenance jobs besides compaction? 45:13 Are there any notable breaking changes to be aware of when moving from 3.x to 4.0? 47:14 Do you have or plan to add integration with DuckLake? 47:32 I'm interested in a detailed comparison of StarRocks 4.0 vs Apache Doris 4.0 (and possibly vs VeloDB latest version) — features, price/performance, etc. 48:25 If I understand correctly, queries on StarRocks 4.0 do not have out-of-memory errors even if they are not efficient? 49:09 So with StarRocks 4.0, can we write optimally sized files in Iceberg and also run compaction from StarRocks via the ALTER command? Is there any optimization while reading small Iceberg files? 50:07 How does this work in the case of querying from streaming platforms like Kafka, and how is it better than Apache Pinot? 51:41 Are JSON improvements only for shared-nothing architecture? 52:14 Can you do multi-statement transactions with submit task? 52:57 Has spill performance for the REGEXP operator on Iceberg tables improved? On 3.4.x and 3.5.x there were memory issues on CN. 53:45 Have there been improvements in monitoring (real-time query tracking in the UI)? 54:48 We already had case insensitivity, right? 55:22 It looks like optimizations and features are being created for the lakehouse implementation of StarRocks to improve real-time performance. With everything that has been added, how does performance compare to the shared-nothing architecture? 57:04 Are incremental materialized views intended to replace, enhance, or live alongside the existing partitioned materialized view feature? 57:45 You shared the Compaction API — does it work for Athena-Iceberg tables as well? 59:10 How are JSON queries optimized in the shared architecture? In your slides, I saw that when writing to local storage, it organizes segment files beforehand so they can be queried optimally — but what about the regular case with object store and writing data in Parquet? 1:00:37 I may have missed it, but how often does the Compaction API run? Do we set it on a cron schedule? 1:02:01 So you should write Iceberg data via StarRocks to get the benefits when querying? Writing it via Spark won’t organize it that optimal way, right? ----------------------------------------------------------------------------------------------------------------------- Learn more at https://www.starrocks.io/ Connect with us: LinkedIn: https://www.linkedin.com/company/celerdata/ Twitter: https://twitter.com/celerdata CelerData Website: https://celerdata.com/ StarRocks GitHub: https://github.com/StarRocks/StarRocks Slack: https://starrocks.io/redirecting-to-slack #DataAnalytics #DataEngineering #ApacheIceberg#RealTimeAnalytics #RealTimeData #OLAP #DataAnalyst #DataEngineer #DataInfrastructure #DataLake#Database #AnalyticalDatabase #AI #AgenticAnalytics