Delta Lake for Rust Hacking: Performance Deep Dive (Dec 14, 2025)

Join us for a deep-dive technical session into the internals of Delta Lake for Rust. In this hacking session, we focus on squeezing every ounce of performance out of the DataFusion engine, tackling long-standing technical debt, and refining how we handle large-scale data modifications. We cover everything from the nitty-gritty of memory allocation to the architectural shifts required for full Deletion Vector support. 🔍 Key Discussion Points 🔹 Performance Optimization: We explore techniques to drastically reduce memory allocations during log processing to ensure the engine stays lean and fast. 🔹 Column Mapping & Deletion Vectors: A look at our roadmap for removing technical debt to provide robust support for data modifications and schema evolution. 🔹 The Metrics Balance: How to collect high-fidelity execution metrics without introducing performance regressions, specifically across the Rust/Python boundary. 🔗 delta-rs on GitHub: https://github.com/delta-io/delta-rs 0:00 - Double Encoding in Commit Logs 2:10 - Reviewing the Protocol Specification 7:32 - Table Provider Enhancements in DataFusion 12:25 - Implementation of Deletion Vectors and Performance Constraints 14:40 - Lazy Statistics Model and Optimization 22:20 - Handling Projection and Predicate Pushdown 34:00 - Column Mapping and Tech Debt Removal 46:30 - Metrics Collection in Rust and Python 51:30 - Performance Metrics: Memory Allocation Reduction