Delta Lake for Rust Hacking: Kernelizing the Read Path (Dec 9, 2025)

Join R. Tyler Croy and Robert Pack for a deep dive into Rust development around Delta Lake. In this session, they discuss the journey to implementing the Delta Kernel for enhanced protocol support and performance. They cover the crucial transition from passing raw strings to URL types for better type safety, and the architectural shifts required to kernelize the read path to support features like deletion vectors and column mapping more efficiently. They also touch on improving DataFusion integration and optimizing memory management by reducing unnecessary clones and leveraging streaming to handle large transaction logs effectively. 🔗 delta-rs on GitHub: https://github.com/delta-io/delta-rs Chapters 0:00 - Introduction & Audio Technical Check 1:30 - PR Progress: Ripping out Badge Strings for URL Types 2:41 - The Journey to Proper Delta Protocol Support 3:42 - What Does "Kernelize" Mean for Read Path? 5:49 - Log Replay with Delta Kernel 7:03 - Understanding Log Optimization (Checkpoints/Checksums) 9:48 - Column Mapping & Deletion Vectors Challenges 13:52 - Table Providers in DataFusion 16:10 - Refactoring Delta Table for Better Ergonomics 22:15 - Improving API Clarity and Expectations 25:35 - Exploring Advanced Operations & DataFrames 33:49 - Performance Testing & Optimization Strategies 40:24 - Reducing Memory Allocations & Clones in Rust 48:12 - Using Boxed Streams for Efficient Processing