Delta Lake for Rust Hacking: S3 Directory Buckets & Object Store Fixes (Feb 3, 2026)

Join Robert Pack and R. Tyler Croy for a technical deep dive into the integration between delta-rs and AWS S3’s newer "Directory Buckets" (Express One Zone). The session focuses on a critical challenge: handling unordered file listings in S3 and how that impacts transaction log replays in Rust. Key Discussion Points 🔹 S3 Directory Buckets: Exploring the trade-offs of Express One Zone storage—blazing fast performance, but with unordered listing behavior that complicates log processing. 🔹 Kernel vs. Delta-rs API: A debate on where to place listing logic—adding surface area to the Kernel or using extension traits in delta-rs. 🔹 Live Implementation: The team works through a fix to auto-detect Directory Buckets via the URL and resolves boolean comparison bugs in the test suite. 🔹 Roadmap: Moving more operations into DataFusion and preparing for the next crate release. 🔗 GitHub: https://github.com/delta-io/delta-rs 0:00 Optimizing Commit Log handling 12:25 Implementing Deletion Vectors for data modifications 22:20 Enhancing Predicate Pushdown for query performance 34:00 Removing technical debt in Column Mapping 46:30 Metrics Collection: Balancing accuracy vs. performance 51:30 Deep Dive: Drastically reducing memory allocations