Delta Lake for Rust Hacking: DataFusion Table Provider & Kernel Scan Integration (Dec 16, 2025)

Join Robert Pack and R. Tyler Croy for a focused technical session on the delta-rs ecosystem. This episode deep-dives into the "Table Provider"—the critical bridge that allows Apache DataFusion to query Delta tables using the new Delta Kernel. The conversation centers on performance trade-offs: how to handle data scanning efficiently while ensuring that filters and deletions are processed at the storage level rather than slowing down the CPU. Key Discussion Points 💡 Async Kernel Scans: Turning internal kernel iterations into smooth, async streams for DataFusion. 💡 The Projection Problem: Managing custom logic so that filtering columns don't get "lost" during query optimization. 💡 Deletion Vector Debate: Deciding whether to filter deleted rows immediately or push that logic down into the Parquet reader for better speed. 💡 Future Roadmap: A look at Row IDs, Schema Evolution, and upcoming DataFusion optimizations. delta-rs on GitHub: https://github.com/delta-io/delta-rs 0:00 Getting Started 0:30 Discussing Column Mapping and Deletion Vectors 1:58 Firefox Screen Sharing Troubles 4:40 Data Fusion Table Provider PR 9:06 Kernel Scan Implementation 15:30 Handling Predicate Pushdown 22:45 Performance Issues with Async/Blocking Tasks 41:00 Applying Deletion Vectors 53:30 Row ID Challenges in Kernel