DuckLake: The Lakehouse That's Just SQL and Parquet - Talk Python to Me Ep. 562
How many files does your query read before it reads any data? On some data lakes, you go through JSON and metadata files first, just to learn which Parquet files matter. DuckLake asks one SQL question instead. The metadata lives in a real database. The data stays in plain Parquet. That's the entire format. Pedro Holanda joined DuckDB in 2018, when it was still a research prototype at CWI. He's the lead DuckLake developer. Guillermo Sanchez Dionis works on DuckLake and the new Quack protocol. With Quack as the catalog, DuckLake handles 200 transactions a second under heavy contention. No other open table format comes close. ▬▬▬▬ About the podcast ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ This video is the uncut, live recording of the Talk Python To Me podcast ( https://talkpython.fm ). We cover Python-focused topics every week and publish the edited and polished version in audio form. Subscribe in your podcast player of choice (100% free) at https://talkpython.fm/subscribe. ▬▬▬▬ Guests ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Guillermo Sanchez Pedro Holanda ▬▬▬▬ Links and resources from the show ▬▬▬▬▬▬▬▬▬▬▬▬ PhD on progressive indexes: https://ir.cwi.nl/pub/31048 SQLite: https://www.sqlite.org Litestream: https://litestream.io boring hardware: https://talkpython.fm/episodes/show/531/talk-python-in-production DuckDB: https://duckdb.org episode 491: https://talkpython.fm/episodes/show/491/duckdb-and-python-ducks-and-snakes-living-together Iceberg: https://iceberg.apache.org manifesto: https://ducklake.select/manifesto/ DuckLake: https://ducklake.select spec: https://ducklake.select/docs/stable/specification/introduction this diagram: https://blobs.talkpython.fm/ducklake-architecture.png?cache_id=13839f Data inlining: https://ducklake.select/2026/04/02/data-inlining-in-ducklake/ ducklake-dataframe: https://github.com/pdet/ducklake-dataframe Polars course: https://training.talkpython.fm/courses/polars-for-power-users CSV parser: https://duckdb.org/2025/04/16/duckdb-csv-pollock-benchmark.html Zero-copy Arrow: https://duckdb.org/2021/12/03/duck-arrow.html ART index: https://duckdb.org/2022/07/27/art-storage.html async I/O: https://duckdb.org/2026/07/31/asynchronous-io v1.0: https://ducklake.select/2026/04/13/ducklake-10/ Git-like branching: https://ducklake.select/roadmap.html ▬▬▬▬ Dive deeper ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Listen to the Talk Python To Me podcast at https://talkpython.fm and explore over 275 hours of Python courses at https://training.talkpython.fm/courses. ▬▬▬▬ Follow us on Social ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Mastodon: Michael - https://fosstodon.org/@mkennedy & Talk Python - https://fosstodon.org/@talkpython Bluesky: Michael - https://bsky.app/profile/mkennedy.codes & Talk Python - https://bsky.app/profile/talkpython.fm X: Michael - https://x.com/mkennedy & Talk Python - https://x.com/talkpython