A Deep Dive Into DuckLake's Sorted Tables Feature

Learn how DuckLake's sorted tables solve common lakehouse performance problems. Substack Companion Article: https://thefulldatastack.substack.com/p/understanding-ducklakes-sorted-tables This video is sponsored by MotherDuck: https://motherduck.com/ KEY TOPICS: • Solve common lakehouse performance problems with Duck Lake's sorted tables • Why query performance degrades when scanning millions of rows • How sorting data by high-cardinality columns like user IDs dramatically improves query efficiency • How parquet files use min-max statistics and row groups to enable predicate pushdown • Allow query engine to skip unnecessary files and row groups • Set sort orders with a single ALTER TABLE command • Manage the inlining threshold • Flush inline data to parquet while maintaining sort order • Use metadata tables for observability • Leverage compaction to optimize lakehouse architecture • "Set it and forget it" approach to maintaining query performance at scale • Building a new lakehouse or optimizing an existing one My Links: Substack: https://thefulldatastack.substack.com/ LinkedIn: https://www.linkedin.com/in/hoytemerson/ X: https://x.com/HoytEmerson My content company: https://www.earlysignal.tech/ Timestamps: 00:00 – Introduction 00:32 – Why sorted data matters 03:25 – Row groups, predicate pushdown, and file skipping optimization 05:10 – Duck Lake's sorting approach and demo setup 07:09 – Creating unsorted table, inserting data, explaining inlining threshold 09:55 – Inserting additional rows 10:08 – Understanding ordered vs unordered data, ALTER TABLE SET SORTED BY 11:20 – MotherDuck Sponsorship 12:36 – Practical demo 29:15 – Conclusion