Building a Production DuckLake from Scratch - Catalog, Storage and Execution

Getting the catalog and storage set up for a production-grade DuckLake is shockingly easy. I walk through what you need to do and how to get things connected to your local computer using both the DuckDB CLI and the amazing DuckDB UI. Read the companion article here: https://thefulldatastack.substack.com/p/ducklake-in-production-catalog-storage This video was sponsored by MotherDuck: https://motherduck.com/ What You'll Learn: - How production DuckLake architecture differs from local deployments - Why you should use managed Postgres as your metadata layer - What your options are for your cloud storage backend - Creating and managing DuckDB secrets securely - Writing the attach statement to connect everything together - Using DuckDB UI as your team interface Tech Stack: - DuckDB 1.5.4+ (execution engine) - DuckLake (metadata management) - PostgreSQL via Superbase (catalog database) - AWS S3 (cloud storage) - DuckDB Secrets Manager (local credential management) Resources Mentioned: - DuckDB: https://duckdb.org - DuckLake Documentation: https://ducklake.select/ - Superbase: https://supabase.com My Links: Substack: https://thefulldatastack.substack.com/ LinkedIn: https://www.linkedin.com/in/hoytemerson/ X: https://x.com/HoytEmerson My content company: https://www.earlysignal.tech/ Timestamps: 00:00 - Intro 01:10 - Local vs Prod DuckLake setup 04:28 - Going through the Substack article 07:32 - Setting the scene for our use case 09:29 - Setting up Supabase is really easy 10:45 - What you need from your S3 setup 12:40 - DuckDB's secrets manager 16:26 - Explain why we make a specific schema 20:22 - Why setting a data path is important 21:22 - The ATTACH statement 23:13 - ATTACH statement in terminal 24:37 - Introducing DuckDB UI 26:32 - End credits This is part of our mini-series on creating a production-grade DuckLake. Stay tuned for upcoming videos on business logic, pipelines, security and maintenance strategy. If you found this helpful, please like and subscribe!