DuckLake in Production Part 2: Build Pipelines and Business Logic (Using AI)

In this video, we dive into Stage 2 of building a production DuckLake where we move data into the lakehouse using AI-written pipelines and Motherduck Flights. This is where a lakehouse becomes more than just an expensive folder! This video is sponsored by MotherDuck: https://motherduck.com/ Companion Substack Article: https://thefulldatastack.substack.com/p/ducklake-in-production-catalog-storage In This Video: 1. Setting Up Source Data — Using Parkbench to generate realistic test data and populate Postgres 2. Bronze Layer Pipeline — Extract data from Postgres and load it into DuckLake using MotherDuck Flights 3. Gold Layer Transformation — Clean and normalize JSON data, unnest complex fields for analytical use 4. Metrics Layer — Build business metrics from the gold layer 5. ML Feature Engineering — Create feature-engineered datasets with one-hot encoding and other ML-ready transformations Key Technologies: - DuckDB + DuckLake (open-source data lakehouse) - Motherduck (managed DuckDB with Flights scheduling) - AWS S3 (storage) - Supabase Postgres (source data) - Claude Cowork (pipeline generation and debugging) - Parkbench (benchmarking and test data generation) Links: MotherDuck Flights: https://motherduck.com/product/flights/ Parkbench: https://github.com/early-signal-tech/parkbench My Links: Substack: https://thefulldatastack.substack.com/ LinkedIn: https://www.linkedin.com/in/hoytemerson/ X: https://x.com/HoytEmerson My content company: https://www.earlysignal.tech/ 0:00 Intro - The importance of data pipelines 1:23 The 3 stages of a production DuckLake 2:24 Use case: SMB data team on AWS 4:34 Setting up source data with Parkbench 9:18 Bronze layer pipeline architecture 10:37 Motherduck Flights introduction & AI integration 12:41 Using Motherduck MCP with Claude 14:12 Performance benchmarks (2x speed improvement) 16:20 Bronze layer results & table inspection 17:12 Gold layer transformation pipeline 23:19 Conclusion & Stage 3 preview