DuckLake in Production Part 3: The Maintenance Strategy

DuckLake is fun to set up, but a production-grade lake needs a maintenance strategy. In part 3 of the Production-Grade DuckLake series, I cover how to keep your catalog and S3 storage lean without breaking time travel. This video is sponsored by MotherDuck: https://motherduck.com/ You'll see why DuckLake has its own maintenance needs. Metadata and inlined data live in the catalog database, and parquet files pile up in your bucket. I also cover the two things that shape your strategy most: your ingestion pattern and your time travel policy. Then I demo a rolling 7-day time travel window in the DuckDB CLI: - Expire snapshots with ducklake_expire_snapshots, using a dry run first - Remove the underlying parquet files with ducklake_cleanup_old_files - Find and delete orphan files with ducklake_delete_orphan_files I also name the other maintenance tools: flush inlined data, merge adjacent files, rewrite data files, and CHECKPOINT. The video doesn't cover scheduling. That's in the pipelines video, which uses MotherDuck Flights. 🔗 Links - Part 1: Catalog & Storage setup: https://www.youtube.com/watch?v=G0nS-5qiGNk - Part 2: Data pipelines: https://www.youtube.com/watch?v=XjXApp5Wqdg My Links: Substack: https://thefulldatastack.substack.com/ LinkedIn: https://www.linkedin.com/in/hoytemerson/ X: https://x.com/HoytEmerson My content company: https://www.earlysignal.tech/