Iceberg REST Catalog in Docker: Full Local Lakehouse Setup

Spark, Trino, and DuckDB reading and writing the same Iceberg table on one laptop — with Lakekeeper as the REST catalog refereeing every commit. 🚀 Check Out My Data/AI Courses: https://whop.com/the-data-guy-llc/ Use Code dataguysub for 25% off! 🚀 Get Source Code and Bonus Content: https://patreon.com/TheDataGuy?utm_medium=unknown&utm_source=join_link&utm_campaign=creatorshare_creator&utm_content=copyLink ⚡ Want to work together? Book time on my website: https://thedataguygeorge.com/ In this video I build a complete local lakehouse stack with Docker Compose: Lakekeeper (a Rust-based Iceberg REST catalog) backed by Postgres, MinIO for S3-compatible object storage, and three query engines all wired to the same warehouse. You'll see how the Iceberg REST catalog spec lets any engine talk to any catalog, how credential vending keeps storage secrets in exactly one place, and what actually happens when two writers commit to the same table at the same time — optimistic concurrency, automatic retries, and a clean linear history with zero lost data. I also walk the catalog landscape (Apache Polaris, Unity Catalog, Nessie, and the managed options) and cover what changes when you take this setup to production. ⏱️ Chapters 0:00 - Intro: why your lakehouse needs a catalog 1:00 - Storage, compute, catalog: the three layers 1:42 - The two-writer disaster story 2:00 - The Iceberg REST catalog spec 2:29 - Catalog options: Polaris, Unity, Nessie 3:34 - Why Lakekeeper for this build 4:10 - Docker Compose: Lakekeeper, Postgres, MinIO 5:31 - Spark config and credential vending 6:35 - Defining the warehouse for MinIO 7:43 - Bootstrapping and the Lakekeeper UI 8:50 - First queries from Spark 10:30 - Wiring up Trino and DuckDB 11:43 - Write in DuckDB, read in Spark instantly 13:00 - Concurrent writers vs the catalog referee 15:27 - Production notes and wrap-up 🔗 Links Lakekeeper: https://lakekeeper.io Apache Iceberg: https://iceberg.apache.org MinIO: https://min.io Trino: https://trino.io DuckDB: https://duckdb.org Apache Spark: https://spark.apache.org Apache Polaris: https://polaris.apache.org Nessie: https://projectnessie.org #ApacheIceberg #DataEngineering #Lakekeeper