Next Gen Change Data Capture: Streaming Postgres with Estuary

Change Data Capture (CDC) has a reputation for being complicated but it doesn't have to be. In this video, I break down what CDC actually is (hint: it's just your database's write-ahead log being tailed in real time, no extra queries required), then walk through a full hands-on setup using Estuary to stream changes from a local Postgres database straight into MotherDuck. This video is sponsored by Estuary, a new content partner I'm working with. We'll be doing more videos together digging into CDC and streaming pipelines, so if that's a topic you're into, stick around. https://estuary.dev/ What we cover: - What change data capture actually is, and why it beats batch snapshotting - How CDC reads a database's internal log (Postgres WAL / MySQL binlog) instead of querying it - Setting up a local Postgres instance with Docker Compose that streams data continuously - Using ngrok to expose the local Postgres instance for Estuary to connect to - Configuring an Estuary Postgres capture end to end - Materializing the captured data into MotherDuck - Watching inserts land in MotherDuck in real time, straight from the local Postgres log Timestamps: 0:00 – What is change data capture? 1:01 – Batch snapshots vs. CDC (write-ahead logs & bin logs explained) 1:52 – Introducing today's sponsor: Estuary 2:56 – Walking through Estuary's Postgres capture example 3:59 – Setting up ngrok for the local Postgres tunnel 4:22 – Spinning up the local Postgres with Docker Compose 6:05 – Creating an Estuary capture from Postgres 9:00 – Testing, saving, and publishing the capture 10:09 – Setting up the MotherDuck materialization 12:50 – Testing and publishing the materialization 13:50 – Querying live streamed data in MotherDuck 15:12 – Wrap-up and what's coming next with Estuary Links: - Estuary Postgres capture example (GitHub): https://github.com/estuary/examples/tree/main/postgres-simple-capture - My setup guide with the extra "gotchas" not covered in the repo: https://github.com/early-signal-tech/estuary-examples/blob/main/postgres-simple-capture-setup.md My Links: Substack: https://thefulldatastack.substack.com/ LinkedIn: https://www.linkedin.com/in/hoytemerson/ X: https://x.com/HoytEmerson My content company: https://www.earlysignal.tech/ If you enjoyed this video, please like and subscribe — more videos with Estuary on CDC and streaming pipelines are coming soon. You can also catch upcoming live streams with the Estuary team on Substack and X, linked below.