Transforming Postgres Streaming Data into S3 with Estuary Derivations

In this video, I add a derivation to my Postgres → Estuary → S3 Parquet pipeline. A derivation transforms data as it streams through Estuary, so I convert a timestamp column into a date without touching the source. We set up flowctl, write the derivation in SQL, pick a primary key, hit a type error, and use Claude Code with the Estuary skill to fix it. We also cover a gotcha where only the key column came through, and how the fix comes together. This video is sponsored by Estuary: https://estuary.dev/ Previous Estuary Videos: CDC Intro: https://www.youtube.com/watch?v=_tgiCopf8aY Unbundling a Data Warehouse: https://www.youtube.com/watch?v=mtq63fDyqtA My Links: Substack: https://thefulldatastack.substack.com/ LinkedIn: https://www.linkedin.com/in/hoytemerson/ X: https://x.com/HoytEmerson My content company: https://www.earlysignal.tech/ Timestamps: 00:00 Intro 01:00 Estuary sponsorship 01:14 Review the project so far 02:00 Intro to flowctl 03:25 Create Derivation Draft 05:15 flowctl auth login 05:43 Start derivation 06:24 Develop derivation draft 09:05 Choose primary key 1303 Give errors to an agent 16:46 Check published derivation 18:08 Going over a "gotcha" issue 19:50 Agents come in to save the day 22:18 Outro