How Modern Data Moves: AI, Data Pipelines and the Future of Data Engineering (with David Yaffe)

Today I sit down with Dave Yaffe, CEO and co-founder of Estuary, to break down the ever-evolving world of data movement and engineering. We cover the challenges of building and maintaining data pipelines, the fate of terms like "Kappa" and "Lambda" architecture, the realities of connector nightmares, and how large language models (LLMs) and AI are impacting everything from data workflows to API documentation. We’ll also explore trends like data contracts, vector database hype, and the pragmatic future of data tools. Plus what’s coming next for Estuary’s platform. Whether you’re a data engineer, an architect, or simply curious about where the industry’s heading, this conversation is packed with honest insights. Links: Estuary: https://estuary.dev/ Dave's LinkedIn: https://www.linkedin.com/in/davidyaffe/ You can find me here: LinkedIn: https://www.linkedin.com/in/hoytemerson/ Substack: https://substack.com/@hoytemerson X: https://x.com/HoytEmerson I run a boutique devrel content company for data tech startups: https://www.earlysignal.tech/ Timstamps: 00:00 - Data chaos: How are companies really handling their data? 02:02 - Data pipelines, Kappa and Lambda architecture: Are they still relevant? 08:13 - The changing language of data engineering 14:45 - Upstream data organization: Is it possible to avoid a swamp? 17:21 - Iteration matters: The necessity of re-architecting data systems 18:44 - The rise of AI & data contracts: Real solutions or new buzzwords? 20:43 - Where does AI actually help in data architecture? 26:17 - Connector horror stories: How bad can APIs really get? 29:12 - Data contracts at the source: Can we finally get agreement? 33:53 - Vector database hype: Is it really over? 37:01 - Next-gen connections: What data will we move next year? 43:35 - What’s next for Estuary in 2024? Distributed systems, skills, and untrusted Python