ADBC Intro: Stream Data from PostgreSQL to DuckDB in Python | Modern Database Connectivity Explained

Learn how to use Apache Arrow Database Connectivity (ADBC) to revolutionize the way you connect to databases! In this comprehensive tutorial, I'll show you how ADBC provides a modern alternative to JDBC and ODBC, specifically designed for columnar data and analytics workloads. This video about ADBC is made in partnership with Columnar, creator of the dbc cli tool that lets you easily manage your ADBC drivers as well as fantastic quick starts for ADBC. Columnar: https://columnar.tech/ ADBC Quick Starts: https://github.com/columnar-tech/adbc-quickstarts dbc docs: https://docs.columnar.tech/dbc/ 🔥 What You'll Learn: - Why traditional database connectivity tools (JDBC/ODBC) fall short for modern analytics - How ADBC leverages Apache Arrow for zero-copy data transfer - Bulk ingesting data into DuckDB using Polars and Arrow - Streaming data from PostgreSQL to DuckDB efficiently - Real-world use cases for ADBC in data engineering 💡 Key Highlights: - No serialization/deserialization overhead - Works seamlessly with columnar databases - Perfect for analytics and in-memory data processing - Compatible with multiple data sources simultaneously 📚 Resources Mentioned: - Matt Toppel's "In-Memory Analytics with Apache Arrow" book: https://www.amazon.com/Memory-Analytics-Apache-Arrow-hierarchical/dp/1801071039 - Code examples and scripts available 🔗 Links: - Substack Article: https://open.substack.com/pub/thefulldatastack/p/adbc-an-intro-to-nextgen-database?utm_campaign=post-expanded-share&utm_medium=web - Early Signal (Bespoke Technical Content): https://www.earlysignal.tech/ Timestamps: 0:00 - Introduction to Database Connectivity Problems 0:57 - Why JDBC/ODBC Don't Work Well for Analytics 2:30 - Introduction to Apache Arrow and ADBC 5:00 - Setting Up the Development Environment 11:45 - Bulk Ingesting Data into DuckDB 14:20 - Code Walkthrough: ADBC Ingest Function 17:02 - Testing the Bulk Ingest Script 19:15 - Streaming Data from PostgreSQL Setup 21:58 - Using Record Batches for Streaming 25:34 - Use Cases and Future Possibilities with ADBC If you found this helpful, please like and subscribe for more data engineering content! #ADBC #ApacheArrow #DuckDB #Python #DataEngineering #PostgreSQL #Analytics #DatabaseConnectivity #ModernDataStack