Machine Learning with ADBC, DuckDB & XGBoost: No Pandas, Just Pure Arrow Tables
In this video, I walk through an interesting use case combining ADBC (Arrow Database Connectivity), DuckDB, and XGBoost to build a machine learning classification model, all without using traditional dataframes! This video is made in partnership with Columnar, creator of the dbc cli tool that lets you easily manage your ADBC drivers as well as fantastic quick starts for ADBC. dbc tool: https://docs.columnar.tech/dbc/ ADBC agent skills: https://console.columnar.tech/agent-skills columnar: https://columnar.tech/ 🔑 What You'll Learn: - How to use ADBC drivers to connect to DuckDB and work with Arrow tables - Setting up DuckLake for data storage and retrieval - Pre-processing data using pure SQL in DuckDB (no Pandas required!) - Building a train/test split function using PyArrow Compute API instead of scikit-learn - Training an XGBoost classification model with Arrow tables as input 🛠️ Technologies Used: - ADBC (Arrow Database Connectivity) - DuckDB & DuckLake - PyArrow & Arrow Compute API - XGBoost - Python context managers for clean database connections 💡 Why This Matters: ADBC standardizes database connectivity across different systems, and Arrow tables provide efficient, zero-copy data transfer between systems. This approach keeps dependencies minimal while maintaining high performance, perfect for modern data workflows and AI agent development. 🎯 Who Is This For: - Data engineers looking to optimize their ML pipelines - Machine learning practitioners interested in Arrow-native workflows - Anyone curious about DuckDB, ADBC, or modern data processing techniques Machine learning isn't dead; it's evolving! While everyone's focused on AI engineering, ML fundamentals are still crucial. 🔗 Links & Resources: - My Substack Article: https://open.substack.com/pub/thefulldatastack/p/training-xgboost-ducklake-adbc?utm_campaign=post-expanded-share&utm_medium=web - Intro to ADBC video: https://youtu.be/qjK4mb6sZ2E?si=-T3tiWkTRqH7yver - Previous DuckLake video: https://youtu.be/0B0J-4aXfKc?si=5Q-Fg0bgRjS6U0n0 - Connect on LinkedIn: https://www.linkedin.com/in/hoytemerson/ - I run a bespoke devrel content company called Early Signal: https://earlysignal.tech/ ⏱️ Timestamps: - 0:00 - Introduction to ADBC & the use case - 2:30 - DuckDB and DuckLake setup - 8:45 - SQL-based data preprocessing - 12:05 - ADBC context managers & Arrow tables - 15:20 - Train/test split with PyArrow Compute - 17:40 - Training the XGBoost model - 19:25 - Results & model saving If you enjoyed this video, please like, subscribe, and share! More ADBC content coming soon. 🏷️ Tags: #MachineLearning #ADBC #DuckDB #XGBoost #PyArrow #DataEngineering #Python #ArrowTables #DuckLake #MLOps #DataScience