Databricks AI Dev Kit + Claude Code: Vibe Coding
In this video, I show you how to use the Databricks AI Dev Kit from Databricks Field Engineering to connect Claude Code (running inside Kiro) to your Databricks Free Edition workspace using MCP (Model Context Protocol) and AI Skills — no enterprise subscription needed. You'll see a real, end-to-end demo: a single prompt builds a full Medallion Architecture (Bronze → Silver → Gold) e-commerce pipeline, complete with catalog creation, SQL notebooks, a Databricks Job with 3 sequential tasks, data quality fixes, and a Genie Space on the gold dataset. 🔑 What you'll learn: ✅ What the Databricks AI Dev Kit is and how Skills + MCP work together ✅ How to install the dev kit locally with uv and Databricks CLI ✅ How to connect Claude Code (in Kiro) to your Databricks workspace ✅ How to prompt an AI agent to build a Medallion pipeline from scratch ✅ Free Edition limitations and how to work around them (SQL notebooks vs Python) ✅ How to create a Databricks Job with 3 tasks and run it end-to-end ✅ How to validate Bronze, Silver, Gold tables and check data quality metrics ✅ Quick look at Genie Space from the gold dataset ⏱️ CHAPTERS: 00:00 Intro and Goal 01:03 Prereqs and Setup Links 01:41 How AI Dev Kit Works (Skills vs MCP analogy) 04:14 Medallion Architecture Primer 05:20 AI Dev Kit vs Genie Code 07:45 Free Edition Limitations 09:29 Install Dev Kit Locally 12:36 Run Installer and Create CLI Profile 16:46 Open in Kiro and Claude Code 20:05 Skills and MCP Tools Tour 25:21 Test MCP Queries in Databricks 28:21 Medallion Pipeline Prompt Plan 30:39 Prompting the Pipeline 34:53 Catalog and Schema Setup 36:15 Upload Notebooks to Workspace 38:18 Job Failures and Auto-Fixes 40:38 Pipeline Summary and Validation 42:50 Why the Job Failed (Free Edition gotcha) 45:51 Rebuild as SQL Notebooks 46:59 Successful Job Run 48:49 Genie Space and Next Steps 52:21 Wrap Up and Takeaways 🔗 LINKS: → Databricks AI Dev Kit (GitHub): https://github.com/databricks-solutions/ai-dev-kit → Databricks Free Edition signup: https://databricks.com/try → Claude Code: https://claude.ai/code → Kiro IDE: https://kiro.dev → uv (Python package manager): https://docs.astral.sh/uv → Databricks CLI: https://docs.databricks.com/dev-tools/cli → Prompts gist: https://gist.github.com/sudarshan-koirala/0315ce3d066ea410c957b73e548f2302 📌 PREREQUISITES: - Databricks Free Edition account (free, no credit card) - Claude Code installed - Kiro or VS Code - uv and Databricks CLI installed locally --------------------------------------------------------------------- ☕ Buy me a Coffee: https://ko-fi.com/datasciencebasics ✌️Patreon: https://www.patreon.com/datasciencebasics ------------------------------------------------------------------------------------------ 🤝 Connect with me: 📺 Youtube: https://www.youtube.com/@datasciencebasics?sub_confirmation=1 👔 LinkedIn: https://www.linkedin.com/in/sudarshan-koirala/ 🐦 Twitter: https://twitter.com/mesudarshan 🔉Medium: https://medium.com/@sudarshan-koirala 💼 Consulting: https://topmate.io/sudarshan_koirala #databricks #databricksaidevkit #claudecode #datasciencebasics