Tutorial: Genie for Data Engineers, Data Scientists, and Everybody Else

This video provides an overview of our latest Genie tutorial for data engineers and data scientists seeking hands-on experience with Genie. The tutorial uses a real-world use case. You work with almost 700 million rows of flight data from the OpenSky Network. You start with raw IoT data and finish with a live web app. Genie writes the SQL, the pipeline, and the app code for you. The tutorial and the data are free. You can complete all the steps in the Databricks Free Edition. Git link to tutorial: https://github.com/databricks/tmm/tree/main/DSDE-Genie-Tutorial 0:00 Intro: Our newest Genie tutorial 0:16 The Data Set and the Journey: From 700 million raw rows to a live web app, with Genie writing the code 0:51 The Genie family: Genie One, Genie Agents, Genie Code, and Genie ZeroOps 1:30 The data: real IoT data from the OpenSky Network (54,000 aircraft, one full day) 1:52 Module 1: Get the OpenSky dataset from Databricks Marketplace 2:18 Module 2: Use Genie Agents and Genie Code for exploratory data analysis (EDA) 2:48 Module 3: Use Genie Agents to explore and visualize the data 3:16 Module 4: Use Genie Code to build a Spark Declarative Pipeline from one prompt 3:47 Module 5: Use Genie Code to orchestrate the pipeline with Lakeflow Jobs 4:18 Module 6: Use Genie One Mobile on your phone 4:42 Module 7: Build a Databricks App with Genie Code and other coding agents 5:20 Module 8: OpenSharing with VSCode 5:42 Module 9 Reference architecture for end-to-end workflow