Databricks and Apache Spark in a Nutshell | Workshop
FREE workshop for beginner data engineers: Apache Spark and Databricks in one session. Vlad Meshkov (Databricks Architect) and Dmitry Anoshin cover theory and hands-on: what Spark is, why it exists, how it compares to Snowflake and BigQuery, then a full Bronze → Gold pipeline in Databricks with Unity Catalog and Power BI. What you'll learn • Theory: Apache Spark and Databricks intro, history (Google → Hadoop → Spark), in-memory execution, DAG, driver/executor, RDD vs DataFrame, transformations vs actions. • Theory: Redshift vs Snowflake vs BigQuery (shared-nothing vs shared storage, warehouses vs slots). • Theory: Databricks platform, clusters (all-purpose, job, pool, serverless), storage (DBFS, external, Unity Catalog Volumes). • Hands-on: Connect GitHub to Databricks, create Unity Catalog Volume, upload JSON. • Hands-on: Bronze notebook — read JSON, write Parquet. • Hands-on: Gold notebook — read Parquet, write Delta, register table in Unity Catalog. • Hands-on: SQL editor and query gold_events; Power BI connection. Timestamps 1:11 — Workshop start and intro (Dmitry + Vlad) 2:24 — What is Apache Spark and Databricks 9:11 — History: Google GFS, MapReduce, BigTable → Hadoop → Spark 13:41 — Apache Spark: in-memory, 20x faster, Berkeley Lab 15:09 — Spark architecture: DAG, driver, executor, Tungsten, Photon 23:17 — Redshift vs Snowflake vs BigQuery 36:28 — Vlad: Databricks intro — compute, one platform 45:49 — Cluster types: all-purpose, job, pool, serverless 57:03 — Storage: DBFS, external, Unity Catalog Volumes 1:00:24 — Connect GitHub repo to Databricks 1:05:25 — Unity Catalog Volume, upload JSON 1:13:24 — Bronze notebook: JSON → Parquet 1:17:01 — Gold notebook: Parquet → Delta 1:18:05 — Register table in Unity Catalog (gold_events) 1:20:18 — SQL editor: query gold_events Resources mentioned • GitHub repo — Databricks in a Nutshell (Bronze → Gold, Unity Catalog, Power BI): https://github.com/45825243/main • Related video — Databricks Task Creation with Local IDE: Build & Deploy Python Wheel Package: https://youtu.be/Qt9OYHiHL0c (Set up workspace, connect GitHub, upload data to DBFS, develop PySpark ETL locally, create Python Wheel .whl, upload to cluster, automate with Databricks Workflows.) • Repo & solution for wheel project: https://github.com/surfalytics/data-projects/tree/main/de-projects/20_wheel_in_databricks • Learning Spark (O'Reilly, free): for open-source Spark • Surfalytics: FREE Data Engineering course and workshops Key takeaways • Spark is compute only; Databricks adds platform, Delta Lake, Unity Catalog. • Transformations are lazy; actions trigger execution. • Redshift = shared-nothing MPP; Snowflake = shared storage + warehouses; BigQuery = serverless, pay per query. • Use serverless when possible; job clusters for predictable production. • Bronze → Gold: Volume → Parquet (Bronze) → Delta (Gold) → Unity Catalog table → SQL / Power BI. • Work in Git/Repos for versioning and production runs from main. #databricks #apachespark #pyspark #dataengineering #lakehouse #unitycatalog #bronze #gold #etl #snowflake #bigquery #redshift #surfalytics #freecourse #beginner