Hands-on lab | AWS Serverless Data Lake | Module 5.8 | Surfalytics
Hands-on: build a serverless data lake on AWS with S3, Glue (crawler, catalog, jobs, notebooks), and Athena. Use real data, fix real issues, and see how to do the same with infrastructure as code next. 📚 What you'll learn Theory / setup • Serverless data lake: S3 + Glue + Athena; CloudFormation template overview • Glue crawler and Data Catalog; Athena (Presto/Trino) and query result location • Plan: raw data in S3 → crawler → catalog → Athena → Glue transform job → transform bucket • Data: New York yellow taxi trip data (Kaggle) in S3 Hands-on • Create and run Glue crawler; review catalog table; fix duplicate columns • Set up Athena; run queries; use CTAS for SQL-only transformation • Glue Studio visual ETL (source, transform, filter, Parquet target) vs script-based job • Script-based Glue PySpark job; troubleshoot IAM (PassRole) and re-run • Glue notebooks for development; architecture recap (medallion, QuickSight) • QuickSight: connect Glue catalog, Spice; next steps (dbt, Redshift, Databricks) ⏱️ Timestamps 0:00 – Intro and AWS vs Azure/GCP options 2:19 – Serverless data lake and CloudFormation 3:04 – CloudFormation overview (VPC, S3, Glue) 6:06 – Plan: crawler → catalog → Athena 7:46 – New York taxi data in S3 9:03 – Create Glue crawler 14:01 – Run crawler; review catalog 19:46 – Add Athena; set output; run queries 23:41 – Fix table; preview data 31:12 – Plan Glue transform job 32:10 – Glue Studio visual ETL 41:54 – PySpark script vs visual 48:59 – Run job; troubleshoot 50:42 – Script-based job; IAM errors 55:06 – Troubleshooting; notebooks 58:29 – Architecture: S3 → Glue → Athena → QuickSight 1:04:31 – Athena CTAS (SQL-only transform) 1:06:46 – QuickSight 1:07:36 – Fix IAM PassRole 1:11:20 – Final PySpark; permissions 1:15:43 – Next steps (dbt, Redshift, Databricks) 1:17:28 – Wrap-up 📌 Resources mentioned • AWS Glue workshop (Getting Started): https://catalog.us-east-1.prod.workshops.aws/workshops/276faf92-bffc-4843-8a8e-8078add48194/en-US/10-getting-started • AWS Serverless Data Lake workshop (CloudFormation) • New York yellow taxi trip data (Kaggle) • Glue, Athena, QuickSight pricing and docs • Surfalytics: next lesson = infrastructure as code (Terraform/CloudFormation) ✅ Key takeaways • Serverless data lake = S3 + Glue catalog + Athena (+ optional QuickSight). • Use Glue scripts (PySpark) over visual ETL for production; develop in notebooks. • Athena CTAS can replace a Glue job when SQL is enough. • Glue needs IAM PassRole and S3 permissions; fix in IAM or IaC. • Same pattern with your data and account; then automate with Terraform/CloudFormation. --- Surfalytics – FREE course for beginners. Module 5: Cloud Computing. Lesson 8 Hands-On. #Surfalytics #AWS #DataLake #Serverless #Glue #Athena #S3 #QuickSight #DataEngineering #FreeCourse #Beginners