Data Ingestions Patterns | Part 4 | Projects | Glue, Athena, Airflow (MWAA), Iceberg lakehouse

In this FREE Surfalytics workshop (Part 4 of the AWS data ingestion series), we take the same GitHub → S3 ingestion code from earlier parts and run it through the data-processing and orchestration layer: AWS Glue (Python Shell and PySpark), Amazon Athena for serverless SQL, a Managed Airflow (MWAA) environment, and finally a dbt + Apache Iceberg lakehouse on Athena. The whole session is built with an AI coding agent (Claude in Cursor) driving CloudFormation in plan mode—so it's also a practical look at how AI-assisted infrastructure work actually goes, including a Managed Airflow MCP server. If you are a beginner with little cloud experience, this session still connects every choice back to tradeoffs: serverless vs your own clusters, cost, file sizing in Spark, and what to learn first when you join a team that already runs Glue, Athena, Airflow, and dbt. 🎯 What you'll learn GLUE & SPARK • Glue Data Catalog as a managed Hive-style metastore; crawlers that auto-discover schemas in S3 • Glue job types: visual editor, notebooks, Python Shell, and PySpark scripts • GlueContext vs SparkContext and DynamicFrame vs DataFrame—and why many teams stay on plain Spark • Spark cluster sizing, query plans, and the Spark UI; the same logic applied to Snowflake warehouses • coalesce vs repartition to control output file count and size in distributed compute • Glue stores job code as files in S3—enabling file-level versioning and release strategies • Glue Docker image for running and testing Spark jobs locally and in CI/CD ATHENA & LAKEHOUSE • Amazon Athena as a serverless Presto/Trino SQL engine; pricing per TB scanned plus S3 • Partition projections vs Hive-style partition metadata; JSON SerDe handling • dbt + Apache Iceberg lakehouse with dbt-athena-community (Docker image on ECR) • Iceberg internals: data vs metadata folders, manifest files, manifest lists, and snapshots • One dbt project targeting Athena, Redshift, and Snowflake ORCHESTRATION & MWAA • Managed Airflow (MWAA): S3-based DAG syncing, Secrets Manager and CloudWatch integration • Why MWAA is NOT serverless—VPC, 24/7 billing, and when local Airflow is cheaper • Hosting dbt with Airflow: DAGs in S3 vs running dbt in a container on ECS/Batch • The ECS EcsRunTaskOperator as the common production pattern for dbt + Airflow • Adding an Airflow MCP server so an AI agent can inspect and trigger DAGs ⏱️ TIMESTAMPS 0:00 - Recap of Parts 1-3 and the full ingestion roadmap (Glue, Athena, Redshift, streaming) 6:04 - AWS Glue overview: ETL jobs, Data Catalog, crawlers, and connections 9:19 - Glue visual editor, GlueContext vs SparkContext, DynamicFrame vs DataFrame 13:37 - Worker types and Spark cluster sizing: serverless vs your own clusters 18:04 - Glue notebooks and Python Shell: running Athena SQL on a schedule 19:46 - Building Glue Shell and Spark jobs with an AI agent and CloudFormation 24:00 - Standing up the smallest Managed Airflow (MWAA) environment 28:48 - VPC, public vs private subnets, NAT, and lab-account cost awareness 31:17 - Adding Athena to query S3 data and partition projections 1:14:12 - What is Athena? Presto, Trino, and the serverless SQL engine 1:30:40 - MWAA up: S3 DAG syncing, end-to-end DAGs, and an Airflow MCP server 1:45:59 - Building a dbt + Apache Iceberg lakehouse on Athena (Docker + ECR) 1:57:24 - Iceberg internals: data files, metadata, manifests, and snapshots 2:09:05 - Running dbt on ECS from Airflow, repo structure, and tearing down resources 📌 Resources mentioned • AWS Glue (Data Catalog, crawlers, Python Shell, PySpark) • Amazon Athena (Presto / Trino) • Apache Iceberg • Amazon Managed Workflows for Apache Airflow (MWAA) • dbt Core + dbt-athena-community • Amazon ECS / AWS Fargate / Amazon ECR • Amazon CloudWatch + AWS Secrets Manager • AWS CloudFormation • DuckLake (DuckDB + catalog) ✅ Key takeaways • Glue Python Shell is a lightweight scheduled Python runtime without Lambda's timeout limits—great for wrapping Athena SQL. • The same ingestion code runs across Glue Shell, Glue Spark, and Airflow; the difference is packaging, scaling, and observability. • Spark cost control—right-sizing clusters, reading query plans, coalesce vs repartition—is a real data-engineering skill and great interview material. • Athena is serverless Presto/Trino: pay per TB scanned plus S3, and partition projections remove most Hive partition-management pain. • MWAA is powerful but runs in a VPC and bills 24/7—tear it down on lab accounts; local Airflow is cheaper for personal projects. • dbt + Iceberg turns an Athena data lake into a lakehouse, and one dbt project can target Athena, Redshift, and Snowflake. • System design is the differentiator—AI builds infra fast, but knowing the services, tradeoffs, and costs is what ships projects. This is part of the FREE Data Engineering content from Surfalytics.