Data Ingestions Patterns | Part 3 | Projects
In this FREE Surfalytics workshop (Part 3 of the AWS data ingestion series), we continue from Lambda, Serverless Framework, and AWS Batch—and go deeper into how you actually wire services together: job submission and dependencies, why Batch often reaches Lambda through a container, Step Functions for direct Lambda orchestration, EventBridge for schedules, and when to call ECS tasks from Step Functions instead of going only through Batch. If you are a beginner with little cloud experience, this session still connects every choice back to tradeoffs: transparency, timeouts, cost, and what to learn first when you join a team that already runs Airflow, Batch, or Step Functions. 🎯 What you'll learn RECAP & CORE PATTERNS • Quick recap: Python → Boto3 → S3, CloudFormation-by-service, IAM profiles, ECR, Batch, ECS/Fargate • Lambda vs containers for ingestion: debuggability, limits, and typical production paths (including dbt core hosting) ORCHESTRATION & AWS SERVICES • AWS Batch job queue, job definition revisions, passing arguments (e.g. dbt model selection), job dependencies • Why Batch is container-first—and what it means when you need Lambda invoked from Batch (CLI/pre-built images vs your own image) • Step Functions: visual state machines, retries, parallelism, invoking AWS services without extra glue • EventBridge as the scheduler/cron layer for Step Functions (and how that compares to Batch-only schedules) • ECS task definitions inside ECS vs job definitions in Batch; running the same Python from Lambda, Batch, and ECS • VPC, subnets, security groups: why they show up in almost every AWS deployment PRACTICAL ENGINEERING • Public ECR gallery vs your private registry; tagging `latest` vs versioned images for prod vs testing • CloudWatch logs across Batch/ECS; where observability differs from Airflow-managed runs • AI-assisted infra: CLAUDE.md / agent rules, plans vs blind deploys, parallel work (e.g. git worktrees) ⏱️ TIMESTAMPS 0:00 - Workshop recap: from Python scripts and Docker to AWS Lambda 3:17 - Triggering Lambda from CLI and why monitoring execution status matters 6:15 - Lambda orchestration options: Step Functions and Batch wrappers 9:03 - What Lambda is in a nutshell for data engineering pipelines 11:10 - Event-driven ingestion: S3-triggered Lambda for near real-time processing 13:17 - When not to use Lambda: timeout, heavy transforms, and larger workloads 15:04 - Building and deploying Lambda with Serverless Framework (Node + serverless.yml) 17:24 - Running Docker-based ingestion with AWS Batch and ECR 34:35 - Reviewing AWS Batch resources: compute environment, queues, and job definitions 47:05 - Pushing Docker images to ECR and production deployment patterns 1:33:20 - Submitting and running Batch jobs from job definitions 1:36:01 - End-to-end flow: ECR → AWS Batch → ECS/Fargate → CloudWatch logs 📌 Resources mentioned • AWS Lambda • Serverless Framework • AWS Batch • Amazon ECR (private + public gallery) • Amazon ECS / AWS Fargate • AWS Step Functions • Amazon EventBridge • Amazon CloudWatch • AWS CloudFormation ✅ Key takeaways • Batch orchestrates scheduled container work; Step Functions shines when you need multi-step workflows with branching, retries, and clear execution history. • EventBridge is often the right “cron” for Step Functions; know how that differs from Batch-only scheduling. • The same ingestion code can run as Lambda, Batch job, or ECS task—pick the path that matches latency, volume, and operability. • VPC networking fundamentals follow you across Lambda, Batch, ECS, and almost every data stack on AWS. • Strong repo context (profiles, stack layout, agent rules) speeds up safe IaC changes as much as good code does. This is part of the FREE Data Engineering content from Surfalytics. Like, subscribe, and share if you want more practical workshops on ingestion, orchestration, dbt, and cloud data platforms.