Data Ingestions Patterns | Part 2 | Projects
In this FREE Surfalytics workshop, we continue our beginner-friendly Data Engineering series and build the same ingestion pipeline using multiple AWS patterns: Lambda, Serverless Framework, and AWS Batch with Docker. If you are a beginner with little cloud experience, this session helps you understand not just how to run the code, but how to choose the right AWS service for production. 🎯 What you'll learn THEORY • Why one ingestion task can be solved with multiple AWS tools • Lambda strengths and limitations for data ingestion • Event-driven ingestion with S3 triggers • When to use Step Functions, Batch, Glue, or EMR instead • Infrastructure and IAM design lessons from real deployment issues HANDS-ON • Deploying Lambda with Serverless Framework (serverless.yml) • Building and pushing Docker images to ECR • Running ingestion jobs with AWS Batch + ECS/Fargate • Understanding job definitions, queues, and compute environments • Reviewing logs and observability in CloudWatch Repo: https://github.com/surfalytics/data-ingestion-github-to-snowflake ⏱️ TIMESTAMPS 0:00 - Workshop recap: from Python scripts and Docker to AWS Lambda 3:13 - Triggering Lambda from CLI and why monitoring execution status matters 6:09 - Lambda orchestration options: Step Functions and Batch wrappers 9:00 - What Lambda is in a nutshell for data engineering pipelines 11:04 - Event-driven ingestion: S3-triggered Lambda for near real-time processing 13:12 - When not to use Lambda: timeout, heavy transforms, and larger workloads 15:03 - Building and deploying Lambda with Serverless Framework (Node + serverless.yml) 17:20 - Running Docker-based ingestion with AWS Batch and ECR 34:33 - Reviewing AWS Batch resources: compute environment, queues, and job definitions 47:03 - Pushing Docker images to ECR and production deployment patterns 1:33:11 - Submitting and running Batch jobs from job definitions 1:36:00 - End-to-end flow: ECR - AWS Batch - ECS/Fargate - CloudWatch logs 📌 Resources mentioned • AWS Lambda • Serverless Framework • AWS Batch • Amazon ECR • Amazon ECS / AWS Fargate • Amazon CloudWatch ✅ Key takeaways • Tooling choice depends on latency, data volume, and operational constraints. • Lambda is great for lightweight jobs, but not for every ingestion workload. • Batch + Docker + ECR is a production-ready pattern for many data engineering pipelines. • CI/CD and IAM structure matter as much as code quality. • Observability and alerting should be implemented early. This is part of the FREE Data Engineering content from Surfalytics. Like, subscribe, and share if you want more practical workshops on ingestion, orchestration, dbt, and cloud data platforms.