Data Ingestions Patterns | Part 1 | Projects

Surfalytics community workshop Data Ingestion – Part 1 (Module 4): batch ingestion patterns from a local Python script through Docker toward AWS Lambda, with GitHub API → CSV → S3, IAM least privilege, CloudFormation, Secrets Manager, and honest AI-assisted development (Cursor agent CLI, planning vs execution). 📚 What you'll learn Context and • Why every ingestion needs a business question (not only a data source) • Roadmap: prototype Python → Docker → Lambda → scheduling → Polars/DuckDB → Batch/Step Functions, Glue/Athena, Fivetran, Airflow (stretched across sessions) • Prerequisites: AWS, VS Code/Cursor, GitHub, target Snowflake, Docker Hands-on themes • Snowflake trial setup; UV for Python environments • IAM users, AWS CLI profiles, Boto3, least-privilege lab user • CloudFormation YAML: S3 bucket, IAM policies, outputs; `aws cloudformation deploy`; delete stack to avoid surprise cost • GitHub CLI (`gh`) to create and sync a repo; multi-agent workflows in Cursor • GitHub REST API with Python `requests`: pagination, rate limits, JSON → pandas → CSV; testing with curl • AWS Secrets Manager for tokens; why planning stacks upfront beats endless patches • Dockerfile (e.g. Python slim), `requirements.txt`, env files and gitignore • Lambda basics: timeouts, memory, roles, layers, deployment options (AWS CLI vs Serverless Framework), CI/CD mention for scheduled jobs • Operational tip: maintain `AGENTS.md` / `CLAUDE.md` so agents know your AWS profile and lab assumptions ⏱️ Timestamps 0:22 – Welcome & overview 2:24 – Prerequisites: AWS, VS Code, GitHub, Snowflake, Docker 3:25 – Python prototype: S3 → Snowflake 3:46 – Same script in Docker 3:56 – Lambda basics 5:15 – Scheduling (Batch, Glue, Airflow) 7:50 – Teaser: Bulk Snowflake load 9:19 – Cursor agent CLI, multi-tab workflow 12:52 – Snowflake trial signup 16:23 – Python setup: UV, venv 18:12 – AWS: IAM users, CLI, Boto3 20:18 – IAM users: admin vs lab roles 26:36 – IaC: CloudFormation setup (S3, policies) 27:39 – GitHub CLI: create repo 46:05 – Python: venv, install deps 47:31 – S3 bucket listing test 50:15 – What is an API? GitHub vs DB 51:21 – Handling API rate limits 53:04 – Python requests library 56:25 – From JSON, to pandas, to CSV 59:31 – Test endpoints with curl first 1:06:04 – Step recap: API → CSV → (S3) 1:10:21 – Secrets Manager for tokens 1:39:15 – Docker build & run to S3 1:43:49 – Dockerfile basics 2:03:14 – Container run uploads to S3 2:03:34 – Package for AWS Lambda 2:06:22 – Lambda: settings & limitations 2:09:45 – Lambda deploy; intro CI/CD 2:17:26 – Lambda layers (pandas/numpy) 2:21:00 – Next time: Serverless framework, Polars/DuckDB, more Snowflake 2:24:49 – Lesson: document AWS profile/context 2:27:26 – Closing & next session 📌 Resources mentioned • AWS: IAM, S3, CloudFormation, Secrets Manager, Lambda, CloudWatch (monitoring/alerting mentioned) • Cursor / VS Code agent CLI; GitHub CLI • Python: UV, venv, requests, pandas (Polars/DuckDB teased for later) • Snowflake trial; Surfalytics curriculum (Module 4/5) 📌 Repo: https://github.com/surfalytics/data-ingestion-github-to-snowflake ✅ Key takeaways • Start from why the data is needed, then pick endpoints and tables, not “all columns from the API.” • Infrastructure as code (here CloudFormation) makes labs repeatable and deletable—important when your card is on the account. • API ingestion is a repeatable skill: auth, rate limits, pagination, JSON parse, land files in object storage, then load to the warehouse. • Docker packages what works locally for repeatable runs; Lambda adds operations constraints (time, memory, packaging)—know the tradeoffs for interviews. • Document repo context for AI (profiles, intended deploy path) to reduce wasted agent loops. --- Surfalytics – Community workshop. Data Ingestion Patterns – Part 1. #Surfalytics #DataEngineering #DataIngestion #ETL #AWS #Python #Docker #Lambda #S3 #CloudFormation #GitHubAPI #AnalyticsEngineering #Snowflake #CursorAI #Workshop