EP12 : Forge & Perform AI Model Customization at Scale | AWS for AI Podcast | Amazon Web Services

Join host Hamza Mimi for a deep conversation on model customization with two guests tackling it from opposite vantage points. Oliver Graeser, Head of Data & Insights at Perform AI (formerly Parcel Perform) — the Singapore-based delivery experience platform processing billions of shipments across 160+ countries — shares how his team fine-tuned Amazon Nova and achieved something counter-intuitive: a smaller model that outperformed larger ones on accuracy, cost, and speed. Joining him is Karan Bhandarkar, Principal Product Manager for Forge and model customization at AWS, who reveals the patterns, anti-patterns, and tools he sees across customers worldwide.Together they unpack why customization is an economics question before it's a technology question, and give a practical roadmap for teams moving from prompting to production customization. This episode covers critical topics including: • Why and when to customize vs. prompt engineer, RAG, or build an agentic harness • Building a robust evaluation framework and defining what "good" looks like • The ROI conversation: fine-tuning cost vs. inference cost (and talking to your CFO) • The hallucination problem and how fine-tuning tackled it • Pre-training, mid-training, fine-tuning, and reinforcement learning explained • Data quality over quantity — and how little data you may actually need • Catastrophic forgetting and the role of general-purpose data mixing • LoRA / parameter-efficient fine-tuning and the pay-as-you-go economics of serving • Customizing with SageMaker AI, Amazon Bedrock, and Nova Forge Whether you're a startup, enterprise, or public sector organization running high-volume AI workloads, this episode offers invaluable guidance on building scalable, cost-efficient, production-ready AI solutions in today's rapidly evolving landscape. OUTLINE: 00:00:00 : Introduction 00:00:43 : Meet Oliver — Perform AI 00:03:34 : Meet Karan — AWS Forge 00:07:04 : Why bother customizing when frontier models keep winning? 00:11:51 : The only 3 reasons to customize: quality, speed, cost 00:13:58 : Does the math actually work? The ROI reality 00:19:47 : When you should NOT customize 00:22:11 : How a small model beat the giants in production 00:25:36 : Making the model stop hallucinating 00:28:04 : Pre-training to RL: the 4 stages demystified 00:31:39 : You need way less data than you think 00:39:29 : Catastrophic forgetting (and how to avoid it) 00:44:08 : Why evals matter more than your model 00:48:48 : Prompt vs. fine-tune vs. agent — it's not either/or 00:50:34 : How to actually do it on AWS 00:56:00 : From notebook to production 00:56:15 : Quickfire round 01:01:16 : A CTO's playbook for customization 01:04:53 : Closing remarks To learn more about Perform AI : https://bit.ly/4ywZ6OM Parcel Perform case study blog : https://aws.amazon.com/blogs/machine-learning/fine-tune-amazon-nova-models-for-accurat Subscribe to AWS: https://go.aws/subscribe Create a free AWS account: https://go.aws/signup Try AWS for free: https://go.aws/free Connect with an expert: https://go.aws/contact Explore more: https://go.aws/more Next steps: Explore on AWS in Analyst Research: https://go.aws/reports Discover, deploy, and manage software that runs on AWS: https://go.aws/marketplace Join the AWS Partner Network: https://go.aws/partners Learn more on how Amazon builds and operates software: https://go.aws/library Do you have technical AWS questions? Ask the community of experts on AWS re:Post: https://go.aws/3lPaoPb Why AWS? Amazon Web Services is the world’s most comprehensive and broadly adopted cloud, enabling customers to build anything they can imagine. We offer the greatest choice of innovative cloud capabilities and expertise, on the most extensive global infrastructure with industry-leading security, reliability, and performance. #AWS #AmazonWebServices #CloudComputing #AWS #AmazonWebServices #CloudComputing #AWSForAI #artificial-intelligence #ModelCustomization #FineTuning #SmallLanguageModels #AmazonNova #SageMaker #AmazonBedrock #LoRA #MachineLearning #TechTalks