How Capital One Delivers Multi-Agent Systems [Rashmi Shetty] - 765
In this episode, Rashmi Shetty, senior director of enterprise generative AI platform at Capital One, joins us to explore how the company is designing, deploying, and scaling multi-agent systems in a highly regulated environment. Rashmi walks us through Chat Concierge, a multi-agent chat experience for auto dealerships that handles intent disambiguation, tool invocation, and human handoffs to deliver safer, more personalized customer journeys. We discuss Capital One’s platform-centric approach to AI agents and how it separates design from runtime governance, embedding policies, guardrails, and cyber controls across agent threat boundaries. Rashmi shares how the team approaches the developer experience for agent builders, observability, and evals for stochastic, multi-agent workflows; and strategies for model specialization, including fine-tuning and distillation. We also cover standards and abstraction, closed-loop learning from production telemetry, and key lessons for enterprises building agentic systems. 🗒️ For the full list of resources for this episode, visit the show notes page: https://twimlai.com/go/765. 🔔 Subscribe to our channel for more great content just like this: https://youtube.com/twimlai?sub_confirmation=1 🗣️ CONNECT WITH US! =============================== Subscribe to the TWIML AI Podcast: https://twimlai.com/podcast/twimlai/ Follow us on Twitter: https://twitter.com/twimlai Follow us on LinkedIn: https://www.linkedin.com/company/twimlai/ Join our Slack Community: https://twimlai.com/community/ Subscribe to our newsletter: https://twimlai.com/newsletter/ Want to get in touch? Send us a message: https://twimlai.com/contact/ 📖 CHAPTERS =============================== 00:00 - Introduction 04:05 - Motivation for multi-agentic systems 05:33 - Chat Concierge 09:26 - Regulations 11:19 - Governance: agent building vs. agent runtime execution 12:26 - Purpose of the enterprise platform 13:50 - Policy-bound agentic operations 15:30 - Developer requirements for agentic systems vs classic ML/AI architecture 19:40 - Observability for agentic systems 22:36 - Observability challenges in multi-agentic workflows 25:33 - Agentic evaluation frameworks vs. classic ML evaluation frameworks 27:50 - Reasoning and specialized models 29:40 - Key architectural decisions 32:23 - Agentic standards 33:33 - Human-in-the-loop integration 38:23 - Managing complexity for teams 43:36 - Model risk office 45:02 - Lessons learned 48:13 - Next steps 🔗 LINKS & RESOURCES =============================== AI at Capital One - https://www.capitalone.com/tech/ai/ Capital One AI Research - https://www.capitalone.com/tech/ai-research/ Publications - https://www.capitalone.com/tech/publications/ Evolving MLOps Platforms for Generative AI and Agents with Abhijit Bose - #714 - https://twimlai.com/podcast/twimlai/evolving-mlops-platforms-for-generative-ai-and-agents 📸 Camera: https://amzn.to/3TQ3zsg 🎙️Microphone: https://amzn.to/3t5zXeV 🚦Lights: https://amzn.to/3TQlX49 🎛️ Audio Interface: https://amzn.to/3TVFAIq 🎚️ Stream Deck: https://amzn.to/3zzm7F5