Observability in AI apps. Eval Engineering for AI Developers, lesson 2 - add observability to AI
Learn Eval Engineering in this free, 5-part, hands-on course presented by @jimbobbennett 90% of AI agents don't make it successfully to production. The biggest reason is the AI engineers building these apps don't have a clear way of evaluating that these agents are doing what they should do, and using the results of this evaluation to fix them. In this course, you will learn all about evals for AI applications. You'll start with some out-of-the-box metrics and learn about evals, then move onto understanding observability for AI apps, analyzing failure states, defining custom metrics, then finally using these across your whole SDLC. This will be hands on, so be prepared to write some code, create some metrics, and do some homework! In this second lesson, you will - Use observability to visualize the components of a typical multi-agent AI application - Learn about the different components that make up these applications - Apply some out-of-the-box metrics to start to get an understanding of how your application is working Prerequisites: - A basic knowledge of Python - Access to an OpenAI API key - A free Galileo account (we will be using Galileo as the evals platform). Sign up at https://galileo.ai/sign-up. - Course materials from https://github.com/rungalileo/eval-engineering Catch the rest of the lessons here: https://youtube.com/playlist?list=PLS7keRo8770OODJn9HAEN8JKbIzTHME60 0:00:10 - Introduction & Welcome 0:05:19 - Course Schedule & Overview 0:07:22 - Prerequisites & Setup 0:10:39 - Homework Review: Context Adherence 0:13:54 - Introduction to Observability 0:16:35 - Demo: Runzi Multi-Agent App 0:25:46 - What is Observability? 0:26:46 - Components of Observability: Spans, Traces, Sessions & Metrics 0:35:05 - Demo: Finding Failures with Observability 0:47:09 - Evaluations & Observability 0:54:33 - Adding Metrics: Instruction Adherence & Tone 1:19:36 - When to Add Metrics 1:23:08 - Homework: Breaking Runzi & Custom Metrics 1:26:41 - Q&A Session