Custom metrics. Eval Engineering for AI Developers, lesson 4 - learn how to write custom AI metrics
Learn Eval Engineering in this free, 5-part, hands-on course presented by @jimbobbennett 90% of AI agents don't make it successfully to production. The biggest reason is the AI engineers building these apps don't have a clear way of evaluating that these agents are doing what they should do, and using the results of this evaluation to fix them. In this course, you will learn all about evals for AI applications. You'll start with some out-of-the-box metrics and learn about evals, then move onto understanding observability for AI apps, analyzing failure states, defining custom metrics, then finally using these across your whole SDLC. This will be hands on, so be prepared to write some code, create some metrics, and do some homework! In this fourth lesson, you will - Build datasets of known inputs and outputs for cases that pass and fail - Learn how to build custom metrics for your failure cases - Determine the success of your metrics by measuring true and false positives and negatives Prerequisites: - A basic knowledge of Python - Access to an OpenAI API key - A free Galileo account (we will be using Galileo as the evals platform). Sign up at https://galileo.ai/sign-up. - Course materials from https://github.com/rungalileo/eval-engineering Catch the rest of the lessons here: https://youtube.com/playlist?list=PLS7keRo8770OODJn9HAEN8JKbIzTHME60 0:00:31 - Introduction & Overview 0:11:55 - Evaluation Driven Development: Moving Beyond Bug Fixes 0:16:45 - Data Strategy: Training, Dev, and Test Sets 0:27:07 - Creating Datasets from Trace Logs 0:32:12 - Anatomy of a Metric Prompt: Role, Criteria, Rubric & Examples 0:44:19 - Demo: Building a Custom Metric in Galileo 0:51:28 - Testing the Metric with Dev Data (Experiments) 1:01:42 - Iterating and Refining the Metric Prompt 1:08:24 - Scoring Your Metric: True Positive (TPR) & True Negative Rates (TNR) 1:23:44 - Homework: Using Metrics to Fix the App