Evaluating and testing AI safely
How do you know whether a model swap or other change is truly an improvement before you go live? A new model comes out, cheaper and faster, and the vendor swears it performs just as well. A new tool gets connected. A prompt gets tuned. Any one of these changes can silently break a production workflow your customers depend on — and the first sign of trouble is often an angry customer, not a dashboard. We'll walk through what a gold benchmark query set actually is — a small, zero-tolerance set of monetization-critical questions, layered with real queries pulled from user history (especially the downvoted ones) and hypothetical edge cases a subject matter expert wants tested before a customer ever asks them. From there, we cover some of the key metrics worth focusing on — semantic similarity, faithfulness, and groundedness — and why running these at scale, with an LLM judging another LLM's output, is a real cost tradeoff worth being deliberate about. We'll also get into what changes when evaluation is automated and hooked directly into CI/CD, so every proposed change is tested against your own custom benchmarks before it's even eligible for deployment review — no one has to remember to ask. We will share our experience and take questions from the audience. **What you'll learn** - Why staging tells you a change is safe to make, and evaluation is the separate question of whether it's actually good - How to build a gold benchmark: monetization-critical queries, real user history, and subject-matter-expert edge cases - Three key metrics to focus on — semantic similarity, faithfulness, and groundedness — and what each one is actually catching - Why LLM-judging-LLM evaluation carries a real cost tradeoff, and how to run it efficiently - How hooking evaluation into CI/CD turns "we should probably test this" into an automatic, repeatable gate before every deployment - A live look at running an evaluation against staging vs. production, with a pass-rate breakdown by query category, inside the Ejento platform