Evaluating Agents in Production: Traces, LLM-as-Judge, and Prompt Management at Wonder

Kartik Arora, AI Engineer at Wonder, explains how his team builds a generative AI application that takes every aspect of your life into account and plans all 21 of your meals for the week, then gets them delivered. He walks through what changed when they moved off manual log debugging to LangSmith traces, evaluators, and LLM-as-judge scoring of how their agent responds to different users, and how prompt management took a feedback loop that used to run minutes to hours and made it automatic. With four engineers and two PMs, the team now hands a bug ticket to a coding agent that reads LangSmith, locates the issue in their code, and fixes it end to end. Chapters: 0:00 What Wonder is building, all 21 meals planned and delivered 0:21 Before LangSmith, debugging an agent through logs 0:59 Evaluating agent responses with LLM-as-judge 1:09 Prompt management and a feedback loop that went from hours to automated 1:33 Four engineers, two PMs, and fixing bugs through the MCP server 2:13 Letting non-engineers own the prompts 2:35 Watching the AI infrastructure ecosystem get built Resources: → LangSmith: https://www.langchain.com/langsmith → LLM-as-judge evaluators: https://docs.langchain.com/langsmith/llm-as-judge → Prompt engineering in LangSmith: https://docs.langchain.com/langsmith/prompt-engineering-quickstart → LangSmith MCP Server: https://docs.langchain.com/langsmith/langsmith-mcp-server