Three Questions Every Agent Engineer Should Be Able To Answer

Jake Broekhuizen of LangChain Labs argues that owning your own intelligence comes down to three questions every agent engineer should be able to answer: where is your agent failing, how can you reliably make it fail again, and what needs to happen so it stops failing. He covers using a fine-tuned judge to mine production traces when frontier LLM-as-judge evaluators get expensive at scale, building world specs and tasks that mimic production with LangChain's internal GTM agent as the example, and making fixes at both the harness and the model layer. He also explains why application teams, who see how real people use their agents, can build tasks and environments that a frontier lab never could. Chapters: 0:00 What it means to own your own intelligence 0:38 The three questions every agent engineer should answer 1:30 Question 1: mining production traces to find failures 2:27 Fine-tuned judges for monitoring agents at scale 4:10 Question 2: reproducing failures beyond input and output evals 5:25 World specs and tasks that mimic production 6:20 Example: the world and tasks for LangChain's GTM agent 7:45 Keeping tasks hard and hard to reward hack 8:41 Turning production data into tasks and worlds 9:15 What app teams see that frontier labs can't 11:08 Question 3: fixing failures at the harness and model layers 12:34 Staying ahead as models and users change Resources: → How we built LangChain's GTM Agent: https://www.langchain.com/blog/how-we-built-langchains-gtm-agent → LangSmith Observability: https://www.langchain.com/langsmith/observability Recorded at LangChain Presents: The Learning Loop, an evening on continual learning at LangChain's San Francisco office.