Building Engine v2: How LangChain approaches agent improvement

Ben Tannyhill, Product Manager at LangChain, explains how his team builds and evaluates LangSmith Engine, the agent in LangSmith that finds issues in production traces, proposes fixes, and creates evals to catch regressions. He covers the per-task benchmarks behind Engine's 2x better error identification on IssueBench, how the team runs Engine on LangChain's own go-to-market and coding agents and on Engine itself, and why a smaller model is sometimes the better choice for a simple task. He then walks through what Engine v2 adds, from detecting cross-trace metric trends and inefficient agent paths to validated fixes and red teaming (both in private beta for LangSmith Deployment customers), and previews the team's plans for Engine v3. This session was captured at Interrupt New York City, the agent conference by LangChain. LangChain products in this talk: LangSmith Engine, LangSmith, LangSmith Deployment Speakers: Ben Tannyhill, Product Manager, LangChain Chapters: 0:00 What does an agent for agent engineering do? 2:11 Why an agent with many tasks needs a benchmark for each 4:40 Keeping benchmarks honest by dogfooding internal agents 5:57 Running Engine on its own traces with a meta Engine 6:45 When is a smaller model better than a frontier model? 8:06 How hill climbing doubled error identification on IssueBench 9:09 What Engine v1 could never catch in the agent problem space 10:59 Why can a model swap look fine trace by trace? 12:25 Catching agents that go on inefficient side quests 13:02 How validated fixes reproduce the bug and test the fix 16:08 Red teaming your agent before it ever touches a user 17:29 What comes after Engine v2? 18:41 Could Engine propose a cheaper model backed by your evals? 19:38 Bring your own key support coming to self-hosted Engine Resources: → LangSmith Engine: https://www.langchain.com/langsmith/engine → LangSmith Engine docs: https://docs.langchain.com/langsmith/engine-overview → LangSmith Deployment: https://www.langchain.com/langsmith/deployment → LangSmith: https://www.langchain.com/langsmith → Interrupt, the agent conference by LangChain: https://interrupt.langchain.com