How to go from your agent's traces to a fine-tuned model in one workflow

Introducing LangSmith Fine-Tuning and smithtune, a CLI that helps teams turn LangSmith trajectories into custom fine-tuned models for their agents. It handles the entire fine-tuning process from one CLI: dataset creation and preparation from LangSmith trajectories, training with Fireworks or Baseten, and evaluation with LangSmith. You can run smithtune directly or work with your coding agent to run commands and inspect the results. In this walkthrough, Jake Broekhuizen fine-tunes a Qwen model using Open SWE reviewer traces and compares it against the base model in a live LangSmith experiment. Chapters: 0:00 One workflow: traces to a fine-tuned model 0:18 What supervised fine-tuning actually teaches an agent 0:56 Why turning traces into training data is normally painful 1:23 Why trajectory format is the foundation for training data 2:06 Starting from your LangSmith tracing project 2:48 Diving into the SmithTune CLI 3:17 Pulling trajectories with your coding agent 4:01 Defining a rubric for what a good trajectory looks like 4:15 Judging trajectories with a model council 4:41 Pushing the golden dataset back to LangSmith 5:04 Preparing your dataset for training 6:30 Reviewing the plan before a paid training job 7:03 Training and comparing base vs fine-tuned Qwen 8:25 Deploy step, or iterate and try again Resources: → SmithTune on GitHub: github.com/langchain-ai/smithtune → LangSmith: https://www.langchain.com/langsmith → Open SWE: https://swe.langchain.com