Skill Issues: How We Evaluate Data Agents For Performance and Costs

Jacopo Tagliabue shared how we build evals for data engineering agents, and what those evals told us about which models to trust. The argument: your infrastructure and your model are mostly fixed, so the lever that is actually yours is what you write in CLAUDE.md and your skills. Which makes the question how you measure whether those instructions are working. Jacopo starts with what evals are, then goes through the recent Supabase and Snowflake benchmarks and where Bauplan's sits against them. Scoring is the hard part. LLM-as-a-judge fails in a familiar way: the model says it did the work, the judge marks it complete, and nothing actually ran. With tasks and verifiers in hand, GEPA runs over the skills in a loop until they stop improving. The results are interesting. Skills improve every model tested, smaller models benefit far more than large ones, and open source models can match Opus on our tasks for roughly a third of the token cost. Blog Post: https://bauplanlabs.com/post/skill-issues Research Paper: https://arxiv.org/abs/2606.01185 Github Repo: https://github.com/BauplanLabs/skill-issues Slide Deck: https://drive.google.com/file/d/1awko58kCjGicU7fRdSzqx-qPxzhyapdz/view Chapters: 00:00 - Intro and why agents in data engineering 06:55 - Token maxing, and what ROI actually means now 21:10 - What evals are, using homework as the model 31:00 - The LLM-as-a-judge problem 42:46 - Wrap and VLDB happy hour Try Bauplan: https://app.bauplanlabs.com/ Helpful Links: 📚 Official Documentation: https://docs.bauplanlabs.com 💻 GitHub Repo: https://github.com/BauplanLabs/ 🌐 Bauplan Homepage: https://bauplanlabs.com 📄 Technical Blog: https://www.bauplanlabs.com/blog 📣 Follow Us: Twitter: https://x.com/Bauplan_labs LinkedIn: https://www.linkedin.com/company/bauplanlabs/