Vanta and LangSmith: Assessing Compliance Evidence Like an Auditor

"I can finally find the data that I want without having to fight my tooling." Zheng Tao, Senior AI Engineer at Vanta, is building a capability that helps assess compliance evidence the way an auditor would. With other vendors and Vanta's own in-house tooling, she would spend 10 minutes trying to find a trace and give up. She explains how LangSmith observability lets her look through traces with full text search and metadata fields, and why she's a big fan of LangSmith evaluations, especially the Align Evaluators feature, which forces you to have an opinion and label your ground truth as good or bad. Chapters: 0:00 Assessing compliance evidence and trying to find traces 0:23 LangSmith observability with full text search and metadata fields 0:41 LangSmith evaluations and the Align Evaluators feature 1:04 Why LangSmith gets it Resources: → LangSmith: https://www.langchain.com/langsmith → Align your LLM-as-judge evaluators: https://docs.langchain.com/langsmith/improve-judge-evaluator-feedback → Filter and search traces: https://docs.langchain.com/langsmith/filter-traces → Vanta: https://www.vanta.com