Why data science matters more than ever for AI evaluation

Speed is no longer the hard part of building with AI. Confidence is. Generative assistants, agents, and copilots let teams go from idea to working prototype in an afternoon, but a working prototype isn't proof that a system behaves the way you expect. In this talk, James Blair, senior product manager at Posit, makes the case for data scientists as the right people to lead AI evaluation. Unlike traditional software, AI systems can take different paths to different outputs from the same input, so testing for one expected result doesn't work. Data scientists already have the tools for this problem: experimental design, statistical rigor, and the discipline to separate real improvement from random noise. James walks through what that looks like in practice: evaluating not just a model's final output but the tool calls, sources, and intermediate steps that produced it, building evaluation criteria specific to each system's domain, and treating evaluation as a continuous practice that runs through production, not just development. He also covers how Posit's open-source tooling for R and Python, paired with AWS infrastructure and model access through Bedrock, gives teams a way to build evaluations that scale and hold up over time. Watch to learn why rigorous evaluation, not raw model capability, decides whether an AI system is ready for production. See how Posit and AWS work together: https://posit.co/solutions/aws