Collaborative Gym: Human-Agent Collaboration | Snorkel AI Reading Group

Yijia Shao of Stanford University dives into her paper "Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration" at the Snorkel AI Reading Group in San Francisco. Read the paper: https://arxiv.org/abs/2412.15701 Subscribe to be notified of future Reading Group and other learning events: https://luma.com/snorkel-ai Follow Yijia on: X: https://x.com/EchoShao8899 LinkedIn: https://www.linkedin.com/in/shaoyj/ == CHAPTERS == 0:00 Introduction 1:31 Yijia's intro & talk overview 2:55 Beyond full automation: the METR trend 3:56 Agents entering real-world work 4:36 AI slop and the Operator example 6:02 Human agency and the skeptics 8:22 Introducing Collaborative Gym (Co-Gym) 8:45 Recap: today's fully autonomous agents 9:38 Adding a human: dual control & two-way communication 11:21 Collaboration acts: communication & intelligent deferral 15:14 The notification system & avoiding livelock 18:38 Agent design: ReAct agent + planner variant 20:17 Experiments: three tasks & setup 21:53 Results: collaboration vs. autonomy, with real users 23:37 Where models fall short: communication & situational awareness 25:26 What's next: a new frontier, upskilling humans, the podcast 30:46 Q&A Yijia covers the motivation for human-agent collaboration, the Co-Gym framework and its non-turn-taking interaction paradigm, the evaluation suite for both collaboration outcomes and processes, the simulated and real-world conditions, and benchmark results showing collaborative agents outperform fully autonomous ones, alongside the communication and situational-awareness failures that remain.