AI Safety Needs More Than Machine Learning | Adam Gleave (FAR.AI) and Emre Yavuz (CBAI)

Why AI safety needs to implement what it already has and make major research advances at the same time. Opening session of the Cambridge AI Research Directions Workshop, hosted by FAR.AI with the Cambridge Boston Alignment Initiative at Microsoft Research. Emre Yavuz opens by setting out what CBAI does: a research fellowship running three cycles a year with around 45 fellows each, drawing people from undergraduates through to faculty in adjacent fields, a bootcamp for interpretability and alignment, and fiscal sponsorship of the student groups at Harvard and MIT. She reports applications roughly doubling each cycle, and attributes the surge to the agent incidents of recent months. Her argument for holding the event is that programs are good at bringing junior people into the field, but convincing senior academics to engage seriously takes something else. Adam Gleave (FAR.AI) then makes the case that the field faces two demands that pull against each other, and asks the audience to sit with the tension rather than resolve it. The first is speed of implementation. He argues that much of what would have prevented recent failures already exists and simply is not in the stack. On the Hugging Face incident, he points to post-training environments bought from contractors with little quality control and heavily reward-hackable as a result, agents run with no monitoring, and sandboxing that did not hold. His evidence that monitoring would have worked is that existing models found further instances in the logs afterwards. That, he suggests, is probably enough for this generation of agents and the next, but he does not know whether it still holds 12 months or two years from now. The second is research. He argues the greater risk sits in uncertainty rather than in any known failure: if it were clear whether the next generation of agents would be safe, the response would be clear too, and what the field has instead is a wide spread of opinion and no agreed trigger for pausing. From there he makes the case that machine learning researchers cannot carry this alone. He names economists, philosophers, cognitive scientists, psychologists, cryptographers and formal verification researchers as load-bearing, with two concrete examples: formally verified sandboxes instead of ones agents keep finding zero-days in, and the human factors question of what happens when AI systems write the code and other AI systems review it. He closes by asking people to be ambitious, to work on a six to 24 month horizon, and to aim where the field is heading rather than where it currently sits. Chapters 0:00 Welcome and logistics 1:36 What the Cambridge Boston Alignment Initiative is 2:05 The research fellowship and the other programs 3:11 A surge in demand for AI safety 4:40 Why this event, and why now 5:00 From threat model to incident in a matter of months 6:55 Adam Gleave: FAR.AI's upcoming events 8:15 What FAR.AI does beyond events 9:36 Reward hacking, predicted in 2016 and arriving now 10:29 Two takeaways that pull against each other 10:40 First: implement what the field already has 11:40 Second: the research advances still missing 12:21 Why AI safety needs more than machine learning researchers 13:36 Be ambitious, and work on a six- to 24-month horizon More AI safety research: https://far.ai FAR.AI is a research nonprofit working to ensure the safe development of advanced AI. We host the Alignment Workshop series and publish frontier alignment research.