Red-Teaming & Jailbreaking AI: Why Open-Weight Models Stay Vulnerable | Kellin Pelrine (FAR.AI)

Kellin Pelrine (FAR.AI) on the state of AI red-teaming: closed models are much harder to jailbreak, while open-weight models stay trivially easy. Pelrine gives a red-teaming status report. Over the past year, leading closed-weight models have become far more robust, moving the problem from open research questions toward engineering and adoption. Open-weight models are a different story: they stay easy to jailbreak with simple prompt-based attacks that need no special expertise or resources, and the "jailbreak tax," the capability loss from breaking a model, largely disappears for stronger models. He then makes the constructive case that robust open models are achievable, pointing to pretraining data filtering, gradient routing to isolate dangerous capabilities, and government work on risk modeling and evaluation. There is no single fix, but the risk can be reduced. Chapters 0:00 Red-teaming and jailbreaking: the state of play 0:06 Good news: closed models are much harder to jailbreak 0:48 Bad news: open-weight models stay trivial to jailbreak 1:13 No GPUs, fine-tuning, or expertise required 1:58 Simple input-space attacks, not just tampering 2:16 Does jailbreaking destroy capabilities? The jailbreak tax 2:48 What can we do? Technical measures that work 3:24 Pretraining filtering and gradient routing 3:55 The government and policy side 4:13 Takeaway: no single fix, but real progress More AI safety research: https://far.ai Alignment Workshop playlist: https://youtube.com/playlist?list=PLBY5kyt_LfFg&si=B9I56daxBmDAeRwQ FAR.AI is a research nonprofit working to ensure the safe development of advanced AI. We host the Alignment Workshop series and publish frontier alignment research.