Self-Distillation: A New Way to Teach and Align AI Models | Andreas Krause (ETH Zürich)

Andreas Krause (ETH Zurich) on self-distillation, a third way to teach AI models beyond supervised learning and reinforcement learning. Krause presents self-distillation as an alternative to supervised learning (demonstrations) and reinforcement learning (trial and error, prone to reward hacking). The idea: turn transient in-context learning into durable in-weight learning by letting the model act as its own teacher, reflecting on directional feedback, an error message, a user's correction, or a constitution, to produce a training signal. Their method, Self-Distillation Policy Optimization, gives token-level credit assignment, pinpointing which token caused a wrong answer instead of penalizing a whole response. He shows gains on coding (LiveCodeBench), on learning from real user preferences (the WildChat dataset), and on constitutional alignment of Apertus, an openly trained multilingual model, using the Swiss AI Charter, with higher alignment scores at little cost to capability. Chapters 0:00 Self-distillation for alignment 0:30 How we teach models today: supervision vs. RL 1:09 The idea: rich directional feedback 1:55 From in-context learning to in-weight learning 2:47 Self-Distillation Policy Optimization (SDPO) 3:45 Token-level credit assignment (a coding example) 4:48 Results on coding (LiveCodeBench) 5:10 Learning from user preferences 6:17 Results on real conversations (WildChat) 7:10 Constitutional alignment on Apertus 8:09 Takeaways More AI safety research: https://far.ai Alignment Workshop playlist: https://youtube.com/playlist?list=PLBY5kyt_LfFg&si=JKJxHkO-MLker_fi FAR.AI is a research nonprofit working to ensure the safe development of advanced AI. We host the Alignment Workshop series and publish frontier alignment research.