On Policy Distillation - How the big AI labs actually train their LLMs

A visual breakdown of On Policy Distillation (OPD), On Policy Self Distillation (OPSD), and how AI labs use these techniques to train Large Language Models! Study research papers at: paperbreakdown.com I am building this, please go check it out and give me feedback! Follow me on Twitter: https://x.com/neural_avb To join our Patreon, visit: https://www.patreon.com/NeuralBreakdownwithAVB Members get access to everything behind-the-scenes that goes into producing my videos - including code and additional material. Plus, it supports the channel in a big way and helps to pay my bills. Links: Thinking Machines Blog: https://thinkingmachines.ai/blog/on-policy-distillation/ Great article on X: https://x.com/willcb/status/2050038277454143918 SDPO article: https://x.com/neural_avb/status/2021216205684781094 Dual On Policy Distillation: https://x.com/neural_avb/status/2075632209038582131 Great git repo: https://github.com/chrisliu298/awesome-on-policy-distillation Timestamps: 0:00 - Intro 2:06 - On Policy Distillation 11:15 - Reverse KL vs Forward KL 18:13 - On Policy Self Distillation 23:20 - Privileged Illusion 24:52 - When to use what?