I Trained Qwen 4B with Distillation to Catch Qwen 27B

Can a 27 billion parameter model teach a 4 billion model to be as good as itself? Last time, RL got Qwen 4B from 52% to 58.4% and then hit a wall. This time I used Qwen 27B as a teacher and tried all three kinds of distillation on two DGX Sparks: training on the teacher's answers, training on its probabilities, and on-policy distillation where the 27B grades every token the 4B writes. Sponsored by Dex, the talent agent for senior engineers. Free for engineers: https://meetdex.link/promptengineering-1 Links - Previous video (RL, the sandbox, the 58% wall): https://youtu.be/Pl_2TelxjdY - Code: https://github.com/PromtEngineer/smoldataenvs-grpo-spark - SmolDataEnvs (the tasks): https://huggingface.co/datasets/FineEnvs/SmolDataEnvs - Thinking Machines, On-Policy Distillation: https://thinkingmachines.ai/blog/on-policy-distillation/ - Qwen3 Technical Report (strong-to-weak distillation): https://arxiv.org/abs/2505.09388 - Hinton, Vinyals, Dean (2015), Distilling the Knowledge in a Neural Network: https://arxiv.org/abs/1503.02531 - On-policy distillation papers: MiniLLM https://arxiv.org/abs/2306.08543 · GKD https://arxiv.org/abs/2306.13649 Chapters 00:00 - Introduction & The Distillation Idea 02:21 - Benchmark Setup & Small Data Environments 03:35 - Why Reinforcement Learning Hit a Wall 04:40 - The 3 Ways to Distill LLMs 08:04 - Hardware Setup (Dual DGX Spark) 08:37 - Method 1: Training on Teacher Answers 10:00 - Sponsor Break 11:00 - Method 2: Distilling Teacher Probabilities 13:30 - Method 3: Live On-Policy Distillation 16:42 - Comparing All 6 Experiment Results 17:27 - The 58% Ceiling & What the Small Model Actually Knows 19:07 - The 4-Tries Headroom Test 20:24 - Hardware & Practical Implementation Tips 21:21 - Recommendations & Which Method to Use 22:19 - Next Steps: Testing 2B and 9B Models #distillation #qwen #llm #reinforcementlearning #localai #dgxspark