I trained a TINY Reasoning Language Model from scratch with RL | Chapter 4, SLM Training

In RL, we will input a prompt to a model, get it to generate a response, and then inform it whether it was correct or incorrect. The model collects tons and tons of experiences like this, and learns from this feedback about how to respond to prompts correctly... meaning which responses tend to give it the highest scores. In this video, we do all that but for freeform language generation tasks - question answering, fact generation, knowledge graph triplets. Follow me on Twitter: https://x.com/neural_avb To join our Patreon, visit: https://www.patreon.com/NeuralBreakdownwithAVB Members get access to everything behind-the-scenes that goes into producing my videos - including code. Plus, it supports the channel in a big way and helps to pay my bills. Course Video 1 (CPT): https://youtu.be/B8Ur62D3J3U Course Video 2 (SFT): https://youtu.be/gvZIUEL6Ruc Course Video 3 (DPO): https://youtu.be/qQwg2nFtZ1Y Couse Video 4 (RL): https://youtu.be/kxypcfrkUBI Course repo (WIP): https://github.com/avbiswas/finetuning_recipes Dataset generation library: https://github.com/avbiswas/text-albumentations (Harness) Neural-txt: https://github.com/avbiswas/neural-txt Training Dataset (generated locally): https://huggingface.co/datasets/paperbd/paper_instructions_300K-v1 https://huggingface.co/datasets/paperbd/paper_preference_150K-v1 #deeplearning #largelanguagemodels #reinforcementlearning 0:00 - Intro 3:28 - Verifiable vs Unverifiable domains 5:29 - GRPO 8:00 - Reward Function 14:24 - Training part 1 23:35 - Training part 2 28:50 - Training part 3 31:23 - Reward Hacking