Direct Preference Optimization, Model Alignment, and Post training | Chapter 3, SLM Training

The goal of preference optimization is to teach the model: "which response is good" and "which response is bad"... We will learn about DPO (Direct Preference Optimization) and how to generate preference datasets and train with Unsloth and Huggingface TRL. Follow me on Twitter: https://x.com/neural_avb To join our Patreon, visit: https://www.patreon.com/NeuralBreakdownwithAVB Members get access to everything behind-the-scenes that goes into producing my videos - including code. Plus, it supports the channel in a big way and helps to pay my bills. You can start training cool models and deploy them hassle-free right here: https://inference.net/?utm_source=yt&utm_medium=video&utm_id=avb Study research papers at: paperbreakdown.com Follow me on Twitter: https://x.com/neural_avb --- Course Video 1 (CPT): https://youtu.be/B8Ur62D3J3U Course Video 2 (SFT): https://youtu.be/gvZIUEL6Ruc Course Video 3/this one (DPO): https://youtu.be/qQwg2nFtZ1Y Course repo (WIP): https://github.com/avbiswas/finetuning_recipes Dataset generation library: https://github.com/avbiswas/text-albumentations (Harness) Neural-txt: https://github.com/avbiswas/neural-txt Training Dataset (generated locally): https://huggingface.co/datasets/paperbd/paper_instructions_300K-v1 https://huggingface.co/datasets/paperbd/paper_preference_150K-v1 ---