Greedy? Min-p? Beam Search? How LLMs Actually Pick Words – Decoding Strategies Explained
How do large language models like ChatGPT actually decide which word comes next? In this video, we break down the core decoding strategies used in text generation: from greedy decoding to top-k, top-p (nucleus sampling), and the newer min-p sampling. You’ll learn how these methods turn probability distributions into actual words, and how just changing the sampling strategy can make the same model sound repetitive, brilliant, or totally unpredictable. We also show hands-on examples using GPT-2 to demonstrate how decoding choices affect output, without changing the model itself. AI Coffee Break Merch! 🛍️ https://ai-coffee-break-with-letitia.myspreadshop.de/ 💻 Collab used in this video: https://drive.google.com/file/d/14OyZRyUE3iVTVK5MG8ZyXReuzZwgg5Xg/view?usp=sharing 📃 Minh, Nguyen Nhat, Andrew Baker, Clement Neo, Allen Roush, Andreas Kirsch, Wand AI Independent, Ravid Shwartz-Ziv, and Wand AI. "Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs." In The Thirteenth International Conference on Learning Representations. 2024. https://arxiv.org/abs/2407.01082 Outline: 00:00 From Probabilities to Words 00:57 Recap: Next Token Prediction Basics 02:05 Deterministic: Greedy Decoding 02:55 Why Sampling Matters 03:52 Random Decoding 04:56 Top-k Sampling 05:46 Top-p Sampling (Nucleus Sampling) 06:54 Temperature 08:11 Min-p Sampling 09:29 Repetition & Frequency Penalty 09:52 Beam Search 10:43 Summary & Takeaways Thanks to our Patrons who support us in Tier 2, 3, 4: 🙏 Dres. Trost GbR, Siltax, Vignesh Valliappan, Michael, Sunny Dhiana, Andy Ma ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 🔥 Optionally, pay us a coffee to help with our Coffee Bean production! ☕ Patreon: https://www.patreon.com/AICoffeeBreak Ko-fi: https://ko-fi.com/aicoffeebreak Join this channel as a Bean Member to get access to perks: https://www.youtube.com/channel/UCobqgqE4i5Kf7wrxRxhToQA/join ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 🔗 Links: AICoffeeBreakQuiz: https://www.youtube.com/c/AICoffeeBreak/community Twitter / X: https://twitter.com/AICoffeeBreak LinkedIn: https://www.linkedin.com/in/letitia-parcalabescu/ Threads: https://www.threads.net/@ai.coffee.break Bluesky: https://bsky.app/profile/aicoffeebreak.bsky.social Reddit: https://www.reddit.com/r/AICoffeeBreak/ YouTube: https://www.youtube.com/AICoffeeBreak Substack: https://aicoffeebreakwl.substack.com/ Web: https://explanationmark.de/letitia https://aicoffeebreak.com #AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research Video editing: Nils Trost Music 🎵 : Space Navigator – Sarah, The Illstrumentalist