Build A Reasoning Model From Scratch 2: Loading a Base Model, Text Generation, and KV Caching
The second video in the Build A Reasoning Model (From Scratch) series illustrates how to load a pre-trained model, how LLM text generation works, and how to make it more efficient with KV caching. (These are the basics to get started, later videos will revisit text-generation with sampling in more detail.) - Reasoning playlist: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs - Reasoning Book: https://mng.bz/Nwr7 (Manning) & https://amzn.to/4aAKiFY (Amazon) - Reasoning GitHub repo: https://github.com/rasbt/reasoning-from-scratch - Reasoning From Scratch Playlist: https://www.youtube.com/playlist?list=PLfrHAQCRxtUs Other resources mentioned in the video: - LLMs from Scratch book: https://amzn.to/4fqvn0D - LLMs from Scratch repo: https://github.com/rasbt/LLMs-from-scratch - LLMs from Scratch playlist: https://www.youtube.com/watch?v=yAcWnfsZhzo&list=PLTKMiZHVd_2IIEsoJrWACkIxLRdfMlw11&index=1 - MPS bug article: https://scottkirila.studio.site/blog/kernel-bug - BPE from scratch: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch02/05_bpe-from-scratch - KV cache from scratch https://magazine.sebastianraschka.com/p/coding-the-kv-cache-in-llms 00:00 Introduction and reasoning model demo 01:55 How to work through the book 05:00 Chapter 2 overview 08:25 Checking PyTorch and hardware support 10:26 Apple silicon and MPS caveats 15:00 Cloud GPU options 16:08 Tokens and tokenization 18:20 Qwen3 and the Reasoning From Scratch package 23:05 Encoding and decoding text 26:24 Downloading weights and selecting a device 31:01 Loading the pretrained Qwen3 model 34:32 How LLMs generate text 36:47 Input tensors and batch dimensions 41:48 Running the model in inference mode 44:11 Logits and next-token predictions 49:21 Greedy decoding with argmax 52:28 Building a streaming text generator 01:01:28 Generating text and handling end-of-sequence tokens 01:06:00 Benchmarking text generation 01:14:34 How KV caching works 01:17:22 Adding KV caching and measuring the speedup 01:24:31 Model compilation with torch.compile 01:30:33 Combining compilation with KV caching 01:32:53 Comparing CPU and GPU performance 01:35:32 Recap and next steps