Prefill vs Decode

Inference is not one single process. This lesson breaks down its two phases: prefill, where the model reads your entire prompt at once, and decode, where it generates one token at a time, plus the KV cache that ties them together. 🖊️ Learning objectives: - What happens during prefill and decode - What the KV cache stores and why it matters - Why prefill is compute bound and decode is memory bound As the KV cache grows with every token and you multiply that across thousands of users, memory becomes the critical bottleneck. That is exactly the problem we set out to solve at SambaNova, designing a chip around the data flow graph itself. For more resources, you may check out our blog here, where you will find information on: - The decode era of AI: why dataflow matters more than ever - Solving the decode bottleneck: why agentic inference needs hybrid hardware #AI #Inference #KVCache #LLM #SambaNova