How is Deepseek V4 so cheap (A deep dive into Sparse Attention)

To understand Deepseek v4 architecture, we need to first look at the different ways it manipulates the attention components. These are all important and major parts of the architecture: - DeepSeek Sparse Attention with Grouped Query Attention - Compressed Sparse Attention - Heavily Compressed Attention - Sliding Window Attention You can start training cool models and deploy them hassle-free right here: https://inference.net/?utm_source=yt&utm_medium=video&utm_id=avb Study research papers at: paperbreakdown.com Follow me on Twitter: https://x.com/neural_avb Join our Patreon: https://www.patreon.com/NeuralBreakdownwithAVB Patreon Members get access to everything behind-the-scenes that goes into producing my videos - including code. Plus, it supports the channel in a big way and helps to pay my bills. Mixture of Experts: https://youtu.be/Fg8urTOImpY Manifold Hyperconnections: https://youtu.be/rkNrmlDpTKE Check the DeepSeek paper here: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf Timestamps: 0:00 - Intro 1:43 - Training and deploying models 2:45 - Attention Mechanism and KV Cache 6:08 - Compressed Sparse Attention (CSA) 10:14 - Sliding Window Attention (SWA) 12:41 - Heavily Compressed Attention (HCA) 15:37 - MHCs, MoE, MTP, and more!