LLM Building Blocks & Transformer Alternatives

Resources: - Understanding and Coding the KV Cache in LLMs from Scratch: https://magazine.sebastianraschka.com/p/coding-the-kv-cache-in-llms - The Big Architecture Comparison: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison - Beyond Standard LLMs: Linear Attention Hybrids, Text Diffusion, Code World Models, and Small Recursive Transformers - Reasoning From Scratch book: https://mng.bz/Nwr7 Description: Learn the core components of modern transformer-based large language models (LLMs) and the practical techniques that make inference faster and cheaper. We walk through Grouped-Query Attention (GQA), Multi-Head Latent Attention (MLA), and Sliding Window Attention (SWA), show where Mixture of Experts (MoE) fits into today’s architectures, and finish with a look at promising alternatives and hybrid models beyond standard transformers. Chapters: 00:00 Intro 01:13 Main theme: bigger models and cheaper inference 02:13 Grouped-Query Attention (GQA) 05:44 Multi-Head Latent Attention (MLA) 09:51 Sliding Window Attention (SWA) 13:57 Mixture of Experts 17:01 LLM and transformer alternatives #LLM #Transformers #DeepLearning #MachineLearning #Inference #MoE #GQA #MLA #SWA #KVCache