Why embeddings are not enough? | Building LLMs From Scratch - courses.vizuara.ai

Why embeddings alone are not enough. Take the sentence: "Your journey starts with one step." The embedding for "journey" captures what the word means. Genuinely useful. But it tells you nothing about how "journey" relates to the other words sitting next to it. How important is "your" to "journey"? What about "starts"? Or "step"? The embedding vector has none of that. And if you are trying to predict the next word, that is exactly the information you need — which words in this sentence are closely tied to "journey". Put differently: how much attention should each word receive when you are looking at "journey"? That is what the attention mechanism produces. It takes the embedding vector and transforms it into a context vector — an enriched embedding. It still carries the semantic meaning. It now also carries how that word relates to every other word in the sentence. Every embedding vector gets a corresponding context vector. And that is the entire goal of attention. Simplified attention, causal attention, the multi-head attention inside modern LLMs — different machinery, one job: Turn embedding vectors into context vectors. Building LLMs From Scratch - courses.vizuara.ai