Attention is just dot products | Building LLMs From Scratch - courses.vizuara.ai

Forget everything you know about machine learning for a moment. You are a student with nothing but intuition and mathematics. Here is the problem. You have one query vector, and a set of other vectors. How do you measure the importance of each one relative to the query? Rephrase it and the answer gets closer: what mathematical operation gives you the alignment between two vectors? Here is why alignment is the right idea. Embedding vectors encode meaning. If two vectors point in similar directions - a small angle between them - their meanings are similar. So in "Your journey starts with one step", the vector for "starts" sits closer in direction to "journey" than the vector for "one" does. "starts" should get more importance. The operation that measures this is the **dot product**. Look at the formula: the dot product of two vectors is the product of their magnitudes times the cosine of the angle between them. → Vectors aligned → angle 0 → cos(0) = 1 → dot product is maximum → Vectors perpendicular → angle 90° → cos(90°) = 0 → dot product is zero Higher dot product, more aligned. Lower dot product, less aligned. That is precisely what attention needs — and that is what an attention score is. Building LLMs From Scratch - courses.vizuara.ai