Why word position matters? | Building LLMs From Scratch - courses.vizuara.ai

Token embeddings have a blind spot: position. Take two sentences. → "The cat sat on the mat." → "On the mat, the cat sat." The word "cat" appears in both — in completely different places. But its token embedding is identical in both cases. The vector carries meaning and nothing else. Position matters for understanding a sentence. So it has to be encoded somewhere too. That is positional encoding. There are two kinds: absolute and relative. In absolute positional encoding, each position in the sequence gets its own embedding vector, which is added to the token embedding. Take four tokens — cat, sat, on, the — and suppose for simplicity their token embeddings are all [1, 1, 1]. Their positional embeddings differ, because their positions differ: → cat, position 1 → [1.1, 1.2, 1.3] → sat, position 2 → [2.1, 2.2, 2.3] → on, position 3 → [3.1, 3.2, 3.3] → the, position 4 → [4.1, 4.2, 4.3] Add them together and you get the input embeddings: [2.1, 2.2, 2.3], [3.1, 3.2, 3.3], [4.1, 4.2, 4.3], [5.1, 5.2, 5.3]. Identical token embeddings going in. Four distinct vectors coming out. The position is now inside the numbers. Building LLMs From Scratch - courses.vizuara.ai