Position Embeddings: Learned, Sinusoidal, Relative | The attention family series - Lecture 9

This is Lecture 9 of The Attention Family. Attention by itself does not know the order of words: "All dogs are animals" and "All animals are dogs" give every word the same output. In this lecture we see four ways to give attention the position of each word, and where each one enters the attention computation. In this lecture I cover: - The problem: nothing in Q, K, V or the softmax uses the position of a token, so reordering the words only reorders the outputs - Does adding a position vector make "dogs" look like a different word? Measured on the real GPT-2 weights: in 99.999% of cases the closest word vector is still the same word - The big picture: position can enter at the input, at the key, or at the score (and in lecture 10, by rotating the query and the key) - Method 1, learned absolute positions (GPT, BERT, GPT-2): one learned row per position, and why the table stops at the training length - Method 2, sinusoidal positions (Attention Is All You Need, 2017): the formula worked out number by number for position 2, and why each dimension changes at a different speed - Why adding position to the input makes the score depend on the two positions separately, not on how far apart they are - Method 3, relative positions (Shaw et al., 2018): a learned vector for each clipped distance, added to the key inside the score - Method 4, relative bias (T5 = Text-to-Text Transfer Transformer, 2019): one learned number per distance bucket, added to the attention score, and how this moves the attention weights Slides for this lecture: https://lecture9-attention-mechanisms.vercel.app Colab notebook (a small GPT trained twice on short stories, once without and once with position vectors, comparing the loss and the stories they write): https://colab.research.google.com/drive/1TKmSTApZEahX6CMY3oyY-FU7DY--qVmp?usp=sharing Next lecture: RoPE, ALiBi, and how a model trained on short text is made to read 128K tokens. Dr. Sreedath Panat MIT PhD, IIT Madras BTech, Co-founder, Vizuara AI Labs