The 1994 algorithm inside GPT 2 | Building LLMs From Scratch - courses.vizuara.ai

The compression algorithm hiding inside GPT-2. Take this data: "aaabdaaabac" Step 1 - find the most common pair of consecutive bytes. Scan left to right and "aa" wins; it occurs four times. Step 2 - replace that pair with a byte that does not appear in the data. Use Z. → "ZabdZabac" Repeat. Now "ab" is the most frequent pair, appearing twice. Replace it with Y. → "ZYdZYac" Repeat again. The only pair left is "ac", and it appears just once. There is nothing to gain, so the process stops here. "aaabdaaabac" → "ZYdZYac" That replacing of common byte pairs is literally where the word "encoder" comes from. The whole algorithm is three lines: → scan the data left to right → find the byte pair that occurs most → replace it with a byte that does not exist in the data, and repeat That is byte pair encoding. Simple enough to explain in a minute, and it is the tokenizer running underneath GPT-2. Building LLMs From Scratch - courses.vizuara.ai