How tokenizers turn text into numbers

A tokenizer splits text into tokens and maps them to numeric IDs. I walk through this process with the Qwen3 tokenizer, then show how decoding reconstructs the original text. An excerpt from my reasoning model series: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs&index=1