Building a GPT from Scratch with yoctoGPT in PyTorch (1)

GPT from Scratch: Architecture and Character-Level Training — Session 1 🧠 Build a working GPT model from scratch in pure PyTorch — and train it on a free Google Colab T4 GPU in under a minute. In this first session of our free 3-part webinar series, Dr. Yves J. Hilpisch walks through the yoctoGPT project: a minimal, readable GPT implementation with no external model frameworks, no magic, just clear PyTorch code. 📐 Topics in this session: • Why build a GPT from scratch? Understanding over abstraction • Transformer architecture refresher: self-attention, embeddings, residuals • yoctoGPT project overview: modules, CLIs, installation • Character-level data preparation pipeline • Notebook 1: training a tiny GPT on a philosophy corpus • Sampling strategies: temperature, top-k, top-p (nucleus) • Text quality metrics: readability, diversity, Char-KL ⚡ Measured results on a free T4 GPU: • Tiny model (2L/128d): ~40s, val loss 1.36, ~75 it/s • Deeper model (6L/384d): ~109s, val loss 1.37, 13.7 it/s • Deeper gpt_fast (SDPA): ~219s, val loss 1.41, 10.3 it/s • Key insight: on a small corpus, the tiny model wins — more parameters without more data means overfitting 🔗 Resources: 📑 Slides: https://tpq.io/tpq_yoctogpt_1.pdf 📂 Notebook repo: https://github.com/yhilpisch/yoctoGPT 📊 TDS: https://thedatascientist.dev 📈 CPF: https://python-for-finance.com 🤖 TAE: https://theaiengineer.dev ₿ TCE: https://thecryptoengineer.dev 📌 Disclaimer: This content is provided solely for illustration and educational purposes. Training texts consist primarily of public-domain works. No warranties or guarantees of any kind are given. Use at your own risk. #GPT #PyTorch #Transformer #MachineLearning #DeepLearning #CharacterLevel #LanguageModel #GoogleColab #AIEngineering #LLM #AttentionIsAllYouNeed #yoctoGPT