Building a GPT from Scratch with yoctoGPT in PyTorch (2)

Scaling Up yoctoGPT: BPE Tokenization, Model Variants — Session 2 🚀 Scale up from a tiny character-level GPT to a production-style training pipeline — benchmarked across four Colab GPUs including the new 96 GB G4. In this second session of our free 3-part webinar series, Dr. Yves J. Hilpisch shows how three improvements — more data, better tokenization, and optimised architectures — unlock better quality and faster training without changing the core codebase. 📐 Topics in this session: • Scaling the character-level corpus: 42× more text, 9.2M characters • Notebook 2: large char-level training with resume and early stopping • Why BPE tokenization beats character-level encoding • Notebook 3: token-level training with subword units • Vocab size trade-off: 4000 vs 8000 tokens • Model variants: gpt_fast (SDPA, FlashAttention) vs gpt_plus (RoPE, RMSNorm, SwiGLU) • Performance tools: AMP (bf16/fp16), auto-microbatch, torch.compile, KV cache • Regularisation at scale: dropout, weight decay, label smoothing, EMA • Benchmarking across T4, L4, A100, and G4 (RTX PRO 6000 Blackwell) ⚡ Measured results across Colab GPUs: • Char-level val loss: 1.36 → 1.10 (T4) → 0.97 (L4) → 0.94 (A100) → 0.89 (G4) • More data eliminates overfitting — the larger G4 model (12L/768d) wins on the 9.2M char corpus • Token-level val loss: 4.49 (T4) → 3.87 (L4) → 3.59 (A100, vocab 4000) • G4 token runs: val loss 3.65 (vocab 4000) and 4.08 (vocab 8000) — bigger model overfits the 2.6M token corpus • Key insight: bigger models help when the corpus is large enough; on small data they overfit 🔗 Resources: 📑 Slides: https://tpq.io/tpq_yoctogpt_2.pdf 📂 Notebook repo: https://github.com/yhilpisch/yoctoGPT 📊 TDS: https://thedatascientist.dev 📈 CPF: https://python-for-finance.com 🤖 TAE: https://theaiengineer.dev ₿ TCE: https://thecryptoengineer.dev 📌 Disclaimer: This content is provided solely for illustration and educational purposes. Training texts consist primarily of public-domain works. No warranties or guarantees of any kind are given. Use at your own risk. #GPT #PyTorch #Transformer #BPE #Tokenization #LanguageModel #GoogleColab #AIEngineering #LLM #DeepLearning #FlashAttention #yoctoGPT #ScalingUp