OpenAI Astra and Recurrent Depth / Looped Transformers

A short video explaining the rumored recurrent depth (looped transformer) architectural design in OpenAI's new (upcoming) Astra model. In particular, I discuss what looped transformers are and go over the related literature. UPDATE: Expanding on the video, I also put together a longer write-up on the topic: https://magazine.sebastianraschka.com/p/gpt-6-astra-looped-transformers-and Other links and references The Information article on OpenAI Astra: https://www.theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns Looped depth sharing explainer: https://sebastianraschka.com/llm-architecture-gallery/looped-depth-sharing/ LLM Architecture Gallery: https://www.sebastianraschka.com/llm-architecture-gallery/ Nanbeige4.2-3B technical report: https://arxiv.org/abs/2607.22083 Mixture-of-Recursions: https://arxiv.org/abs/2507.10524 Universal Transformers: https://arxiv.org/abs/1807.03819 Scaling up Test-Time Compute with Latent Reasoning: https://arxiv.org/abs/2502.05171 00:00 OpenAI Astra and the Recurrent-Depth Rumor 01:37 Claims About Recurrent Depth 03:02 Nanbeige4.2-3B’s Looped Transformer 05:21 Why Reuse the Same Layers? 07:43 Training From Scratch vs. Retrofitting 09:47 Why Use Two Passes? 10:35 KV-Cache Implications 12:00 Mixture-of-Recursions and Dynamic Depth 16:18 Validation-Loss Comparison 18:16 Attention and KV Caching With Dynamic Depth 19:55 Recurrent Depth and Chain of Thought 23:26 Recursive Self-Improvement 24:27 Universal Transformers 25:27 Latent Reasoning With Recurrent Depth 27:09 Summary Corrections: 3:09 I incorrectly said that Nanbeige was developed in India. Nanbeige4.2-3B was developed by Nanbeige LLM Lab and Boss Zhipin in China. I had confused it with Sarvam AI, which is based in India. If you enjoyed this video, I also write about LLM research and model architectures in my Ahead of AI newsletter: https://magazine.sebastianraschka.com/