The Race to Production-Grade Diffusion LLMs [Stefano Ermon] - 764

Today, we're joined by Stefano Ermon, associate professor at Stanford University and CEO of Inception Labs to discuss diffusion language models. We dig into how diffusion approaches—traditionally used for images—are being adapted for text and code generation, the technical challenges of applying continuous methods to discrete token spaces, and how diffusion models compare to traditional autoregressive LLMs. Stefano introduces Mercury 2, a commercial-scale diffusion LLM that can generate multiple tokens simultaneously and achieve inference speeds 5-10x faster than small frontier models, paving the way for latency-sensitive applications like voice interactions and fast agentic loops. We also cover the open research challenges in diffusion LLM training, serving infrastructure requirements, and post-training for diffusion-based systems. Finally, Stefano shares his perspective on whether diffusion models can rival or surpass autoregressive LLMs at scale, the advantages for highly controllable generation, and what the future of multimodal diffusion models might look like. 🗒️ For the full list of resources for this episode, visit the show notes page: https://twimlai.com/go/764. 🔔 Subscribe to our channel for more great content just like this: https://youtube.com/twimlai?sub_confirmation=1 🗣️ CONNECT WITH US! =============================== Subscribe to the TWIML AI Podcast: https://twimlai.com/podcast/twimlai/ Follow us on Twitter: https://twitter.com/twimlai Follow us on LinkedIn: https://www.linkedin.com/company/twimlai/ Join our Slack Community: https://twimlai.com/community/ Subscribe to our newsletter: https://twimlai.com/newsletter/ Want to get in touch? Send us a message: https://twimlai.com/contact/ 📖 CHAPTERS =============================== 00:00 - Introduction 04:11 - Origins of diffusion models 07:24 - From image diffusion to text diffusion 08:07 - Discrete data challenges 09:54 - Limitations of embeddings 11:10 - Diffusion versus autoregressive models 14:31 - Masking 19:27 - Reasoning in diffusion models 20:58 - Context windows and variable-length outputs 25:36 - RL pre-training and post-training 28:44 - Can autoregressive models be converted to diffusion? 31:11 - Serving challenges for diffusion models 34:50 - Speed, cost, quality tradeoffs 40:07 - Current limitations of Mercury 2 model 43:06 - Open science questions 45:22 - Hallucinations and generalization challenges 47:46 - Controllable generation 51:41 - Will diffusion overtake autoregressive models? 53:57 - Diffusion in agentic applications 56:00 - Gemini diffusion 57:51 - Key research groups working on diffusion models 01:00:09 - Cross-pollination between image and text diffusion 01:01:18 - Future of multimodal diffusion models 🔗 LINKS & RESOURCES =============================== Inception - https://www.inceptionlabs.ai/ Ermon Group - https://cs.stanford.edu/~ermon/website/index.html Introducing Mercury 2 - https://www.inceptionlabs.ai/blog/introducing-mercury-2 Domain Knowledge in Machine Learning Models for Sustainability with Stefano Ermon - 15 - https://twimlai.com/podcast/twimlai/domain-knowledge-in-machine-learning-models-for-sustainability 📸 Camera: https://amzn.to/3TQ3zsg 🎙️Microphone: https://amzn.to/3t5zXeV 🚦Lights: https://amzn.to/3TQlX49 🎛️ Audio Interface: https://amzn.to/3TVFAIq 🎚️ Stream Deck: https://amzn.to/3zzm7F5