High-Efficiency Diffusion Models for On-Device Image Generation and Editing [Hung Bui] - 753

In this episode, Hung Bui, Technology Vice President at Qualcomm, joins us to explore the latest high-efficiency techniques for running generative AI, particularly diffusion models, on-device. We dive deep into the technical challenges of deploying these models, which are powerful but computationally expensive due to their iterative sampling process. Hung details his team's work on SwiftBrush and SwiftEdit, which enable high-quality text-to-image generation and editing in a single inference step. He explains their novel distillation framework, where a multi-step teacher model guides the training of an efficient, single-step student model. We explore the architecture and training, including the use of a secondary 'coach' network that aligns the student's denoising function with the teacher's, allowing the model to bypass the iterative process entirely. Finally, we discuss how these efficiency breakthroughs pave the way for personalized on-device agents and the challenges of running reasoning models with techniques like inference-time scaling under a fixed compute budget. 🗒️ For the full list of resources for this episode, visit the show notes page: https://twimlai.com/go/753. 🔔 Subscribe to our channel for more great content just like this: https://youtube.com/twimlai?sub_confirmation=1 🗣️ CONNECT WITH US! =============================== Subscribe to the TWIML AI Podcast: https://twimlai.com/podcast/twimlai/ Follow us on Twitter: https://twitter.com/twimlai Follow us on LinkedIn: https://www.linkedin.com/company/twimlai/ Join our Slack Community: https://twimlai.com/community/ Subscribe to our newsletter: https://twimlai.com/newsletter/ Want to get in touch? Send us a message: https://twimlai.com/contact/ 📖 CHAPTERS =============================== 00:00 - Introduction 04:44 - VinAI Research 07:03 - Building an AI team 09:02 - First AI residency program 09:47 - Focus on model efficiency 11:49 - Training a Vietnamese LLM 16:29 - Model optimizations for Vietnamese LLM 17:54 - Benchmarks 19:33 - Image generation 22:35 - Distillation 24:30 - Secondary 'coach' network 28:03 - Results 29:55 - SwiftEdit paper 35:46 - On-device agents 37:26 - Research directions 40:44 - Inference time-scaling on mobile devices 46:35 - Lessons learned on acquisition 48:16 - Qualcomm AI Residency Program 🔗 LINKS & RESOURCES =============================== SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation - https://arxiv.org/abs/2312.05239 SwiftEdit: Lightning Fast Text-Guided Image Editing via One-Step Diffusion - https://arxiv.org/abs/2412.04301 PhoGPT: Generative Pre-training for Vietnamese - https://arxiv.org/abs/2311.02945 Qualcomm AI Residency Program - https://www.qualcomm.com/research/artificial-intelligence/ai-residency-program Distilling Transformers and Diffusion Models for Robust Edge Use Cases with Fatih Porikli - 738 - https://twimlai.com/podcast/twimlai/distilling-transformers-and-diffusion-models-for-robust-edge-use-cases/ 📸 Camera: https://amzn.to/3TQ3zsg 🎙️Microphone: https://amzn.to/3t5zXeV 🚦Lights: https://amzn.to/3TQlX49 🎛️ Audio Interface: https://amzn.to/3TVFAIq 🎚️ Stream Deck: https://amzn.to/3zzm7F5