Token-Efficient Long Video Understanding for Multimodal LLMs | Paper explained
Long videos are a nightmare for language models—too many tokens to handle, plus many tokens are redundant, slow inference, and limited context windows. STORM ⛈️ changes that. In this AI Coffee Break, we explain STORM, a new architecture from NVIDIA and collaborators that improves long video understanding using Mamba layers for temporal modeling and token compression. The result? Better accuracy than GPT-4o on key benchmarks and up to 8× more efficiency. AI Coffee Break Merch! 🛍️ https://ai-coffee-break-with-letitia.myspreadshop.de/ ☕ Grab a cup of coffee and learn: • Why current Video LLMs struggle with long sequences • How STORM uses Mamba to inject temporal context before compression • How it reduces visual tokens by a factor of 8—without sacrificing performance • Benchmarks: MVBench, MLVU, and beyond • Why this is a big step toward real-world video comprehension 📃 Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang et al. "Token-Efficient Long Video Understanding for Multimodal LLMs." (2025) https://arxiv.org/abs/2503.04130 Outline: 00:00 Long sequence struggle 01:36 Video LLMs so far 02:20 The idea for STORM 03:31 Training details 07:01 Results 08:18 Discussion Thanks to our Patrons who support us in Tier 2, 3, 4: 🙏 Dres. Trost GbR, Siltax, Vignesh Valliappan, Michael, Sunny Dhiana, Andy Ma ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 🔥 Optionally, pay us a coffee to help with our Coffee Bean production! ☕ Patreon: https://www.patreon.com/AICoffeeBreak Ko-fi: https://ko-fi.com/aicoffeebreak Join this channel as a Bean Member to get access to perks: https://www.youtube.com/channel/UCobqgqE4i5Kf7wrxRxhToQA/join ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 🔗 Links: AICoffeeBreakQuiz: https://www.youtube.com/c/AICoffeeBreak/community Twitter / X: https://twitter.com/AICoffeeBreak LinkedIn: https://www.linkedin.com/in/letitia-parcalabescu/ Threads: https://www.threads.net/@ai.coffee.break Bluesky: https://bsky.app/profile/aicoffeebreak.bsky.social Reddit: https://www.reddit.com/r/AICoffeeBreak/ YouTube: https://www.youtube.com/AICoffeeBreak Substack: https://aicoffeebreakwl.substack.com/ Web: https://explanationmark.de/letitia https://aicoffeebreak.com #AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research Video editing: Nils Trost