Why Models Are AI’s Next Training Dataset | Damian Borth

For more than a decade, AI has advanced by training ever-larger models on ever-larger datasets. But as high-quality training data becomes harder to find and pretraining grows increasingly expensive, researchers are looking for new ways to keep foundation models improving. In this episode, Damian Borth, professor of AI and machine learning at the University of St. Gallen, argues we’ve been overlooking an important source of knowledge: the models we’ve already trained. His group’s work on weight space learning treats trained neural networks themselves as data, learning from the distilled results of millions of GPU hours of optimization rather than starting from raw data each time. We explore what it means to build foundation models of neural networks, how knowledge can be transferred across architectures and domains, why this approach could dramatically reduce the cost of developing specialized models, and whether future AI systems may be trained on collections of existing models instead of ever-growing datasets. 🗒️ Full show notes: https://twimlai.com/go/772. 🔔 Subscribe to our channel for more great content just like this: https://youtube.com/twimlai?sub_confirmation=1 📖 CHAPTERS =============================== 00:00 - Introduction 01:46 - Research focus 10:50 - Weight space learning 17:17 - Symmetries and transformations in weight space 21:11 - Training on the Hugging Face Model Zoo 22:59 - Filtering models 25:05 - Tokenization and normalization 31:35 - Compression, pruning, and distillation techniques 33:46 - Privacy-preserving dataset prompting 36:50 - How generated weights map to architectures 41:17 - Task vectors 42:07 - High vs low frequency noise 43:38 - Workshops and research direction 🗣️ CONNECT WITH US! =============================== Subscribe to the TWIML AI Podcast: https://twimlai.com/podcast/twimlai/ Follow us on Twitter: https://twitter.com/twimlai Follow us on LinkedIn: https://www.linkedin.com/company/twimlai/ Join our Slack Community: https://twimlai.com/community/ Subscribe to our newsletter: https://twimlai.com/newsletter/ Want to get in touch? Send us a message: https://twimlai.com/contact/