Liam Fedus & Barret Zoph - AI scaling with mixture of expert models

Liam Fedus and Barrett Zoph, research scientists at Google Brain, came on the podcast to talk about AI scaling, sparsity and the present and future of MoE models. Intro music: ➞ Artist: Ron Gelinas ➞ Track Title: Daybreak Chill Blend (original mix) ➞ Link to Track: https://youtu.be/d8Y2sKIgFWc 0:00 Intro 2:15 Guests’ backgrounds 8:00 Understanding specialization 13:45 Speculations for the future 21:45 Switch transformer versus dense net 27:30 More interpretable models 33:30 Assumptions and biology 39:15 Wrap-up