Foundation Models and Symbolic Regression | AI For Science

At the AI for Science event, Miles Kremer (University of Cambridge) discusses why physics extrapolates far beyond its data while standard machine learning often fails outside training distributions, reframing Occam’s razor as the reuse of familiar, broadly useful concepts rather than an inherent preference for simplicity. He connects symbolic regression and foundation models as two approaches that generalise by reusing building blocks: symbolic regression through domain-relevant operators and compact expressions, and foundation models through shared representations learned from diverse data. He highlights Polymathic AI’s work, including multi-physics pretraining, the Walrus next-step prediction model for simulations, and the Ion astronomy model for joint multimodal embeddings and search. He also presents interpretability experiments that identify and transfer activation-space directions corresponding to physical concepts like vortices and diffusion. Miles Cranmer is Assistant Professor in Data Intensive Science at the University of Cambridge in the Department of Applied Mathematics & Theoretical Physics and the Institute of Astronomy. Miles is a Schmidt Sciences AI2050 Early Career Fellow. Miles and his group work broadly on AI for the physical and mathematical sciences, with a particular focus on improving the ability of AI to generalise to out-of-distribution data. 00:00 Speaker Introduction 00:29 Why Physics Generalizes 01:51 Rethinking Occams Razor 04:45 Reuse Beats Training Scratch 07:48 ML Failure Case Pneumonia 09:00 Symbolic Regression Basics 10:48 Operators As Priors 13:11 Foundation Models Connection 18:20 Polymathic AI Mission 19:21 Multi Physics Pretraining 21:51 Cat Videos Transfer Surprise 24:03 Data Aggregation Datasets 25:07 Walrus Physics Model 28:38 Ion Astronomy Model 32:41 Ion Search With Embeddings 35:30 Interpreting Learned Physics 40:25 Takeaways And Closing