PyTorch Walkthrough of Muon, Dion, and Orthogonalized Optimizer Variants

Muon has attracted a lot of attention for fast convergence, but getting those optimizers to work in a real training stack is a different challenge. At PyTorch Conference North America, Jen Wei will walk through Muon, Dion, and Dion3 and look at what happens when you add the messy details: sharding, communication, schedulers, and everything else that can go wrong. If you are interested in optimizers and training dynamics, see you at PyTorchCon NA. Get your ticket: https://hubs.la/Q04v4SL60