Why a 10x Smaller Open Model Beats the Frontier

Ryan Wang started his PhD at MIT in 2018, right as the transformer architecture was reshaping AI, and spent late nights restarting GPU racks in a campus basement. Years later, the startup he co-founded, Eigen AI, would rank first among inference providers serving the same open model, earning a shoutout in Jensen Huang’s 2026 GTC keynote. Along the way he learned that at scale, the model is the easy part. The real challenge is in everything it takes to serve it. In this episode, Anu and Markiesha sit down with Ryan, now the SVP of Applied AI at Nebius, to unpack why inference optimization has become one of the biggest levers in AI economics including how optimization is now squeezing four to five times more tokens out of the same GPUs, and why a fine-tuned 70B open model can beat frontier closed models on specific tasks. Ryan also shares the prompt-design choice that keeps your cache intact, why cached context costs a tenth of recomputing it, and when founders should buy inference instead of building a team and delves into why cheaper tokens will grow the inference market rather than shrink it. What You’ll Learn: • Why everything about inference changes at ten times the scale • The prompt-design choice that keeps your cache intact • How open models can outperform frontier closed models • When founders should buy inference as a service and when to build an in-house team • Why Ryan thinks the gap between open and closed models has shrunk from a year to a few months • The story behind Eigen AI’s monumental GTC shoutout • Why he sees dramatic expansion of the inference market • What changes he sees coming to inference infra, including agentic AI infrastructure Learn more: www.nebius.com/podcast Timestamps 00:00 What Is Inference Optimization and Why Does It Matter? (Ryan Wong, Nebius Token Factory) 01:39 How Does an AI Researcher Become an Inference Optimization Founder? 05:17 How Do Sparse Attention and Pruning Make LLM Inference Faster? 07:39 Why Is Scaling AI Inference Harder Than Running a Single Model? 10:14 Why Does Open Source Matter for AI? Fine-Tuned Small Models vs. Closed Frontier Models 11:51 What Is KV Cache and How Do You Manage Cache Hit Rates at Scale? 13:52 Should Startups Build Their Own Inference Stack or Buy Inference as a Service? 15:40 How Did Eigen AI Reach #1 Inference Speed in Jensen Huang's GTC Keynote? 23:19 What's Next for AI Inference? Falling Token Costs, Jevons Paradox, and Specialized Models 29:49 How Fast Is the Open vs. Closed Model Gap Closing?