Closing the Sim-to-Real Gap: Training Humanoids in 3D Gaussian Environments
Learn how to turn real-world captures into photorealistic simulation environments for training and validating humanoid robots with NVIDIA Isaac Sim and Isaac Lab. Join Eugene Chong (Niantic Spatial), Julian Nubert (Flexion), and Gavriel State (NVIDIA) as they demonstrate a 3D Gaussian splatting-based end-to-end real-to-sim workflow using Niantic Spatial's capture and reconstruction pipeline, NVIDIA Omniverse NuRec, Isaac Sim and Isaac Lab, and Flexion's skill factory. You'll see how to reconstruct environments, train robot policies, test robotics software, and prepare humanoid systems for real-world deployment. What you'll learn - Create simulation-ready environments with 3D Gaussian splatting and collision meshes - Render Gaussian splat reconstructions with NVIDIA Omniverse NuRec in Isaac Sim - Train humanoid robot policies in NVIDIA Isaac Lab - Test robotics software in NVIDIA Isaac Sim - Build interoperable simulation workflows with OpenUSD - Close the sim-to-real gap for humanoid robotics ---------------------------------------------------------------------------- Chapters: 0:00 Welcome and Sim-to-Real Humanoid Training Overview 1:16 Niantic Spatial, Flexion, and NVIDIA Isaac Speakers 5:51 Sim-to-Real Robotics and Domain Randomization 9:51 Niantic Spatial Real-to-Sim Reconstruction Pipeline 12:32 Simulation-Ready 3D Gaussian Splatting Quality 14:39 3DGS Rendering and Collision Mesh Layers 17:31 360 Camera Scan to USDZ for Isaac Sim 18:22 Flexion Humanoid RL Training and RGB Policy Stack 24:47 Real Office Digital Twin Navigation Demo 28:51 3D Gaussian Splatting vs NeRF for Robotics 30:54 NVIDIA NuRec Ray Tracing, Isaac Sim, and Particle Fields 35:56 Isaac Lab 3, Newton, OVRTx, and Deformable Physics 39:33 Interactable Scenes and Long-Horizon Humanoid Tasks 42:46 Q&A: Splats, Collision Meshes, Dynamic Objects, and Compute 48:26 Q&A: Squishy Objects, 4D Reconstruction, and VMP 55:17 Isaac Sim, NuRec, Discord, and GTC Berlin Resources 1:00:32 Real-to-Sim Takeaways for Physical AI Developers ---------------------------------------------------------------------------- List of Resources: - Companion article: Niantic Spatial, Flexion, and NVIDIA: Closing the Sim2Real Gap for Humanoids - https://flexion.ai/news/niantic-spatial-flexion-and-nvidia-closing-the-sim2real-gap-for-humanoids - Scaniverse by Niantic Spatial - https://scaniverse.nianticspatial.com - NVIDIA Omniverse NuRec - https://developer.nvidia.com/omniverse/nurec - NVIDIA Isaac Sim - https://developer.nvidia.com/isaac/sim - NVIDIA Isaac Lab - https://developer.nvidia.com/isaac/lab - Newton Physics Engine - https://developer.nvidia.com/newton-physics - NVIDIA Kaolin - https://github.com/NVIDIAGameWorks/kaolin - OpenUSD Gaussian Splats / Particle Fields Docs - https://docs.omniverse.nvidia.com/materials-and-rendering/latest/particle-fields.html - NVLabs COMPASS - https://github.com/NVlabs/COMPASS - Flexion Reflect v1.0 - https://flexion.ai/news/flexion-reflect-v1.0 - NVIDIA Omniverse Discord - https://discord.com/invite/nvidiaomniverse Continue the discussion and ask follow-up questions in the NVIDIA Omniverse Discord thread: https://discord.com/channels/827959428476174346/1534066835065016441 ---------------------------------------------------------------------------- 📆 Check out the full calendar for all of our upcoming events → https://nvda.ws/3JqaWnA ---------------------------------------------------------------------------- ⬇️Get Started → https://nvda.ws/4cZAZO1 👀Explore OpenUSD → https://nvda.ws/3CeozBQ 👥Join the Community → https://nvda.ws/3ZMfc6e #NVIDIA #NVIDIAOmniverse #IsaacSim #IsaacLab #NuRec #OpenUSD #3DGaussianSplatting #Robotics #Humanoids #Simulation #SimToReal #PhysicalAI #DigitalTwins #NewtonPhysics #DeveloperTools