3D Gaussian Splatting for Autonomous Vehicle Simulation
Reconstructing real world sensors’ data into simulation is key in AV developers pipelines, reducing simulations’ domain gap and increasing the scale and coverages of simulated scenarios. In this livestream, NVIDIA will show how open NVIDIA Omniverse NuRec tools solve this for autonomous vehicles, reconstructing real-world driving data into photorealistic, simulation-ready 3D Gaussian Splat scenes that support closed-loop simulation and scenario augmentation. Join us to learn how: - Omniverse NuRec reconstructs real driving clips into photoreal scenes and renders cameras and lidar through one renderer - NuRec Skills drive the full reconstruction workflow with an agent - Asset Harvester and Harmonizer insert vehicles with matched lighting - InstantNuRec produces a 3D Gaussian splat in a single feed-forward pass on one GPU - HiGS through gsplat achieves fast rendering ---------------------------------------------------------------------------- Chapters: 0:00 Welcome and 3D Gaussian Splatting for AV Simulation 2:08 Neural Reconstruction for AV Simulation 4:12 NuRec Fixer, Asset Harvester, and Harmonizer 7:15 NuRec Libraries, Containers, and Agent Skills 8:02 Q&A: Real-Time Gaussian Splatting Digital Twins 10:17 Asset Harvester and Harmonizer Demo 13:26 Physical AI Dataset and NCore Prep 19:25 Asset Harvester 3D Asset Extraction 21:50 Scene Editing: Swap, Remove, and Change Trajectories 26:36 Q&A: LiDAR, Image-Only Input, and Indoor Scenes 29:20 Instant NuRec Feed-Forward Reconstruction 32:29 Instant NuRec Live GUI Demo 37:09 Instant NuRec GitHub Workflow 40:48 Q&A: Actor Behavior and Sim-Ready Assets 43:07 HiGS GPU Rendering for 3D Gaussian Splatting 52:30 HiGS 4K Performance, Quality, and Roadmap 58:43 Q&A: LiDAR Fusion and 3D Model Conversion ---------------------------------------------------------------------------- List of Resources: - NVIDIA Omniverse NuRec documentation (NuRec overview, setup, reconstruction, rendering, and refinement docs) - https://docs.nvidia.com/nurec/ - NuRec for AV developers (AV simulation quickstart) - https://docs.nvidia.com/nurec/av/index.html - NVIDIA Fixer model repository (artifact correction follow-up) - https://github.com/nv-tlabs/Fixer - NVIDIA Asset Harvester GitHub repository (3D Gaussian asset extraction) - https://github.com/NVIDIA/asset-harvester - NVIDIA Harmonizer GitHub repository (Diffusion Harmonizer code) - https://github.com/NVIDIA/harmonizer - NVIDIA Instant NuRec GitHub repository (feed-forward NuRec implementation) - https://github.com/NVIDIA/instant-nurec - HiGS research project (hierarchical 3DGS rendering details) - https://research.nvidia.com/labs/sil/projects/higs/ - NVIDIA Omniverse Discord (community follow-up) - https://discord.com/invite/nvidiaomniverse - NuRec hardware setup and requirements (GPU, driver, OS, and memory requirements) - https://docs.nvidia.com/nurec/basics/hardware.html - NuRec for robotics developers (robotics reconstruction workflow) - https://docs.nvidia.com/nurec/robotics/index.html - NVIDIA Physical AI collection on Hugging Face (datasets and model collection) - https://huggingface.co/collections/nvidia/physical-ai Continue the discussion and ask follow-up questions in the NVIDIA Omniverse Discord thread: ---------------------------------------------------------------------------- 📆 Check out the full calendar for all of our upcoming events → https://nvda.ws/3JqaWnA ---------------------------------------------------------------------------- ⬇️Get Started → https://nvda.ws/4cZAZO1 👀Explore OpenUSD → https://nvda.ws/3CeozBQ 👥Join the Community → https://nvda.ws/3ZMfc6e #NVIDIA #NVIDIAOmniverse #NuRec #3DGaussianSplatting #GaussianSplatting #AutonomousVehicles #AVSimulation #PhysicalAI #DigitalTwins #Simulation #IsaacSim #ComputerVision #CUDA