Lecture 12 - V-JEPA 2 and V-JEPA 2-AC | Zero-Shot Robot Planning

In this lecture, we continue from V-JEPA to V-JEPA 2 (Paper: https://arxiv.org/abs/2506.09985): scaling JEPA across data (22M videos), model size (1B), training length and resolution. Then V-JEPA 2-AC: freeze the encoder, condition the predictor on actions using 62 hours of robot data (teacher-forcing + rollout losses), and plan pick-and-place zero-shot with CEM in latent space. We also see why DINO-WM is "a JEPA, but not an end-to-end one", and how V-JEPA 2-AC realises LeCun's cognitive architecture. Project: Plan a real SO-101 robot arm in V-JEPA 2's feature space — freeze Meta's pretrained V-JEPA 2 encoder, train a small action-conditioned predictor on robot data, and use CEM to reach a goal frame (V-JEPA 2-AC at small scale) https://github.com/RajatDandekar/build-a-world-model-from-scratch/tree/main/lecture-12-vjepa2 All slides, code & Colab notebooks for this series: https://github.com/RajatDandekar/build-a-world-model-from-scratch If you're enjoying the series, a ⭐ on the repo helps more learners find it.