Basketball AI: Player Tracking, Team Detection, and Number Recognition with Python

Watch a full basketball possession analyzed frame by frame using a state-of-the-art AI vision pipeline built from scratch. You'll see how we combine powerful open-source models like RF-DETR for robust object detection, SAM2 for stable, pixel-level player tracking, and a specialized SmolVLM2 for reading blurred jersey numbers. Learn the exact process for identifying every player, classifying shot outcomes, and performing homography to map player movement onto a precise top-down court visualization for advanced tactical analysis. This video breaks down months of work into a complete, reusable computer vision system for sports analytics. - 00:00 Project Overview - 02:04 Detect Players and Numbers with RF-DETR - 07:12 Track Players with SAM2 - 11:59 Team Clustering with SigLIP, UMAP and K-means - 16:31 Fine-Tuning SmolVLM2 - 21:56 Map Player Positions to Court Coordinates - 31:51 Detect Shot Event and Classify Result - 35:54 Conclusions Resources: - “How to Detect, Track, and Identify Basketball Players with Computer Vision” Blogpost: https://blog.roboflow.com/identify-basketball-players - Basketball Player Object Detection Dataset: https://universe.roboflow.com/roboflow-jvuqo/basketball-player-detection-3-ycjdo - Basketball Court Keypoint Detection Dataset: https://universe.roboflow.com/roboflow-jvuqo/basketball-court-detection-2 - Basketball Jersey Number OCR Detaset: https://universe.roboflow.com/roboflow-jvuqo/basketball-jersey-numbers-ocr - How to Detect Track and Identify Basketball Players Notebook: https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/basketball-ai-how-to-detect-track-and-identify-basketball-players.ipynb - Make or Miss - Jumpshot Detection: https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/basketball-ai-make-or-miss-jumpshot-detection.ipynb - Fine-tune RF-DETR on Custom Dataset Notebook: https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/how-to-finetune-rf-detr-on-detection-dataset.ipynb - Segment Video with SAM 2 Notebook: https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/how-to-segment-videos-with-sam-2.ipynb - RF-DETR GitHub: https://github.com/roboflow/rf-detr - Supervision GitHub: https://github.com/roboflow/supervision - Sports GitHub: https://github.com/roboflow/sports Stay updated with the projects I'm working on at https://github.com/roboflow and https://github.com/SkalskiP! ⭐