How to Curate Physical AI Data in FiftyOne: MCAP, Episodes, and Quality Scores | Voxel51

Physical AI's data problem is curation, not collection. Harpreet Sahota, Hacker-in-Residence at Voxel51, goes from a raw MCAP recording to a scored, triaged corpus of robot episodes in FiftyOne. The unit of data in physical AI is the episode, not the frame, and MCAP is the file that holds it. Harpreet covers how MCAP is structured (schemas, channels, messages, chunks), why it became the ROS 2 default since Iron, and where FiftyOne sits as the dataset-first tool next to debugging-first and logging-first tools. Then a dataset tour across a boat (CANOE), wet-road driving (SemanticSpray++), a forest rover (FoMo), and a 31-channel drone flight (TreeScope) with cameras, LiDAR, GPS, IMU, odometry, logs, and transforms on one shared clock, including channels FiftyOne has no decoder for, shown raw in the Message tile. The second half is quality. Motion smoothness (SPARC, LDLJ, jerk RMS, PSD ratio), sensor health (dropout, desync, clock drift, rate stability, clipping), and outlier scores (Isolation Forest, kNN), what each measures, where each falls short, and the one rule: a score decides what a human looks at first. It never decides what gets deleted. A smooth episode can still be a failed demonstration, LDLJ is only comparable across equal-length windows, and outlier scores are dataset-relative, so an exceptionally clean episode reads as an outlier too. FiftyOne 1.19 opens MCAP natively. The scoring plugin is open source. ⏱️ Chapters 0:00 Physical AI's data problem is curation, not collection 1:26 About Harpreet and the five-part agenda 3:06 Part 1: The unit of data is the episode, not the frame 6:55 Why single-frame quality checks miss episode-level failures 9:18 Part 2: Inside the MCAP file format 12:27 Schemas, channels, messages, and two timestamps 14:34 Chunks, the index, and the time axis 16:31 The ROS 2 default since Iron, and serialization agnostic 19:20 Recording formats vs. training formats 21:41 Where FiftyOne sits: dataset first 24:29 Importing an MCAP into FiftyOne in a few lines 27:26 Dataset tour: CANOE, SemanticSpray++, FoMo 32:12 Part 3: Visualizing 31 channels of TreeScope on one clock 36:45 Part 4: Data quality metrics 37:51 The rule: scores rank, humans decide 39:41 Why a smooth episode can still be a failure 41:48 Motion smoothness starts with a speed profile 43:45 SPARC, LDLJ, jerk RMS, and PSD 47:39 Computing motion smoothness with the plugin 49:12 Q&A: scoring without labels 50:07 Sensor health: dropout, desync, clock drift, rate stability, clipping 53:33 Outlier scores: Isolation Forest and kNN 55:31 Worst-first ranking, then tag, don't delete 57:08 The open source demo_quality_scorer plugin 58:46 Part 5: Five steps to put it into practice 1:01:08 Five design properties for a curation loop that works 1:05:01 Resources: Hugging Face org, plugin, blogs, workshop repo 1:08:12 Book a workshop and connect with Harpreet 1:09:33 Closing 📚 Workshop resources Workshop repo: https://github.com/harpreetsahota204/multimodal_data_in_fiftyone_workshop demo_quality_scorer plugin: https://github.com/harpreetsahota204/demo_quality_scorer Import MCAP into FiftyOne with agent skills: https://voxel51.com/blog/mcap-import-fiftyone-agent-skills MCAP authoring errors: https://voxel51.com/blog/mcap-authoring-errors 10 MCAP datasets for physical AI: https://voxel51.com/blog/mcap-datasets-multimodal-robotics-fiftyone Triaging robot episode quality: https://voxel51.com/blog/robot-episode-quality-triage Motion smoothness vs. task success: https://voxel51.com/blog/motion-smoothness-task-success-robolab-egox HF Space, CANOE: https://huggingface.co/spaces/harpreetsahota/canoe-multimodal HF Space, SemanticSpray++: https://huggingface.co/spaces/harpreetsahota/semanticspray-plusplus HF Space, FoMo: https://huggingface.co/spaces/harpreetsahota/fomo-multimodal-sample 🔗 Learn more Multimodal data in FiftyOne: https://docs.voxel51.com/user_guide/multimodal.html What is MCAP: https://voxel51.com/glossary/mcap What is an episode: https://voxel51.com/glossary/episode Data triage: https://voxel51.com/glossary/data-triage FiftyOne open source: https://voxel51.com/fiftyone Voxel51 for multimodal data at scale: https://voxel51.com/multimodal-data Book a physical AI workshop: https://voxel51.com/workshops Get started: pip install fiftyone 💬 Join the community: https://community.voxel51.com FiftyOne is the open source multimodal data platform for physical AI. Voxel51 is the enterprise platform built on it, adding annotation workflows, auto labeling, GPU-backed models, and team collaboration. Together they help ML teams curate, annotate, and evaluate data for visual AI and physical AI models, across images, video, 3D, and multimodal sensor data. #FiftyOne #MCAP #PhysicalAI