SMMG Seminar | ModalityNet: The Art of Modalities in Human-Centric Data
Compared to conventional AI systems, embodied intelligence requires a significantly richer and more diverse set of data modalities, including vision, language, motion dynamics, tactile feedback, depth perception, mesh-level object tracking, scene reconstruction, and audio signals. In this talk, we introduce ModalityNet, a large-scale multimodal data benchmark for embodied intelligence, covering diverse modalities such as vision, language, motion, touch, and depth. We propose a “data pyramid” framework that spans from high-precision teleoperation data to internet and synthetic data, characterizing the trade-off between data diversity and generalization. To bridge the gap between these layers, we introduce two intermediate data types: High-Precision Human-Environment Interaction (HiPHI) data and In-The-Wild (ITW) data. Specifically, HiPHI-Motion with Object and Vision (HiPHI-MOV) focuses on whole-body motion and object interaction, HiPHI-OmniModality (HiPHI-OM) emphasizes fine-grained manipulation with full-modality alignment, and ITW provides diverse behaviors in natural environments. Together, these datasets establish a comprehensive data foundation for embodied intelligence. This talk will also cover the design of these datasets and the corresponding model training paradigms.