Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations

Humans learn abstract concepts through multisensory synergy, and once formed, such representations can often be recalled from a single modality. Inspired by this principle, we introduce Concerto, a minimalist simulation of human concept learning for spatial cognition, combining 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding. Despite its simplicity, Concerto learns more coherent and informative spatial features, as demonstrated by zero-shot visualizations. It outperforms both standalone SOTA 2D and 3D self-supervised models by 14. 2\% and 4. 8\%, respectively, as well as their feature concatenation, in linear probing for 3D scene perception. With full fine-tuning, Concerto sets new SOTA results across multiple scene understanding benchmarks (e. g. , 80. 7\% mIoU on ScanNet). We further present a variant of Concerto tailored for video-lifted point cloud spatial understanding, and a translator that linearly projects Concerto representations into CLIP’s language space, enabling open-world perception. These results highlight that Concerto emerges spatial representations with superior fine-grained geometric and semantic consistency.

Authors

• Yujia Zhang Shanxi University
• Xiaoyang Wu
• Yixing Lao
• Chengyao Wang
• Zhuotao Tian Harbin Institute of Technology, Shenzhen
• Naiyan Wang TuSimple
• Hengshuang Zhao The University of Hong Kong

Keywords

No keywords are indexed for this paper.