Exploiting Contextual Objects and Relations for 3D Visual Grounding

3D visual grounding, the task of identifying visual objects in 3D scenes based on natural language inputs, plays a critical role in enabling machines to understand and engage with the real-world environment. However, this task is challenging due to the necessity to capture 3D contextual information to distinguish target objects from complex 3D scenes. The absence of annotations for contextual objects and relations further exacerbates the difficulties. In this paper, we propose a novel model, CORE-3DVG, to address these challenges by explicitly learning about contextual objects and relations. Our method accomplishes 3D visual grounding via three sequential modular networks, including a text-guided object detection network, a relation matching network, and a target identification network. During training, we introduce a pseudo-label self-generation strategy and a weakly-supervised method to facilitate the learning of contextual objects and relations, respectively. The proposed techniques allow the networks to focus more effectively on referred objects within 3D scenes by understanding their context better. We validate our model on the challenging Nr3D, Sr3D, and ScanRefer datasets and demonstrate state-of-the-art performance. Our code will be public at https: //github. com/yangli18/CORE-3DVG.

Authors

• Li Yang
• Chunfeng Yuan State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA
• Ziqi Zhang School of Engineering, Westlake University
• Zhongang Qi ARC Lab, Tencent PCG
• Yan Xu
• Wei Liu University of Technology Sydney
• Ying Shan Tencent AI Lab ARC Lab, Tencent PCG
• Bing Li
• Weiping Yang
• Peng Li
• Yan Wang
• Weiming Hu State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences School of Information Science and Technology, ShanghaiTech University

Keywords

No keywords are indexed for this paper.