Regulating Intermediate 3D Features for Vision-Centric Autonomous Driving

Multi-camera perception tasks have gained significant attention in the field of autonomous driving. However, existing frameworks based on Lift-Splat-Shoot (LSS) in the multi-camera setting cannot produce suitable dense 3D features due to the projection nature and uncontrollable densification process. To resolve this problem, we propose to regulate intermediate dense 3D features with the help of volume rendering. Specifically, we employ volume rendering to process the dense 3D features to obtain corresponding 2D features (e.g., depth maps, semantic maps), which are supervised by associated labels in the training. This manner regulates the generation of dense 3D features on the feature level, providing appropriate dense and unified features for multiple perception tasks. Therefore, our approach is termed Vampire, stands for ``Volume rendering As Multi-camera Perception Intermediate feature REgulator''. Experimental results on the Occ3D and nuScenes datasets demonstrate that Vampire facilitates fine-grained and appropriate extraction of dense 3D features, and is competitive with existing SOTA methods across diverse downstream perception tasks like 3D occupancy prediction, LiDAR segmentation and 3D objection detection, while utilizing moderate GPU resources. We provide a video demonstration in the supplementary materials and Codes are available at github.com/cskkxjk/Vampire.

Authors

• Junkai Xu State Key Lab of CAD&CG, Zhejiang University Fabu Inc.
• Liang Peng Fabu Inc.
• Haoran Cheng University of Science and Technology of China
• Linxuan Xia State Key Lab of CAD&CG, Zhejiang University Fabu Inc.
• Qi Zhou School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies
• Dan Deng Fabu Inc.
• Wei Qian
• Wenxiao Wang
• Deng Cai Bytedance Seed, China

Keywords

• CV: Applications
• CV: Object Detection & Categorization
• CV: Representation Learning for Vision
• CV: Scene Analysis & Understanding
• CV: Segmentation
• CV: Vision for Robotics & Autonomous Driving