Depth Estimation Matters Most: Improving Per-Object Depth Estimation for Monocular 3D Detection and Tracking

Monocular image-based 3D perception has become an active research area in recent years owing to its applications in autonomous driving. Approaches to monocular 3D perception including detection and tracking, however, often yield inferior performance when compared to LiDAR-based techniques. Through systematic analysis, we identified that per-object depth estimation accuracy is a major factor bounding the performance. Motivated by this observation, we propose a multi-level fusion method that combines different representations (RGB and pseudo-LiDAR) and temporal information across multiple frames for objects (tracklets) to enhance per-object depth estimation. Our proposed fusion method achieves the state-of-the-art performance of per-object depth estimation on the Waymo Open Dataset, the KITTI detection dataset, and the KITTI MOT dataset. We further demonstrate that by simply replacing estimated depth with fusion-enhanced depth, we can achieve significant improvements in monocular 3D perception tasks, including detection and tracking.

Authors

• Longlong Jing
• Ruichi Yu
• Henrik Kretzschmar
• Kang Li
• Charles R. Qi
• Hang Zhao 0021
• Alper Ayvaci
• Xu Chen
• Dillon Cower
• Yingwei Li 0002
• Yurong You
• Han Deng
• Congcong Li
• Dragomir Anguelov

Keywords

• Training
• Three-dimensional displays
• Systematics
• Automation
• Fuses
• Estimation
• Task analysis
• Depth Estimation
• 3D Tracking
• 3D Detection
• Monocular 3D Detection
• Fusion Method
• Multiple Frames
• Area In Recent Years
• 3D Perception
• Coordinate System
• Significantly Improved
• Image Features
• Average Error
• Detection Performance
• Object Detection
• Detection Model
• Bounding Box
• RGB Images
• Depth Map
• Optical Flow
• Self-driving
• Temporal Consistency
• 3D Object Detection
• RGB Features
• Dense Depth
• Camera Coordinate
• Monocular Images
• Perfect Association
• Feature Encoder
• 3D Space