Multistream Network for LiDAR and Camera-based 3D Object Detection in Outdoor Scenes

Fusion of LiDAR and RGB data has the potential to enhance outdoor 3D object detection accuracy. To address real-world challenges in outdoor 3D object detection, fusion of LiDAR and RGB input has started gaining traction. However, effective integration of these modalities for precise object detection tasks still remains a largely open problem. To address that, we propose a MultiStream Detection (MuStD) network, which meticulously extracts task-relevant information from both data modalities. The network follows a three-stream structure. Its LiDAR-PillarNet stream extracts sparse 2D pillar features from the LiDAR input while the LiDAR-Height Compression stream computes Bird’s-Eye View features. An additional 3D Multimodal stream combines RGB and LiDAR features using UV mapping and polar coordinate indexing. Eventually, the features containing comprehensive spatial, textural, and geometric information are carefully fused and fed to a detection head for 3D object detection. We evaluate our method on the challenging KITTI Object Detection Benchmark, with results available on the official evaluation server. 1. Our approach achieves strong performance, with an average precision (AP) of 85. 39% in 3D detection, 91. 34% in Bird’s Eye View (BEV) detection, and 96. 39% in 2D detection. These results match or surpass existing state-of-the-art methods. In the difficult "Hard" category, our method attains 80. 78% AP in 3D detection and 94. 04% AP in 2D detection, highlighting its robustness in challenging scenarios. Furthermore, our method runs at 67 ms, demonstrating efficiency and real-time capability. Our code will be released through the MuStD GitHub repository at https://github.com/IbrahimUWA/MuStD.

Authors

• Muhammad Ibrahim 0001
• Naveed Akhtar
• Haitian Wang 0002
• Saeed Anwar
• Ajmal Mian

Keywords

• Three-dimensional displays
• Laser radar
• Accuracy
• Urban areas
• Object detection
• Streaming media
• Benchmark testing
• Feature extraction
• Robustness
• Indexing
• 3D Object Detection
• Spatial Information
• Detection Task
• Average Precision
• Geometric Information
• Bird’s Eye
• Lidar Data
• 3D Detection
• Accurate Object Detection
• RGB Data
• Feature Maps
• Point Cloud
• Geometric Features
• RGB Images
• Feature Fusion
• 3D Point
• Precision-recall Curve
• 3D Features
• Mean Average Precision
• 2D Feature
• Parallel Streams
• LiDAR Point Clouds
• Object Distance
• Polar Transformation
• LiDAR Point
• Geometric Details
• Polar Space
• KITTI Dataset
• 3D Convolution