Decoder Modulation for Indoor Depth Completion

Depth completion recovers a dense depth map from sensor measurements. Current methods are mostly tailored for very sparse depth measurements from LiDARs in outdoor settings, while for indoor scenes Time-of-Flight (ToF) or structured light sensors are mostly used. These sensors provide semi-dense maps, with dense measurements in some regions and almost empty in others. We propose a new model that takes into account the statistical difference between such regions. Our main contribution is a new decoder modulation branch added to the encoder-decoder architecture. The encoder extracts features from the concatenated RGB image and raw depth. Given the mask of missing values as input, the proposed modulation branch controls the decoding of a dense depth map from these features differently for different regions. This is implemented by modifying the spatial distribution of output signals inside the decoder via Spatially-Adaptive Denormalization (SPADE) blocks. Our second contribution is a novel on-the- y sensor simulation strategy that allows us to train on a semi-dense sensor data when the ground truth depth map is not available. Our model achieves the state of the art results on indoor Matterport3D dataset [1]. Being designed for semi-dense input depth, our model is still competitive with LiDAR-oriented approaches on the KITTI dataset [2]. Our sensor simulation strategy significantly improves prediction quality with no dense ground truth available, as validated on the NYUv2 dataset [3].

Authors

• Dmitry Senushkin
• Mikhail Romanov
• Ilia Belikov
• Nikolay Patakin
• Anton Konushin

Keywords

• Technological innovation
• Laser radar
• Current measurement
• Modulation
• Predictive models
• Robot sensing systems
• Feature extraction
• Depth Completion
• Significantly Improved
• Sensor Data
• Depth Map
• Structured Illumination
• Depth Measurements
• Simulation Scheme
• KITTI Dataset
• Sparse Measurements
• Ground Truth Depth
• Dense Depth
• Loss Function
• Root Mean Square Error
• Random Sampling
• Batch Normalization
• Semantic Segmentation
• Depth Camera
• Depth Data
• Self-driving
• Depth Estimation
• Labeling Density
• Dense Reconstruction
• Bias Term
• Raw Sensor Data