Modular Sensor Fusion for Semantic Segmentation

Sensor fusion is a fundamental process in robotic systems as it extends the perceptual range and increases robustness in real-world operations. Current multi-sensor deep learning based semantic segmentation approaches do not provide robustness to under-performing classes in one modality, or require a specific architecture with access to the full aligned multi-sensor training data. In this work, we analyze statistical fusion approaches for semantic segmentation that overcome these drawbacks while keeping a competitive performance. The studied approaches are modular by construction, allowing to have different training sets per modality and only a much smaller subset is needed to calibrate the statistical models. We evaluate a range of statistical fusion approaches and report their performance against state-of-the-art baselines on both realworld and simulated data. In our experiments, the approach improves performance in IoU over the best single modality segmentation results by up to 5%. We make all implementations and configurations publicly available.

Authors

• Hermann Blum
• Abel Gawel
• Roland Siegwart
• Cesar Cadena 0001

Keywords

• Semantics
• Image segmentation
• Robot sensing systems
• Training
• Fuses
• Computer architecture
• Semantic Segmentation
• Real-world Data
• Intersection Over Union
• Alignment Data
• Statistical Methods
• Convolutional Neural Network
• Confusion Matrix
• Expectation Maximization
• Modularity
• Fusion Method
• Depth Images
• Classification Output
• Inference Time
• Distribution Of Categories
• Fusion Network
• Fully Convolutional Network
• Sufficient Statistics
• Dirichlet Distribution
• Network Of Experts
• Softmax Output
• Modular Method
• Ground-truth Class
• Complementary Strengths
• Qualitative Examples
• Concentration Parameters
• Scene Understanding
• Fusion Techniques
• Individual Experts
• Neural Network
• Input Modalities