Volumetric Semantically Consistent 3D Panoptic Mapping

We introduce an online 2D-to-3D semantic instance mapping algorithm aimed at generating comprehensive, accurate, and efficient semantic 3D maps suitable for autonomous agents in unstructured environments. The proposed approach is based on a Voxel-TSDF representation used in recent algorithms. It introduces novel ways of integrating semantic prediction confidence during mapping, producing semantic and instance-consistent 3D regions. Further improvements are achieved by graph optimization-based semantic labeling and instance refinement. The proposed method achieves accuracy superior to the state of the art on public large-scale datasets, improving on a number of widely used metrics. We also highlight a downfall in the evaluation of recent studies: using the ground truth trajectory as input instead of a SLAM-estimated one substantially affects the accuracy, creating a large gap between the reported results and the actual performance on real-world data. The code is available: https://github.com/y9miao/ConsistentPanopticSLAM.

Authors

• Yang Miao
• Iro Armeni
• Marc Pollefeys
• Daniel Barath

Keywords

• Measurement
• Accuracy
• Three-dimensional displays
• Semantics
• Robot vision systems
• Prediction algorithms
• Real-time systems
• Trajectory
• Labeling
• Intelligent robots
• Semantic Consistency
• Autonomous Agents
• Semantic Labels
• Semantic Map
• Ground Truth Trajectory
• Energy Function
• Intersection Over Union
• Point Cloud
• Confidence Score
• Semantic Segmentation
• 3D Data
• Depth Images
• 3D Mesh
• Sequence Of Frames
• Mesh Generation
• Instance Segmentation
• 3D Segmentation
• Camera Pose
• Global Coordinate System
• Ground Truth Pose
• Instance Labels
• Public Code
• Si Surface
• Graph Optimization