Semantic Image Alignment for Vehicle Localization

Accurate and reliable localization is a fundamental requirement for autonomous vehicles to use map information in higher-level tasks such as navigation or planning. In this paper, we present a novel approach to vehicle localization in dense semantic maps, including vectorized high-definition maps or 3D meshes, using semantic segmentation from a monocular camera. We formulate the localization task as a direct image alignment problem on semantic images, which allows our approach to robustly track the vehicle pose in semantically labeled maps by aligning virtual camera views rendered from the map to sequences of semantically segmented camera images. In contrast to existing visual localization approaches, the system does not require additional keypoint features, handcrafted localization landmark extractors or expensive LiDAR sensors. We demonstrate the wide applicability of our method on a diverse set of semantic mesh maps generated from stereo or LiDAR as well as manually annotated HD maps and show that it achieves reliable and accurate localization in real-time.

Authors

• Markus Herb
• Matthias Lemberger
• Marcel M. Schmitt
• Alexander Kurz 0003
• Tobias Weiherer
• Nassir Navab
• Federico Tombari

Keywords

• Location awareness
• Image segmentation
• Visualization
• Laser radar
• Semantics
• Cameras
• Feature extraction
• Fluidic
• Vehicle Position
• Image Alignment
• Semantic Image
• Localization Accuracy
• Autonomous Vehicles
• Semantic Segmentation
• Local Approach
• 3D Mesh
• Semantic Labels
• Semantic Map
• Direct Alignment
• Point Cloud
• Semantic Information
• Particle Filter
• Pose Estimation
• Lidar Data
• Camera Frame
• Visual Simultaneous Localization And Mapping
• Relative Pose
• Visual Odometry
• Place Recognition
• Original Prediction
• Map Elements
• Map Formation
• Camera Pose
• Sensor Modalities
• Road Markings