Leveraging Text-Driven Semantic Variation for Robust OOD Segmentation

In autonomous driving and robotics, ensuring road safety and reliable decision-making critically depends on out-of-distribution (OOD) segmentation. While numerous methods have been proposed to detect anomalous objects on the road, leveraging the vision-language space–which provides rich linguistic knowledge–remains an underexplored field. We hypothesize that incorporating these linguistic cues can be especially beneficial in the complex contexts found in real-world autonomous driving scenarios. To this end, we present a novel approach that trains a Text-Driven OOD Segmentation model to learn a semantically diverse set of objects in the vision-language space. Concretely, our approach combines a vision-language model’s encoder with a transformer decoder, employs Distance-Based OOD prompts located at varying semantic distances from in-distribution (ID) classes, and utilizes OOD Semantic Augmentation for OOD representaitons. By aligning visual and textual information, our approach effectively generalizes to unseen objects and provides robust OOD segmentation in diverse driving environments. We conduct extensive experiments on publicly available OOD segmentation datasets such as Fishyscapes, Segment-Me-If-You-Can, and Road Anomaly datasets, demonstrating that our approach achieves state-of-the-art performance across both pixel-level and object-level evaluations. This result underscores the potential of vision-language–based OOD segmentation to bolster the safety and reliability of future autonomous driving systems.

Authors

• Seungheon Song
• Jaekoo Lee

Keywords

• Image segmentation
• Visualization
• Semantics
• Linguistics
• Performance gain
• Transformers
• Road safety
• Safety
• Reliability
• Autonomous vehicles
• Robust Segmentation
• On Out-of-distribution
• Visual Information
• Segmentation Model
• Textual Information
• Semantic Distance
• Transformer Decoder
• False Positive
• Cross-entropy
• Visual Cues
• Pedestrian
• Cross-entropy Loss
• Intersection Over Union
• Confidence Score
• Latent Space
• Semantic Segmentation
• Pixel Level
• WordNet
• Image Encoder
• Text Encoder
• Auxiliary Dataset
• Mean F1
• Candidate Words