Learning depth-sensitive conditional random fields for semantic segmentation of RGB-D images

We present a structured learning approach to semantic annotation of RGB-D images. Our method learns to reason about spatial relations of objects and fuses low-level class predictions to a consistent interpretation of a scene. Our model incorporates color, depth and 3D scene features, on which an energy function is learned to directly optimize object class prediction using the loss-based maximum-margin principle of structural support vector machines. We evaluate our approach on the NYU V2 dataset of indoor scenes, a challenging dataset covering a wide variety of scene layouts and object classes. We hard-code much less information about the scene layout into our model then previous approaches, and instead learn object relations directly from the data. We find that our conditional random field approach improves upon previous work, setting a new state-of-the-art for the dataset.

Authors

• Andreas C. Müller 0001
• Sven Behnke

Keywords

• Semantics
• Three-dimensional displays
• Support vector machines
• Image segmentation
• Training
• Image color analysis
• Labeling
• Semantic Segmentation
• Conditional Random Field
• RGB-D Images
• Support Vector Machine
• Spatial Relationship
• Energy Function
• 3D Scene
• Depth Features
• Structural Vector
• Semantic Annotation
• V2 Dataset
• Hyperparameters
• Random Forest
• Average Accuracy
• Point Cloud
• Class Structure
• Depth Information
• 3D Information
• 3D Point Cloud
• Horizontal Surface
• Standard Support Vector Machine
• Labeling Density
• Conditional Random Field Model
• Normal Orientation
• Pairwise Potential
• Vertical Alignment
• Random Forest Implementation
• Crowded Scenes