Weakly supervised semantic segmentation by knowledge graph inference

The weakly supervised semantic segmentation (WSSS) training based on image-level labels in convolutional neural network (CNN) is usually divided into two stages: multi-label classification and semantic segmentation. However, most of the existing work focuses on the improvement of the multi-label classification network stage, and little effort has been done to improve the performance of the downstream segmentation networks. In addition, CNN-based local convolution lacks in modeling extensive dependencies among categories. Therefore, in this paper, we propose a graph reasoning method to improve both the upstream and the downstream stages of the multi-label classification network and the semantic segmentation networks. In the multi-label classification network, we utilize external knowledge combined with a graph convolutional network (GCN) to perform global reasoning on the dependencies of each category. In the segmentation network, the Graph Reasoning Mapping Module (GRM) is proposed to explore the knowledge acquired from text corpora and facilitate contextual reasoning in various categories of image regions. The proposed GRM module is able to enhance the feature representation of local convolutions on the high-level semantics of the segmentation network, and adaptively learn the semantic consistency of each sample. We achieve state-of-the-art WSSS performance on PASCAL VOC 2012 and MS-COCO 2014 datasets with only image-level supervision. Extensive experiments on multi-label classification networks and semantic segmentation networks demonstrate the effectiveness of our proposed graph reasoning method on WSSS. Our code is available at: https: //github. com/JIA-ZHANG666/GRM_layer.

Authors

• Jia Zhang Shanxi University
• Bo Peng Shanghai Ocean University
• Xi Wu
• Jie Hu

Keywords

• Weakly supervised semantic segmentation
• Convolutional neural networks
• Graph convolutional network
• External knowledge
• Graph reasoning