Deconfound Semantic Shift and Incompleteness in Incremental Few-shot Semantic Segmentation

Incremental few-shot semantic segmentation (IFSS) expands segmentation capacity of the trained model to segment new-class images with few samples. However, semantic meanings may shift from background to object class or vice versa during incremental learning. Moreover, new-class samples often lack representative attribute features when the new class greatly differs from the pre-learned old class. In this paper, we propose a causal framework to discuss the cause of semantic shift and incompleteness in IFSS, and we deconfound the revealed causal effects from two aspects. First, we propose a Causal Intervention Module (CIM) to resist semantic shift. CIM progressively and adaptively updates prototypes of old class, and removes the confounder in an intervention manner. Second, a Prototype Refinement Module (PRM) is proposed to complete the missing semantics. In PRM, knowledge gained from the episode learning scheme assists in fusing features of new-class and old-class prototypes. Experiments on both PASCAL-VOC 2012 and ADE20k benchmarks demonstrate the outstanding performance of our method.

Authors

• Yirui Wu Key Laboratory of Water Big Data Technology of Ministry of Water Resources, Hohai University, Nanjing, China Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, Shenzhen, China
• Yuhang Xia Key Laboratory of Water Big Data Technology of Ministry of Water Resources, Hohai University, Nanjing, China
• Hao Li Shanghai Artificial Intelligence Laboratory & Northwestern Polytechnical University
• Lixin Yuan Key Laboratory of Water Big Data Technology of Ministry of Water Resources, Hohai University, Nanjing, China
• Junyang Chen College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China
• Jun Liu Carnegie Mellon University
• Tong Lu National Key Lab for Novel Software Technology, Nanjing University, Nanjing, China
• Shaohua Wan Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, Shenzhen, China

Keywords

No keywords are indexed for this paper.