Decoupling Continual Semantic Segmentation

Continual Semantic Segmentation (CSS) requires learning new classes without forgetting previously acquired knowledge, addressing the fundamental challenge of catastrophic forgetting in dense prediction tasks. However, existing CSS methods typically employ single-stage encoder-decoder architectures where segmentation masks and class labels are tightly coupled, leading to interference between old and new class learning and suboptimal retention-plasticity balance. We introduce DecoupleCSS, a novel two-stage framework for CSS. By decoupling class-aware detection from class-agnostic segmentation, DecoupleCSS enables more effective continual learning, preserving past knowledge while learning new classes. The first stage leverages pre-trained text and image encoders, adapted using LoRA, to encode class-specific information and generate location-aware prompts. In the second stage, the Segment Anything Model (SAM) is employed to produce precise segmentation masks, ensuring that segmentation knowledge is shared across both new and previous classes. This approach improves the balance between retention and adaptability in CSS, achieving state-of-the-art performance across a variety of challenging tasks.

Authors

• Yifu Guo
• Yuquan Lu School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
• Wentao Zhang Peking University
• Zishan Xu Tsinghua University
• Dexia Chen Sun Yat-sen University, Guangzhou, China Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China
• Siyu Zhang Business college, Southwest university, Chongqing, China
• Yizhe Zhang
• Ruixuan Wang School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China Key Laboratory of Machine Intelligence and Advanced Computing, MOE, Guangzhou, China Peng Cheng Laboratory, Shenzhen, China

Keywords

No keywords are indexed for this paper.