Exploring Structural Degradation in Dense Representations for Self-supervised Learning

In this work, we observe a counterintuitive phenomenon in self-supervised learning (SSL): longer training may impair the performance of dense prediction tasks (e. g. , semantic segmentation). We refer to this phenomenon as Self-supervised Dense Degradation (SDD) and demonstrate its consistent presence across sixteen state-of-the-art SSL methods with various losses, architectures, and datasets. When the model performs suboptimally on dense tasks at the end of training, measuring the performance during training becomes essential. However, evaluating dense performance effectively without annotations remains an open challenge. To tackle this issue, we introduce a Dense representation Structure Estimator (DSE), composed of a class-relevance measure and an effective dimensionality measure. The proposed DSE is both theoretically grounded and empirically validated to be closely correlated with the downstream performance. Based on this metric, we introduce a straightforward yet effective model selection strategy and a DSE-based regularization method. Experiments on sixteen SSL methods across four benchmarks confirm that model selection improves mIoU by $3. 0\\%$ on average with negligible computational cost. Additionally, DSE regularization consistently mitigates the effects of dense degradation. Code is available at \url{https: //github. com/EldercatSAM/SSL-Degradation}.

Authors

• Siran Dai
• Qianqian Xu Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences
• Peisong Wen School of Computer Science and Tech., University of Chinese Academy of Sciences Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences
• Yang Liu College of Computer Science and Software Engineering, Shenzhen University
• Qingming Huang School of Computer Science and Technology, University of Chinese Academy of Sciences Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences Key Laboratory of Big Data Mining and Knowledge Management, University of Chinese Academy of Sciences

Keywords

No keywords are indexed for this paper.