Maintaining Fairness in Logit-based Knowledge Distillation for Class-Incremental Learning

Logit-based knowledge distillation (KD) is commonly used to mitigate catastrophic forgetting in class-incremental learning (CIL) caused by data distribution shifts. However, the strict match of logit values between student and teacher models conflicts with the cross-entropy (CE) loss objective of learning new classes, leading to significant recency bias (i.e. unfairness). To address this issue, we rethink the overlooked limitations of KD-based methods through empirical analysis. Inspired by our findings, we introduce a plug-and-play pre-process method that normalizes the logits of both the student and teacher across all classes, rather than just the old classes, before distillation. This approach allows the student to focus on both old and new classes, capturing intrinsic inter-class relations from the teacher. By doing so, our method avoids the inherent conflict between KD and CE, maintaining fairness between old and new classes. Additionally, recognizing that overconfident teacher predictions can hinder the transfer of inter-class relations (i.e., dark knowledge), we extend our method to capture intra-class relations among different instances, ensuring fairness within old classes. Our method integrates seamlessly with existing logit-based KD approaches, consistently enhancing their performance across multiple CIL benchmarks without incurring additional training costs.

Authors

• Zijian Gao College of Computer Science and Technology, National University of Defense Technology, Changsha, China. State Key Laboratory of Complex & Critical Software Environment, Changsha, China.
• Shanhao Han College of Computer Science and Technology, National University of Defense Technology, Changsha, China.
• Xingxing Zhang School of Computer Science, Tsinghua University, Beijing, China.
• Kele Xu National University of Defense Technology
• Dulan Zhou College of Computer Science and Technology, National University of Defense Technology, Changsha, China. State Key Laboratory of Complex & Critical Software Environment, Changsha, China.
• Xinjun Mao College of Computer Science and Technology, National University of Defense Technology, Changsha, China. State Key Laboratory of Complex & Critical Software Environment, Changsha, China.
• Yong Dou College of Computer Science and Technology, National University of Defense Technology National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology
• Huaimin Wang National University of Defense Technology State Key Laboratory of Complex & Critical Software Environment Hunan Institute of Advanced Technology, Changsha, China

Keywords

No keywords are indexed for this paper.