Resource-Aware Federated Self-Supervised Learning with Global Class Representations

Due to the heterogeneous architectures and class skew, the global representation models training in resource-adaptive federated self-supervised learning face with tricky challenges: $\textit{deviated representation abilities}$ and $\textit{inconsistent representation spaces}$. In this work, we are the first to propose a multi-teacher knowledge distillation framework, namely $\textit{FedMKD}$, to learn global representations with whole class knowledge from heterogeneous clients even under extreme class skew. Firstly, the adaptive knowledge integration mechanism is designed to learn better representations from all heterogeneous models with deviated representation abilities. Then the weighted combination of the self-supervised loss and the distillation loss can support the global model to encode all classes from clients into a unified space. Besides, the global knowledge anchored alignment module can make the local representation spaces close to the global spaces, which further improves the representation abilities of local ones. Finally, extensive experiments conducted on two datasets demonstrate the effectiveness of $\textit{FedMKD}$ which outperforms state-of-the-art baselines 4. 78\% under linear evaluation on average.

Authors

• Mingyi Li School of Computer Science and Technology, Shandong University
• Xiao Zhang Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
• Qi Wang
• Tengfei Liu Beijing University of Technology
• Ruofan Wu Ant Group
• Weiqiang Wang Independent Researcher
• Fuzhen Zhuang Institute of Artificial In telligence, Beihang University Zhongguancun Laboratory
• Hui Xiong Hong Kong University of Science and Technology (Guangzhou)
• Dongxiao Yu Shandong University

Keywords

No keywords are indexed for this paper.