Why Go Full? Elevating Federated Learning Through Partial Network Updates

Federated learning is a distributed machine learning paradigm designed to protect user data privacy, which has been successfully implemented across various scenarios. In traditional federated learning, the entire parameter set of local models is updated and averaged in each training round. Although this full network update method maximizes knowledge acquisition and sharing for each model layer, it prevents the layers of the global model from cooperating effectively to complete the tasks of each client, a challenge we refer to as layer mismatch. This mismatch problem recurs after every parameter averaging, consequently slowing down model convergence and degrading overall performance. To address the layer mismatch issue, we introduce the FedPart method, which restricts model updates to either a single layer or a few layers during each communication round. Furthermore, to maintain the efficiency of knowledge acquisition and sharing, we develop several strategies to select trainable layers in each round, including sequential updating and multi-round cycle training. Through both theoretical analysis and experiments, our findings demonstrate that the FedPart method significantly surpasses conventional full network update strategies in terms of convergence speed and accuracy, while also reducing communication and computational overheads.

Authors

• Haolin Wang Graduate School of Health Sciences, Hokkaido University, Sapporo, Japan
• Xuefeng Liu University of Chicago
• Jianwei Niu State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University, Beijing, China Zhongguancun Laboratory, Beijing, China Zhengzhou University Research Institute of Industrial Technology, Zhengzhou University, Zhengzhou, China
• Wenkai Guo
• Shaojie Tang Department of Management Science and Systems, University at Buffalo, Buffalo, New York, United States

Keywords

No keywords are indexed for this paper.