CSER: Communication-efficient SGD with Error Reset

The scalability of Distributed Stochastic Gradient Descent (SGD) is today limited by communication bottlenecks. We propose a novel SGD variant: \underline{C}ommunication-efficient \underline{S}GD with \underline{E}rror \underline{R}eset, or \underline{CSER}. The key idea in CSER is first a new technique called ``error reset'' that adapts arbitrary compressors for SGD, producing bifurcated local models with periodic reset of resulting local residual errors. Second we introduce partial synchronization for both the gradients and the models, leveraging advantages from them. We prove the convergence of CSER for smooth non-convex problems. Empirical results show that when combined with highly aggressive compressors, the CSER algorithms accelerate the distributed training by nearly $10\times$ for CIFAR-100, and by $4. 5\times$ for ImageNet.

Authors

• Cong Xie
• Shuai Zheng School of Software Engineering, Xi'an Jiaotong University
• Sanmi Koyejo
• Indranil Gupta
• Mu Li Harbin Institute of Technology, Shenzhen, China
• Haibin Lin

Keywords

No keywords are indexed for this paper.