Author name cluster

Donglei Wu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

2 papers

1 author row

AAAI Conference 2026 Conference Paper

RCMoE: A Communication-Efficient Random Compression Framework for Resource-Constrained Mixture-of-Experts Training

Donglei Wu
Xiao Cai
Jinglei Tan
Jinda Jia
Guangming Tan
Dingwen Tao
Wen Xia
Zhihong Tian

Mixture-of-Experts (MoE) architecture with experts parallelism scales LLMs efficiently by activating only a subset of experts per input, avoiding proportional training costs. However, the intensive and heterogeneous communication substantially hinders the efficiency and scalability of MoE training in the resource-constrained scenario. Existing communication compression techniques fall short in MoE training due to: (i) Intensive training amplifies compression overhead, compromising training efficiency; (ii) Accumulated compression errors propagate through the network, degrading training quality. In this paper, we propose RCMoE, a communication-efficient Random Compression framework for MoE training with two core modules: (1) Local-Stochastic Quantization compresses the all-to-all communication by stochastically quantizing each row of the expert's intermediate computing results in parallel, effectively improving the compression efficiency and reducing compression error; (2) Probabilistic Thresholding Sparsification compresses the all-reduce communication by probabilistically sampling large gradients at high probability, thereby reducing the computational complexity and maintaining the convergence efficiency. Experiments on four typical MoE training tasks prove that RCMoE achieves higher 5.9x-8.1x total communication compression ratios and 1.3x-10.1x training speedup compared with the state-of-the-art compression techniques while maintaining the MoE training accuracy.

PDF Details DOI

AAAI Conference 2022 Conference Paper

SmartIdx: Reducing Communication Cost in Federated Learning by Exploiting the CNNs Structures

Donglei Wu
Xiangyu Zou
Shuyu Zhang
Haoyu Jin
Wen Xia
Binxing Fang

Top-k sparsification method is popular and powerful for reducing the communication cost in Federated Learning (FL). However, according to our experimental observation, it spends most of the total communication cost on the index of the selected parameters (i. e. , their position information), which is inefficient for FL training. To solve this problem, we propose a FL compression algorithm for convolutional neural networks (CNNs), called SmartIdx, by extending the traditional Top-k largest variation selection strategy into the convolution-kernel-based selection, to reduce the proportion of the index in the overall communication cost and thus achieve a high compression ratio. The basic idea of SmartIdx is to improve the 1: 1 proportion relationship between the value and index of the parameters to n: 1, by regarding the convolution kernel as the basic selecting unit in parameter selection, which can potentially deliver more information to the parameter server under the limited network traffic. To this end, a set of rules are designed for judging which kernel should be selected and the corresponding packaging strategies are also proposed for further improving the compression ratio. Experiments on mainstream CNNs and datasets show that our proposed SmartIdx performs 2. 5×−69. 2× higher compression ratio than the state-of-the-art FL compression algorithms without degrading training performance.

PDF Details