D2MoRA: Diversity-Regulated Asymmetric MoE-LoRA Decomposition for Efficient Multi-Task Adaptation

Low-Rank Adaptation (LoRA) has emerged as a powerful parameter-efficient fine-tuning method for adapting large language models to downstream tasks. Recent studies have leveraged Mixture-of-Experts (MoE) mechanism to effectively integrate multiple LoRA modules, facilitating efficient parameter adaptation for multi-task scenarios. It has been shown that fostering knowledge sharing across LoRA experts can greatly enhance parameter adaptation efficiency. However, the existing approach for LoRA expert knowledge sharing still faces two key limitations: constrained functional specialization and induced expert homogenization. To address these issues, we propose a novel diversity-regulated asymmetric MoE-LoRA decomposition framework, which achieves flexible knowledge sharing through asymmetric expert decomposition and guarantees expert diversity with a dual orthogonality regularization. Extensive experiments on eight public benchmarks, spanning both multi-task and single-task settings, demonstrate the superiority of our approach over existing methods.

Authors

• Jianhui Zuo School of Software, Shandong University
• Xuemeng Song Shandong University
• Haokun Wen School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen) School of Data Science, City University of Hong Kong
• Meng Liu School of Computer, National University of Defense Technology
• Yupeng Hu Shandong University
• Jiuru Wang School of Computer Science and Engineering, Linyi University
• Liqiang Nie Harbin Institute of Technology (Shenzhen)

Keywords

No keywords are indexed for this paper.