RidgeLoRA: Matrix Ridge Enhanced Low-Rank Adaptation of Large Language Models

As one of the state-of-the-art parameter-efficient fine-tuning~(PEFT) methods, Low-Rank Adaptation (LoRA) enables model optimization with reduced computational cost through trainable low-rank matrix. However, the low-rank nature makes it prone to produce a decrease in the representation ability, leading to suboptimal performance. In order to break this limitation, we propose RidgeLoRA, a lightweight architecture like LoRA that incorporates novel architecture and matrix ridge enhanced full-rank approximation, to match the performance of full-rank training, while eliminating the need for high memory and a large number of parameters to restore the rank of matrices. We provide a rigorous mathematical derivation to prove that RidgeLoRA has a better upper bound on the representations than vanilla LoRA. Furthermore, extensive experiments across multiple domains demonstrate that RidgeLoRA achieves better performance than other LoRA variants, and can even match or surpass full-rank training.

Authors

• Junda Zhu
• Jun Ai
• Yujun Li School of Artifcial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University, Xi'an 710072, P.R. China
• Yichun Yin Huawei Noah’s Ark Lab, Shenzhen, Guangdong, China
• Yasheng Wang Huawei Noah’s Ark Lab
• Lifeng Shang Huawei Noah’s Ark Lab, Shenzhen, Guangdong, China
• Qun Liu Huawei Noah's Ark Lab

Keywords

No keywords are indexed for this paper.