ReFT: Representation Finetuning for Language Models

Parameter-efficient finetuning (PEFT) methods seek to adapt large neural models via updates to a small number of weights. However, much prior interpretability work has shown that representations encode rich semantic information, suggesting that editing representations might be a more powerful alternative. We pursue this hypothesis by developing a family of Representation Finetuning (ReFT) methods. ReFT methods operate on a frozen base model and learn task-specific interventions on hidden representations. We define a strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT), and we identify an ablation of this method that trades some performance for increased efficiency. Both are drop-in replacements for existing PEFTs and learn interventions that are 15x--65x more parameter-efficient than LoRA. We showcase LoReFT on eight commonsense reasoning tasks, four arithmetic reasoning tasks, instruction-tuning, and GLUE. In all these evaluations, our ReFTs deliver the best balance of efficiency and performance, and almost always outperform state-of-the-art PEFTs. Upon publication, we will publicly release our generic ReFT training library.

Authors

• Zhengxuan Wu
• Aryaman Arora
• Zheng Wang National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, China Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China
• Atticus Geiger
• Dan Jurafsky
• Christopher D. Manning
• Christopher Potts

Keywords

No keywords are indexed for this paper.