Leveraging robust optimization for llm alignment under distribution shifts

Preference alignment methods are increasingly critical for steering large language models (LLMs) to generate outputs consistent with human values. While recent approaches often rely on synthetic data generated by LLMs for scalability and cost-efficiency reasons, this reliance can introduce distributional shifts that undermine the nuanced representation of human preferences needed for desirable outputs. In this paper, we propose a novel distribution-aware optimization framework that improves preference alignment despite such shifts. Our approach first leverages well-learned classifiers to assign a calibration value to each training sample, quantifying its alignment with the target human-preferred distribution. These values are then incorporated into a robust optimization objective that minimizes the worst-case loss over regions of the data space most relevant to human preferences. By explicitly focusing optimization on the target distribution, our approach mitigates the impact of distributional mismatch and improves the generation of responses that better reflect intended values.

Authors

• Mingye Zhu University of Science and Technology of China, Hefei, Anhui, China Ping An Technology, Shenzhen, Guangdong, China
• Yi Liu City University of Hong Kong
• Zheren Fu
• Yongdong Zhang University of Science and Technology of China
• Zhendong Mao University of Science and Technology of China

Keywords

No keywords are indexed for this paper.