Stochastically Dominant Preference Optimization: Policy Improvement For All

Reinforcement learning from human feedback (RLHF) optimizes policies based on users’ rankings of output samples rather than using user-provided rewards. These methods typically assume users’ underlying utility functions are homogeneous and their rankings differ only due to noise, ultimately optimizing for the average user. Instead, we seek policies that guarantee improvement for all users with respect to their heterogeneous preferences. We introduce stochastic dominance as a stricter guiding criteria for policy optimization that guarantees improvement under any social welfare function. Our approach, stochastically dominant preference optimization (SDPO), avoids explicit reward function estimation while providing individual performance improvement guarantees for users with diverse preferences.

Authors

• Ali Farajzadeh University of Illinois Chicago
• Syed M. Abbas University of Illinois Chicago
• Aadirupa Saha University of Illinois Chicago
• Brian D. Ziebart University of Illinois Chicago

Keywords

• RLHF
• policy improvement
• stochastic dominance