Offline Safe Policy Optimization From Heterogeneous Feedback

Offline Preference-based Reinforcement Learning (PbRL) enables reward and policy learning from human preferences without extensive reward engineering and direct interaction with human annotators. However, ensuring safety remains a critical challenge, especiallyinlong-horizoncontinuouscontroltasks. Previousworks on safe RL from human feedback (RLHF) typically learn reward and cost models before applying constrained RL, which can lead to compounding errors and suboptimal performance. To address these challenges, (a) instead of indirectly learning policies (from rewards and costs), we introduce a framework that learns a policy directly based on pairwise preferences regarding the agent’s behavior in terms of rewards, as well as binary labels indicating the safety of trajectory segments; (b) we propose PreSa (Preference and Safety Alignment), a method that combines preference learning module with safety alignment into a single objective, optimized via a Lagrangian paradigm that directly learns reward-maximizing safe policy without explicitly learning reward and cost models, avoiding the need for constrained RL. Experiments on continuous control tasks with both synthetic and real human feedback demonstrate that PreSa consistently learns safe policies with high rewards, outperforming state-of-the-art baselines, and offline safe RL approaches with ground-truth reward and cost. A full version of this paper is available at: https: //arxiv. org/abs/2512. 20173.

Authors

• Ze Gong Shenzhen Institute of Advanced Technology (SIAT), CAS
• Pradeep Varakantham Singapore Management University
• Akshat Kumar Singapore Management University

Keywords

• Offline Safe RL
• Preference-based RL