Augmented Proximal Policy Optimization for Safe Reinforcement Learning

Safe reinforcement learning considers practical scenarios that maximize the return while satisfying safety constraints. Current algorithms, which suffer from training oscillations or approximation errors, still struggle to update the policy efficiently with precise constraint satisfaction. In this article, we propose Augmented Proximal Policy Optimization (APPO), which augments the Lagrangian function of the primal constrained problem via attaching a quadratic deviation term. The constructed multiplier-penalty function dampens cost oscillation for stable convergence while being equivalent to the primal constrained problem to precisely control safety costs. APPO alternately updates the policy and the Lagrangian multiplier via solving the constructed augmented primal-dual problem, which can be easily implemented by any first-order optimizer. We apply our APPO methods in diverse safety-constrained tasks, setting a new state of the art compared with a comprehensive list of safe RL baselines. Extensive experiments verify the merits of our method in easy implementation, stable convergence, and precise cost control.

Authors

• Juntao Dai Zhejiang University
• Jiaming Ji Institute for AI, Peking University State Key Laboratory of General Artificial Intelligence, Institute for AI, Peking University
• Long Yang Peking University
• Qian Zheng Zhejiang University
• Gang Pan

Keywords

• ML: Reinforcement Learning Algorithms
• PEAI: Safety, Robustness & Trustworthiness
• PRS: Control of High-Dimensional Systems
• PRS: Planning under Uncertainty
• PRS: Planning with Markov Models (MDPs, POMDPs)
• RU: Decision/Utility Theory