A safe multi-agent reinforcement learning algorithm using constraint update projection approach

Traditional reinforcement learning has a major limitation, which is that they optimize agent’s policy purely for maximizing rewards. It completely ignore safety considerations. However, in certain critical engineering fields, ensuring safety is of utmost importance, otherwise it can cause incalculable losses. Therefore, this paper proposes a safe Multi-Agent Constrained Update Projection(MACUP) algorithm, which can safely control agents to complete tasks. We solve this problem from the perspective of policy constraint optimization. Firstly, we derive the new bounds of multi-agent policy performance difference based on a tighter general policy performance difference. It contains generalized advantage estimates, and we utilize these bounds as surrogate functions concerning the objective and constraints. Secondly, to address the coordination issue among multiple agents, we employ a multi-agent sequential policy update framework. Finally, we use a projection method to optimize policies, which has low computational complexity and does not require convex approximation of the surrogate function for solving. It can help us reduce errors. Finally, we have validated our algorithm in two different multi-agent safety environments, and the results show that it is able to satisfy safety constraints while achieving higher rewards.

Authors

• Yang Liu College of Computer Science and Software Engineering, Shenzhen University
• Xiang Feng State Key Laboratory of CAD&CG, Zhejiang University University of Utah
• Huiqun Yu

Keywords

• Safe agent control
• Safe reinforcement learning
• Projection
• Multi-agent sequential update