DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning

In recent years, multi-agent reinforcement learning (MARL) has presented impressive performance in various applications. However, physical limitations, budget restrictions, and many other factors usually impose constraints on a multi-agent system (MAS), which cannot be handled by traditional MARL frameworks. Specifically, this paper focuses on constrained MASes where agents work cooperatively to maximize the expected team-average return under various constraints on expected team-average costs, and develops a constrained cooperative MARL framework, named DeCOM, for such MASes. In particular, DeCOM decomposes the policy of each agent into two modules, which empowers information sharing among agents to achieve better cooperation. In addition, with such modularization, the training algorithm of DeCOM separates the original constrained optimization into an unconstrained optimization on reward and a constraints satisfaction problem on costs. DeCOM then iteratively solves these problems in a computationally efficient manner, which makes DeCOM highly scalable. We also provide theoretical guarantees on the convergence of DeCOM's policy update algorithm. Finally, we conduct extensive experiments to show the effectiveness of DeCOM with various types of costs in both moderate-scale and large-scale (with 500 agents) environments that originate from real-world applications.

Authors

• Zhaoxing Yang Shanghai Jiao Tong University
• Haiming Jin Shanghai Jiao Tong University
• Rong Ding Shanghai Jiao Tong University
• Haoyi You Shanghai Jiao Tong University
• Guiyun Fan Shanghai Jiao Tong University
• Xinbing Wang Shanghai Jiao Tong University
• Chenghu Zhou Shanghai Jiao Tong University

Keywords

• MAS: Multiagent Learning
• ML: Reinforcement Learning Algorithms