Multiagent Q-learning with Sub-Team Coordination

In many real-world cooperative multiagent reinforcement learning (MARL) tasks, teams of agents can rehearse together before deployment, but then communication constraints may force individual agents to execute independently when deployed. Centralized training and decentralized execution (CTDE) is increasingly popular in recent years, focusing mainly on this setting. In the value-based MARL branch, credit assignment mechanism is typically used to factorize the team reward into each individual’s reward — individual-global-max (IGM) is a condition on the factorization ensuring that agents’ action choices coincide with team’s optimal joint action. However, current architectures fail to consider local coordination within sub-teams that should be exploited for more effective factorization, leading to faster learning. We propose a novel value factorization framework, called multiagent Q-learning with sub-team coordination (QSCAN), to flexibly represent sub-team coordination while honoring the IGM condition. QSCAN encompasses the full spectrum of sub-team coordination according to sub-team size, ranging from the monotonic value function class to the entire IGM function class, with familiar methods such as QMIX and QPLEX located at the respective extremes of the spectrum. Experimental results show that QSCAN’s performance dominates state-of-the-art methods in matrix games, predator-prey tasks, the Switch challenge in MA-Gym. Additionally, QSCAN achieves comparable performances to those methods in a selection of StarCraft II micro-management tasks.

Authors

• Wenhan Huang
• Kai Li C 2 DL, Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Scien
• Kun Shao Huawei
• Tianze Zhou Beijing Institute of Technology
• Matthew Taylor University of Alberta Alberta Machine Intelligence Institute (Amii)
• Jun Luo Ant Group
• Dongge Wang EPFL
• Hangyu Mao Kuaishou Technology
• Jianye Hao Tianjin University
• Jun Wang University College London
• Xiaotie Deng CFCS, School of Computer Science & CMAR, Institute for AI, Peking University

Keywords

No keywords are indexed for this paper.