Multiagent Q-learning with Sub-Team Coordination

For cooperative mutliagent reinforcement learning tasks, we propose a novel value factorization framework in the popular centralized training with decentralized execution paradigm, called multiagent Q-learning with sub-team coordination (QSCAN). This framework could flexibly exploit local coordination within sub-teams for effective factorization while honoring the individual-globalmax (IGM) condition. QSCAN encompasses the full spectrum of sub-team coordination according to sub-team size, ranging from the monotonic value function class to the entire IGM function class, with familiar methods such as QMIX and QPLEX located at the respective extremes of the spectrum. Empirical results show that QSCAN’s performance dominates state-of-the-art methods in predator-prey tasks and the Switch challenge in MA-Gym.

Authors

• Wenhan Huang
• Kai Li
• Kun Shao Huawei
• Tianze Zhou Beijing Institute of Technology
• Jun Luo Ant Group
• Dongge Wang EPFL
• Hangyu Mao Kuaishou Technology
• Jianye Hao Tianjin University
• Jun Wang
• Xiaotie Deng

Keywords

• Cooperative multiagent reinforcement learning
• Centralized training with decentralized execution
• Multiagent Q-learning
• Value
• factorization framework
• Sub-team coordination