Subspace-Aware Exploration for Sparse-Reward Multi-Agent Tasks

Exploration under sparse rewards is a key challenge for multi-agent reinforcement learning problems. One possible solution to this issue is to exploit inherent task structures for an acceleration of exploration. In this paper, we present a novel exploration approach, which encodes a special structural prior on the reward function into exploration, for sparse-reward multi-agent tasks. Specifically, a novel entropic exploration objective which encodes the structural prior is proposed to accelerate the discovery of rewards. By maximizing the lower bound of this objective, we then propose an algorithm with moderate computational cost, which can be applied to practical tasks. Under the sparse-reward setting, we show that the proposed algorithm significantly outperforms the state-of-the-art algorithms in the multiple-particle environment, the Google Research Football and StarCraft II micromanagement tasks. To the best of our knowledge, on some hard tasks (such as 27m_vs_30m}) which have relatively larger number of agents and need non-trivial strategies to defeat enemies, our method is the first to learn winning strategies under the sparse-reward setting.

Authors

• Pei Xu Institute of Automation, Chinese Academy of Sciences
• Junge Zhang Institute of Automation, Chinese academy of science School of Artificial Intelligence, University of Chinese Academy of Sciences
• Qiyue Yin National Key Laboratory of Cognition and Decision Intelligence for Complex Systems & Institution of Automation, Chinese Academy of Sciences
• Chao Yu Sun Yat-sen University & Pengcheng Laboratory
• Yaodong Yang Institute for Artificial Intelligence, Peking University
• Kaiqi Huang School of Artificial Intelligence, University of Chinese Academy of Sciences & Institute of Automation, Chinese Academy of Sciences

Keywords

• MAS: Multiagent Learning
• ML: Reinforcement Learning Algorithms