Learning Graph-Enhanced Commander-Executor for Multi-Agent Navigation

This paper investigates the multi-agent navigation problem, which requires multiple agents to reach the target goals in a limited time. Multi-agent reinforcement learning (MARL) has shown promising results for solving this issue. However, it is inefficient for MARL to directly explore the (nearly) optimal policy in the large search space, which is exacerbated as the agent number increases (e. g. , 10+ agents) or the environment is more complex (e. g. , 3𝐷 simulator). Goal-conditioned hierarchical reinforcement learning (HRL) provides a promising direction to tackle this challenge by introducing a hierarchical structure to decompose the search space, where the low-level policy predicts primitive actions in the guidance of the goals derived from the high-level policy. In this paper, we propose Multi-Agent Graph-Enhanced Commander-EXecutor (MAGE-X), a graph-based goal-conditioned hierarchical method for multi-agent navigation tasks. MAGE-X comprises a high-level Goal Commander and a low-level Action Executor. The Goal Commander predicts the probability distribution of the goals and leverages them to assign the most appropriate final target to each agent. The Action Executor utilizes graph neural networks (GNN) to construct a subgraph for each agent that only contains its crucial partners to improve cooperation. Additionally, the Goal Encoder in the Action Executor captures the relationship between the agent and the designated goal to encourage the agent to reach the final target. The results show that MAGE-X outperforms the state-of-the-art MARL baselines with a 100% success rate with only 3 million training steps in multi-agent particle environments (MPE) with 50 agents, and at least a 12% higher success rate and 2× higher data efficiency in a more complicated quadrotor 3𝐷 navigation task.

Authors

• Xinyi Yang
• Shiyu Huang 4Paradigm Inc.
• Yiwen Sun
• Yuxiang Yang Hangzhou Dianzi University
• Chao Yu Sun Yat-sen University Pengcheng Laboratory
• Wei-Wei Tu 4Paradigm Inc.
• Huazhong Yang Tsinghua University
• Yu Wang

Keywords

• Multi-agent Reinforcement Learning
• Goal-conditioned Reinforcement Learning
• Multi-agent Navigation
• Graph Neural Network
• Proc. of the 22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2023)
• A. Ricci
• W. Yeoh
• N. Agmon
• B. An (eds.)
• May 29 – June 2
• 2023
• London
• United Kingdom. © 2023 International Foundation for Autonomous Agents
• and Multiagent Systems (www. ifaamas. org). All rights reserved.