From General Relation Patterns to Task-Specific Decision-Making in Continual Multi-Agent Coordination

Continual Multi-Agent Reinforcement Learning (Co-MARL) requires agents to address catastrophic forgetting issues while learning new coordination policies with the dynamics team. In this paper, we delve into the core of Co-MARL, namely Relation Patterns, which refer to agents’ general understanding of interactions. In addition to generality, relation patterns exhibit task-specificity when mapped to different action spaces. To this end, we propose a novel method called General Relation Patterns-Guided Task-specific Decision-Maker (RPG). In RPG, agents extract relation patterns from dynamic observation spaces using a relation capturer. These task-agnostic relation patterns are then mapped to different action spaces via a task-specific decision-maker generated by a conditional hypernetwork. To combat forgetting, we further introduce regularization items on both the relation capturer and the conditional hypernetwork. Results on SMAC and LBF demonstrate that RPG effectively prevents catastrophic forgetting when learning new tasks and achieves zero-shot generalization to unseen tasks.

Authors

• Chang Yao Beijing Jiaotong University Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence
• Youfang Lin Beijing Jiaotong University Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence
• Shoucheng Song Beijing Jiaotong University Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence
• Hao Wu Beijing Jiaotong University Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence
• Yuqing Ma Institute of Artificial Intelligence, Beihang University State Key Laboratory of Complex & Critical Software Environment, Beihang University
• Sheng Han Beijing Jiaotong University Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence
• Kai Lv Beijing Jiaotong University Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence

Keywords

• Agent-based and Multi-agent Systems: MAS: Coordination and cooperation
• Machine Learning: ML: Multiagent Reinforcement Learning