TMAE:Learning Targeted Multi-Agent Exploration via Causal Inference

Exploration in sparse-reward tasks remains a fundamental challenge in multi-agent reinforcement learning (MARL) due to complex inter-agent interactions and the expansive exploration space. To address this issue, we propose Targeted Multi-Agent Exploration (TMAE), a novel framework that uncovers the causal relationships between the state space and the reward function, thereby reducing the exploration space and enabling more targeted exploration. Specifically, we construct a structural causal model (SCM) to model the causality between sub-state variables and sparse rewards, providing a robust analytical foundation for subsequent causal inference. Through counterfactual causal intervention, TMAE identifies the most critical subspaces for discovering rare but pivotal events while filtering out confounders. By incorporating these causal insights into the exploration process, TMAE prioritizes subspaces with stronger causal effects on sparse rewards, significantly enhancing exploration efficiency. We evaluate TMAE on a range of MARL benchmarks featuring sparse rewards, consistently demonstrating superior exploration efficiency compared to state-of-the-art methods. Furthermore, visualized causal insights derived from TMAE reveal its ability to effectively capture intricate dependencies and priorities in targeted exploration, showcasing strong alignment with prior domain knowledge.

Authors

• Chuxiong Sun Science and Technology on Integrated Information System Laboratory, Institute of Software Chinese Academy of Sciences State Key Laboratory of Intelligent Game
• Dunqi Yao National Key Laboratory of Space Integrated Information System, Institute of Software, Chinese Academy of Sciences University of Chinese Academy of Sciences
• Rui Wang
• Wenwen Qiang Science & Technology on Integrated Information System Laboratory, Institute of Software Chinese Academy of Sciences
• Changwen Zheng Institute of Software, Chinese Academy of Sciences University of the Chinese Academy of Sciences
• Jiangmeng Li Science & Technology on Integrated Information System Laboratory, Institute of Software Chinese Academy of Sciences State Key Laboratory of Intelligent Game

Keywords

No keywords are indexed for this paper.