A Causality-Inspired Spatial-Temporal Return Decomposition Approach for Multi-Agent Reinforcement Learning

Cooperativemulti-agentreinforcementlearning(MARL)hasachieved strongperformance, butitremainslimitedinexplaininghowagents’ decisions contribute to outcomes. This limitation is especially acute under delayed, episodic rewards, where credit must be assigned across both time and agents. We propose CAusally-inspired Spatial- Temporal return decomposition (CAST) for episodic cooperative MARL. CAST provides an interpretable decomposition while relaxing common assumptions on multi-agent reward structure. Temporally, the episodic return is expressed as a sum of per-timestep team rewards. Spatially, team rewards are modeled as general nonlinear mixtures of individual rewards rather than simple additive forms, enabling more flexible and accurate credit assignment. We show that team rewards, individual rewards, and the underlying causal relations are identifiable under our framework, yielding structural constraints that improve interpretability. Experiments on MPE and variants demonstrate state-of-the-art performance and qualitative visualizations that reveal meaningful causal structure.

Authors

• Yudi Zhang Eindhoven University of Technology
• Yali Du King's College London
• Biwei Huang University of California San Diego
• Meng Fang University of Liverpool
• Mykola Pechenizkiy Eindhoven University of Technology

Keywords

• Multi-agent
• Spatial-Temporal Credit Assignment
• Causality