Fairness in Cooperative Multi-objective Multi-agent Reinforcement Learning using Expected Utility

Fairness as equity and compromise across multiple viewpoints is a necessary consideration in any decision that is evaluated from several possibly conflicting perspectives. It is also a property that artificial decision-making agents should uphold to be deployable to real-worldproblems. However, existingworkinsequentialdecisionmaking ensures fairness among agents or objectives but struggles with real-world problems that are both multi-agent and multiobjective. Furthermore, research integrating fairness into Multi- ObjectiveReinforcementLearning(MORL)isfocusedonoptimizing the Scalarized Expected Return (SER) criterion while mostly ignoring the Expected Scalarized Return (ESR) criterion. We argue that fairness in MORL should also be investigated under ESR since it is sometimes more suitable when solving problems where fairness matters. In this paper, we study objective-wise fairness in cooperative multi-agent multi-objective decision-making under ESR. We propose the first algorithm that learns efficient decentralized policies while enforcing fairness across objectives under ESR. We identify a key challenge in this setting related to policy conditioning on globally accumulated returns, which hinders decentralized learning and execution, and we present an approach to address it based on inter-agent communication. Experiments on discrete and continuous control tasks demonstrate that our method outperforms existing baselines.

Authors

• Fares Chouaki LIP6, Sorbonne Université
• Aurélie Beynier LIP6, Sorbonne Université
• Nicolas Maudet LIP6, Sorbonne Université
• Paolo Viappiani LAMSADE, PSL

Keywords

• Multi-Objective Reinforcement Learning
• Multi-agent Reinforcement Learning
• Expected Scalarized Return
• Fairness