Offline Multi-Agent Preference-based Reinforcement Learning with Agent-aware Direct Preference Optimization

Multi-agent Preference-Based Reinforcement Learning (MAPbRL) is promising in offline policy learning by leveraging human preferences to replace complex manual reward designing. Current MAPbRL methods use complicated structures to realize better reward modeling with off-the-shelf MARL algorithms and obtain the joint policy based on it. However, it faces a severe preference-behavior mismatch problem stemming from the instability of RL training and global-local preference inconsistency datasets in offline MARL, resulting in potential suboptimal policy convergence. To address this problem, we propose Agent-aware Multi-Agent Direct Preference Optimization (AMADPO) by utilizing a multi-agent preference predictor to guide agent-aware direct optimization from imbalanced preference labels, which can learn coordination policy from both positive and negative segments. Experimental results in SMAC environment show substantial improvements in global-local preference inconsistency datasets, demonstrating the effectiveness of AMADPO in solving the preference-behavior mismatch problem.

Authors

• Qian Kou National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Application Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University
• Mingyang Li Alibaba Cloud Computing
• Zeyang Liu Xi'an Jiaotong University
• Long Qian
• Zhuoran Chen Xi'an Jiaotong University
• Lipeng Wan Xi'an Jiaotong University
• Xingyu Chen Xi'an Jiaotong University
• Xuguang Lan Xi'an Jiaotong University

Keywords

• Multi-agent reinforcement learning
• Preference-based reinforcement learning
• Offline reinforcement learning