Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model

Preference-based reinforcement learning (PbRL) provides a powerful paradigm to avoid meticulous reward engineering by learning rewards based on human preferences. However, real-time human feedback is hard to obtain in online tasks. Most work suppose there is a "scripted teacher" that utilizes privileged predefined reward to provide preference feedback. In this paper, we propose a RL Self-augmented Large Language Model Feedback (RL-SaLLM-F) technique that does not rely on privileged information for online PbRL. RL-SaLLM-F leverages the reflective and discriminative capabilities of LLM to generate self-augmented trajectories and provide preference labels for reward learning. First, we identify a failure issue in LLM-based preference discrimination, specifically "query ambiguity", in online PbRL. Then LLM is employed to provide preference labels and generate self-augmented imagined trajectories that better achieve the task goal, thereby enhancing the quality and efficiency of feedback. Additionally, a double-check mechanism is introduced to mitigate randomness in the preference labels, improving the reliability of LLM feedback. The experiment across multiple tasks in the MetaWorld benchmark demonstrates the specific contributions of each proposed module in RL-SaLLM-F, and shows that self-augmented LLM feedback can effectively replace the impractical "scripted teacher" feedback. In summary, RL-SaLLM-F introduces a new direction of feedback acquisition in online PbRL that does not rely on any online privileged information, offering an efficient and lightweight solution with LLM-driven feedback. 1 1Corresponding author: Qichao Zhang (zhangqichao2014@ia. ac. cn) and Xiangyuan Lan (lanxy@pcl. ac. cn). Code Page: https: //github. com/TU2021/RL-SaLLM-F This work is licensed under a Creative Commons Attribution International 4. 0 License. Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025), Y. Vorobeychik, S. Das, A. Nowé (eds.), May 19 – 23, 2025, Detroit, Michigan, USA. © 2025 International Foundation for Autonomous Agents and Multiagent Systems (www. ifaamas. org).

Authors

• Songjun Tu State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA, Beijing, China Peng Cheng Laboratory, Shenzhen, China School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
• Jingbo Sun State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA, Beijing, China Peng Cheng Laboratory, Shenzhen, China School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
• Qichao Zhang State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA, Beijing, China School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
• Xiangyuan Lan Pengcheng Laboratory Pazhou Laboratory (Huangpu)
• Dongbin Zhao State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA, Beijing, China Peng Cheng Laboratory, Shenzhen, China School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

Keywords

• Online Preference-based Reinforcement Learning
• Self-augmented
• LLM
• LLM-driven Feedback
• Query Ambiguity.