Efficient Multi-task Reinforcement Learning with Cross-Task Policy Guidance

Multi-task reinforcement learning endeavors to efficiently leverage shared information across various tasks, facilitating the simultaneous learning of multiple tasks. Existing approaches primarily focus on parameter sharing with carefully designed network structures or tailored optimization procedures. However, they overlook a direct and complementary way to exploit cross-task similarities: the control policies of tasks already proficient in some skills can provide explicit guidance for unmastered tasks to accelerate skills acquisition. To this end, we present a novel framework called Cross-Task Policy Guidance (CTPG), which trains a guide policy for each task to select the behavior policy interacting with the environment from all tasks' control policies, generating better training trajectories. In addition, we propose two gating mechanisms to improve the learning efficiency of CTPG: one gate filters out control policies that are not beneficial for guidance, while the other gate blocks tasks that do not necessitate guidance. CTPG is a general framework adaptable to existing parameter sharing approaches. Empirical evaluations demonstrate that incorporating CTPG with these approaches significantly enhances performance in manipulation and locomotion benchmarks.

Authors

• Jinmin He Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences
• Kai Li C 2 DL, Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Scien
• Yifan Zang Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences
• Haobo Fu Tencent AI Lab
• Qiang Fu Tencent AI Lab
• Junliang Xing Department of Computer Science and Technology, Tsinghua University
• Jian Cheng C 2 DL, Institute of Automation, Chinese Academy of Sciences School of Future Technology, University of Chinese Academy of Sciences & AiRiA

Keywords

No keywords are indexed for this paper.