Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning

Context, the embedding of previous collected trajectories, is a powerful construct for Meta-Reinforcement Learning (Meta- RL) algorithms. By conditioning on an effective context, Meta-RL policies can easily generalize to new tasks within a few adaptation steps. We argue that improving the quality of context involves answering two questions: 1. How to train a compact and sufficient encoder that can embed the taskspecific information contained in prior trajectories? 2. How to collect informative trajectories of which the corresponding context reflects the specification of tasks? To this end, we propose a novel Meta-RL framework called CCM (Contrastive learning augmented Context-based Meta-RL). We first focus on the contrastive nature behind different tasks and leverage it to train a compact and sufficient context encoder. Further, we train a separate exploration policy and theoretically derive a new information-gain-based objective which aims to collect informative trajectories in a few steps. Empirically, we evaluate our approaches on common benchmarks as well as several complex sparse-reward environments. The experimental results show that CCM outperforms state-of-the-art algorithms by addressing previously mentioned problems respectively.

Authors

• Haotian Fu
• Hongyao Tang College of Intelligence and Computing, Tianjin University
• Jianye Hao Tianjin University
• Chen Chen University of Virginia
• Xidong Feng
• Dong Li University of Science and Technology of China
• Wulong Liu

Keywords

No keywords are indexed for this paper.