Normalization Enhances Generalization in Visual Reinforcement Learning

Recent advances in visual reinforcement learning (RL) have led to impressive success in handling complex tasks. However, these methods have demonstrated limited generalization capability to visual disturbances, which poses a significant challenge to their real-world application and adaptability. Though normalization techniques have demonstrated huge success in supervised and unsupervised learning, their applications in visual RL are still scarce. In this paper, we explore the potential benefits of integrating normalization into visual RL methods with respect to generalization performance. We find that, perhaps surprisingly, incorporating suitable normalization techniques is sufficient to enhance the generalization capabilities, without any additional special design. We utilize the combination of two normalization techniques, CrossNorm and SelfNorm, for generalizable visual RL. Extensive experiments are conducted on DMControl Generalization Benchmark, CARLA, and ProcGen Benchmark to validate the effectiveness of our method. We show that our method significantly improves generalization capability while only marginally affecting sample efficiency. In particular, when integrated with DrQ-v2, our method enhances the test performance of DrQ-v2 on CARLA across various scenarios, from 14% of the training performance to 97%. Our project page: https: //sites. google. com/view/norm-generalization-vrl/home

Authors

• Lu Li Huazhong Agricultural University
• Jiafei Lyu Shenzhen International Graduate School, Tsinghua University
• Guozheng Ma
• Zilin Wang Shenzhen International Graduate School, Tsinghua University
• Zhenjie Yang
• Xiu Li Tsinghua University
• Zhiheng Li Northeastern University

Keywords

• generalization
• visual reinforcement learning
• normalization
• *: Equal contribution. †: Corresponding authors.
• This work is licensed under a Creative Commons Attribution
• International 4. 0 License.
• Proc. of the 23rd International Conference on Autonomous Agents and Multiagent Systems
• (AAMAS 2024)
• N. Alechina
• V. Dignum
• M. Dastani
• J. S. Sichman (eds.)
• May 6 – 10
• 2024
• Auckland
• New Zealand. © 2024 International Foundation for Autonomous Agents and
• Multiagent Systems (www. ifaamas. org).