Generalization in Reinforcement Learning by Soft Data Augmentation

Extensive efforts have been made to improve the generalization ability of Reinforcement Learning (RL) methods via domain randomization and data augmentation. However, as more factors of variation are introduced during training, optimization becomes increasingly challenging, and empirically may result in lower sample efficiency and unstable training. Instead of learning policies directly from augmented data, we propose SOft Data Augmentation (SODA), a method that decouples augmentation from policy learning. Specifically, SODA imposes a soft constraint on the encoder that aims to maximize the mutual information between latent representations of augmented and non-augmented data, while the RL optimization process uses strictly non-augmented data. Empirical evaluations are performed on diverse tasks from DeepMind Control suite as well as a robotic manipulation task, and we find SODA to significantly advance sample efficiency, generalization, and stability in training over state-of-the-art vision-based RL methods. 1

Authors

• Nicklas Hansen 0001
• Xiaolong Wang 0004

Keywords

• Training
• Automation
• Conferences
• Reinforcement learning
• Task analysis
• Optimization
• Robots
• Data Augmentation
• Variety Of Factors
• Mutual Information
• Extensive Efforts
• Sampling Efficiency
• Domain Adaptation
• Empirical Evaluation
• Robot Manipulator
• Policy Learning
• Reinforcement Learning Methods
• Robotic Tasks
• Diverse Tasks
• Unstable Training
• Benchmark
• Gradient Descent
• Convolutional Layers
• Negative Samples
• Multilayer Perceptron
• Representation Learning
• Latent Space
• Self-supervised Learning
• Visual Learning
• Reinforcement Learning Policy
• Random Cropping
• Auxiliary Task
• Real-world Deployment
• Apply Data Augmentation
• Test Environment
• Robotic Arm
• Overview Of Architecture