Exploring Visual Pre-training for Robot Manipulation: Datasets, Models and Methods

Visual pre-training with large-scale real-world data has made great progress in recent years, showing great potential in robot learning with pixel observations. However, the recipes of visual pre-training for robot manipulation tasks are yet to be built. In this paper, we thoroughly investigate the effects of visual pre-training strategies on robot manipulation tasks from three fundamental perspectives: pre-training datasets, model architectures and training methods. Several significant experimental findings are provided that are beneficial for robot learning. Further, we propose a visual pre-training scheme for robot manipulation termed Vi-PRoM, which combines self-supervised learning and supervised learning. Concretely, the former employs contrastive learning to acquire underlying patterns from large-scale unlabeled data, while the latter aims learning visual semantics and temporal dynamics. Extensive experiments on robot manipulations in various simulation environments and the real robot demonstrate the superiority of the proposed scheme. Videos and more details can be found on https://explore-pretrain-robot.github.io.

Authors

• Ya Jing
• Xuelin Zhu
• Xingbin Liu
• Qie Sima
• Taozheng Yang
• Yunhai Feng
• Tao Kong

Keywords

• Training
• Visualization
• Supervised learning
• Semantics
• Self-supervised learning
• Robot learning
• Task analysis
• Robot Manipulator
• Temporal Dynamics
• Large-scale Data
• Simulation Environment
• Model Architecture
• Manipulation Tasks
• Potential Learning
• Visual Strategies
• Real Robot
• Progress In Recent Years
• Pre-training Dataset
• Visual Representation
• ImageNet
• Video Clips
• Learning Objectives
• Representation Learning
• Pre-training Method
• Visual Encoding
• Robotic Tasks
• Proximal Policy Optimization
• Imitation Learning
• Robot Model
• Strong Generalization Ability
• Classification Head
• Sequence Patterns
• Pre-trained Encoder