RoboEngine: Plug-and-Play Robot Data Augmentation with Semantic Robot Segmentation and Background Generation

Visual augmentation has become a crucial technique for enhancing the visual robustness of imitation learning. However, existing methods are often limited by prerequisites such as camera calibration or the need for controlled environments (e. g. , green screen setups). In this work, we introduce RoboEngine, the first plug-and-play visual robot data augmentation toolkit. For the first time, users can effortlessly generate physics- and task-aware robot scenes with just a few lines of code. To achieve this, we present a novel robot scene segmentation dataset, a generalizable high-quality robot segmentation model, and a fine-tuned background generation model, which together form the core components of the out-of-the-box toolkit. Using RoboEngine, we demonstrate the ability to generalize robot manipulation tasks across six entirely new scenes, based solely on demonstrations collected from a single scene, achieving a more than 200% performance improvement compared to the no-augmentation baseline. All datasets, model weights, and the toolkit are released https://roboengine.github.io/.

Authors

• Chengbo Yuan
• Suraj Joshi
• Shaoting Zhu
• Hang Su
• Hang Zhao 0021
• Yang Gao 0029

Keywords

• Visualization
• Three-dimensional displays
• Codes
• Imitation learning
• Data augmentation
• Diffusion models
• Robustness
• Robots
• Standards
• Videos
• Semantic Segmentation
• Data Visualization
• Segmentation Model
• Robot Manipulator
• Camera Calibration
• High-quality Models
• Fine-tuned Model
• Single Scene
• General Method
• Computer Vision
• Visible Changes
• Diffusion Model
• Visual Disturbances
• Robotic Arm
• Augmentation Methods
• Augmentation Techniques
• Real Robot
• Inpainting
• Number Of Demonstrations
• Scene Description
• Training Policy