A Generative Model-Based Predictive Display for Robotic Teleoperation

We propose a new generative model-based predictive display for robotic teleoperation over high-latency communication links. Our method is capable of rendering photo-realistic images of the scene to the human operator in real time from RGB-D images acquired by the remote robot. A preliminary exploration stage is used to build a coarse 3D map of the remote environment and to train a generative model, both of which are then used to generate photo-realistic images for the human operator based on the commanded pose of the robot. Data captured by the remote robot is used to dynamically update the 3D map, enabling teleoperation in the presence of new and relocated objects. Various experiments validate our proposed method’s performance and benefits over alternative methods.

Authors

• Bowen Xie
• Mingjie Han
• Jun Jin 0001
• Martin Barczyk
• Martin Jägersand

Keywords

• Solid modeling
• Visualization
• Three-dimensional displays
• Two dimensional displays
• Pipelines
• Semantics
• Predictive models
• Communication Links
• RGB-D Images
• Robot Pose
• Photo-realistic Images
• Environmental Changes
• Field Of View
• Learning Models
• Image Quality
• Quantitative Evaluation
• 3D Reconstruction
• 2D Images
• Localization Performance
• Depth Images
• Geometric Model
• Peak Signal-to-noise Ratio
• Feature Matching
• Synthetic Images
• Image Enhancement
• Ground Truth Image
• Camera Pose
• Signed Distance Function
• Voxel Space
• Scene Model
• Multi-view Stereo
• Inference Time
• Visual Feedback
• Workstation