Teaching Robots to Predict Human Motion

Teaching a robot to predict and mimic how a human moves or acts in the near future by observing a series of historical human movements is a crucial first step in human-robot interaction and collaboration. In this paper, we instrument a robot with such a prediction ability by leveraging recent deep learning and computer vision techniques. First, our system takes images from the robot camera as input to produce the corresponding human skeleton based on real-time human pose estimation obtained with the OpenPose library. Then, conditioning on this historical sequence, the robot forecasts plausible motion through a motion predictor, generating a corresponding demonstration. Because of a lack of high-level fidelity validation, existing forecasting algorithms suffer from error accumulation and inaccurate prediction. Inspired by generative adversarial networks (GANs), we introduce a global discriminator that examines whether the predicted sequence is smooth and realistic. Our resulting motion GAN model achieves superior prediction performance to state-of-the-art approaches when evaluated on the standard H3. 6M dataset. Based on this motion GAN model, the robot demonstrates its ability to replay the predicted motion in a human-like manner when interacting with a person.

Authors

• Liangyan Gui
• Kevin Zhang 0002
• Yu-Xiong Wang
• Xiaodan Liang
• José M. F. Moura
• Manuela Veloso

Keywords

• Robots
• Gallium nitride
• Generative adversarial networks
• Predictive models
• Cameras
• Skeleton
• Decoding
• Human Motion
• Deep Learning
• Near Future
• Pose Estimation
• Human-robot Interaction
• Error Accumulation
• Inaccurate Predictions
• Motion Prediction
• Human Pose Estimation
• Human Pose
• Historical Sequence
• Series Of Movements
• Generation Sequencing
• Recurrent Network
• Long Short-term Memory
• Recurrent Neural Network
• Point Cloud
• Variety Of Tasks
• Input Sequence
• Gated Recurrent Unit
• Latent Representation
• 3D Point
• Encoder-decoder Network
• Inference Stage
• Competitive Game
• Global Perspective
• Depth Camera
• Gated Recurrent Unit Layer
• Hidden Size