Vision-Based Gesture Recognition in Human-Robot Teams Using Synthetic Data

Building successful collaboration between humans and robots requires efficient, effective, and natural communication. Here we study a RGB-based deep learning approach for controlling robots through gestures (e. g. , "follow me"). To address the challenge of collecting high-quality annotated data from human subjects, synthetic data is considered for this domain. We contribute a dataset of gestures that includes real videos with human subjects and synthetic videos from our custom simulator. A solution is presented for gesture recognition based on the state-of-the-art I3D model. Comprehensive testing was conducted to optimize the parameters for this model. Finally, to gather insight on the value of synthetic data, several experiments are described that systematically study the properties of synthetic data (e. g. , gesture variations, character variety, generalization to new gestures). We discuss practical implications for the design of effective human-robot collaboration and the usefulness of synthetic data for deep learning.

Authors

• Celso M. de Melo
• Brandon Rothrock
• Prudhvi Gurram
• Oytun Ulutan
• B. S. Manjunath

Keywords

• Deep learning
• Training
• Collaboration
• Gesture recognition
• Intelligent robots
• Videos
• Testing
• Human-robot Teams
• Vision-based Gesture Recognition
• Human Subjects
• Real Videos
• Human-robot Collaboration
• Training Set
• Human Activities
• Human Data
• Important Problem
• Object Detection
• Skin Color
• Forms Of Communication
• Action Recognition
• Target Domain
• Optical Flow
• Domain Adaptation
• US Army
• Input Resolution
• Gesture Classification
• Human Activity Recognition
• Critical Comparison
• Gesture Types
• Gesture Performance
• Action Recognition Model
• Ego-motion
• Deep Learning Models
• Real Test
• Robot Control