Sensor substitution for video-based action recognition

There are many applications where domain-specific sensing, such as accelerometers, kinematics, or force sensing, provide unique and important information for control or for analysis of motion. However, it is not always the case that these sensors can be deployed or accessed beyond laboratory environments. For example, it is possible to instrument humans or robots to measure motion in the laboratory in ways that it is not possible to replicate in the wild. An alternative, which we explore in this paper, is to address situations where accurate sensing is available while training an algorithm, but for which only video is available for deployment. We present two examples of this sensory substitution methodology. The first variation trains a convolutional neural network to regress real-valued signals, including robot end-effector pose, from video. The second example regresses binary signals derived from accelerometer data which signifies when specific objects are in motion. We evaluate these on the JIGSAWS dataset for robotic surgery training assessment and the 50 Salads dataset for modeling complex structured cooking tasks. We evaluate the trained models for video-based action recognition and show that the trained models provide information that is comparable to the sensory signals they replace.

Authors

• Christian Rupprecht 0001
• Colin Lea
• Federico Tombari
• Nassir Navab
• Gregory D. Hager

Keywords

• Robot sensing systems
• Training
• Convolution
• Computer architecture
• Accelerometers
• Action Recognition
• Video-based Action Recognition
• Convolutional Neural Network
• Accelerometer
• Specific Objectives
• Sensory Signals
• Action Recognition Model
• Training Set
• Use Of Tools
• Continuous-time
• Sensor Data
• Hidden Markov Model
• Long Short-term Memory
• Recurrent Neural Network
• Image Object
• Video Analysis
• Convolutional Neural Network Architecture
• Optical Flow
• Dirac Delta
• Real Measurements
• Da Vinci Surgical System
• Sensor Values
• Surgical Training
• Virtual Sensors
• Objects In The Scene
• Bag-of-words
• Latent State
• Virtual Data
• Computer Vision
• Deep Learning