SynthAct: Towards Generalizable Human Action Recognition based on Synthetic Data

Synthetic data generation is a proven method for augmenting training sets without the need for extensive setups, yet its application in human activity recognition is underexplored. This is particularly crucial for human-robot collaboration in household settings, where data collection is often privacy-sensitive. In this paper, we introduce SynthAct, a synthetic data generation pipeline designed to significantly minimize the reliance on real-world data. Leveraging modern 3D pose estimation techniques, SynthAct can be applied to arbitrary 2D or 3D video action recordings, making it applicable for uncontrolled in-the-field recordings by robotic agents or smarthome monitoring systems. We present two SynthAct datasets: AMARV, a large synthetic collection with over 800k multi-view action clips, and Synthetic Smarthome, mirroring the Toyota Smarthome dataset. SynthAct generates a rich set of data, including RGB videos and depth maps from four synchronized views, 3D body poses, normal maps, segmentation masks and bounding boxes. We validate the efficacy of our datasets through extensive synthetic-to-real experiments on NTU RGB+D and Toyota Smarthome. SynthAct is available on our project page 4.

Authors

• David Schneider 0006
• Marco Keller
• Zeyun Zhong
• Kunyu Peng
• Alina Roitberg
• Jürgen Beyerer
• Rainer Stiefelhagen

Keywords

• Training
• Three-dimensional displays
• Pose estimation
• Training data
• Data collection
• Generators
• Recording
• Human Activities
• Action Recognition
• Human Activity Recognition
• Data Generation
• Real-world Data
• Bounding Box
• Depth Map
• Human Pose Estimation
• 3D Pose
• 3D Body
• Household Settings
• Synthetic Data Generation
• Human-robot Collaboration
• Body Pose
• Fine-tuned
• Light Source
• Model Input
• Body Shape
• Need For Data Collection
• Need For Data
• Disparity Map
• Domain Gap
• Motion Capture
• Real Training Data
• Real-world Datasets
• Real Training
• Shape Estimation
• Synthetic Training Data