Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos

Understanding how humans would behave during hand-object interaction (HOI) is vital for applications in service robot manipulation and extended reality. To achieve this, some recent works simultaneously forecast hand trajectories and object affordances on human egocentric videos. The joint prediction serves as a comprehensive representation of future HOI in 2D space, indicating potential human motion and motivation. However, the existing approaches mostly adopt the autoregressive paradigm, which lacks bidirectional constraints within the holistic future sequence, and accumulates errors along the time axis. Meanwhile, they overlook the effect of camera egomotion on first-person view predictions. To address these limitations, we propose a novel diffusion-based HOI prediction method, namely Diff-IP2D, to forecast future hand trajectories and object affordances with bidirectional constraints in an iterative non-autoregressive manner on egocentric videos. Motion features are further integrated into the conditional denoising process to enable Diff-IP2D aware of the camera wearer’s dynamics for more accurate interaction prediction. Extensive experiments demonstrate that Diff-IP2D significantly outperforms the state-of-the-art baselines on both the off-the-shelf and our newly proposed evaluation metrics. This highlights the efficacy of leveraging a generative paradigm for 2D HOI prediction. The code and the video have been released at https://github.com/IRMVLab/Diff-IP2D.

Authors

• Junyi Ma
• Xieyuanli Chen
• Jingyi Xu
• Hesheng Wang 0001

Keywords

• Hands
• Measurement
• Service robots
• Affordances
• Noise reduction
• Predictive models
• Cameras
• Motion capture
• Trajectory
• Videos
• Hand-object Interaction
• Egocentric Videos
• Time Axis
• Human Motion
• Future Trajectories
• Robot Manipulator
• Ego-motion
• Hand Trajectory
• Joint Prediction
• Object Affordances
• Left Side
• Transformer
• Contact Point
• Global Features
• Multilayer Perceptron
• Reversible Process
• Unified Model
• Diffusion Model
• Latent Space
• Latent Features
• Trajectory Prediction
• Forward Process
• Artificial Intelligence Systems
• Past Observations
• Strategy Options
• Conditional Variational Autoencoder
• Homography Matrix
• Human Intention
• Hand Contact
• Displacement Error