What Do Latent Action Models Actually Learn?

Latent action models (LAMs) aim to learn action-relevant changes from unlabeled videos by compressing changes between frames as latents. However, differences between video frames can be caused by \textit{controllable changes} as well as exogenous noise, leading to an important concern -- do latents capture the changes caused by actions or irrelevant noise? This paper studies this issue analytically, presenting a linear model that encapsulates the essence of LAM learning, while being tractable. This provides several insights, including connections between LAM and principal component analysis (PCA), desiderata of the data-generating policy, and justification of strategies to encourage learning controllable changes using data augmentation, data cleaning, and auxiliary action-prediction. We also provide illustrative results based on numerical simulation, shedding light on the specific structure of observations, actions, and noise in data that influence LAM learning.

Authors

• Chuheng Zhang Microsoft Research
• Tim Pearce Microsoft Research
• Pushi Zhang
• Kaixin Wang
• Xiaoyu Chen Beijing Jiaotong University
• Wei Shen
• Li Zhao Shaanxi Normal University
• Jiang Bian University of Florida

Keywords

No keywords are indexed for this paper.