Offline Imitation Learning upon Arbitrary Demonstrations by Pre-Training Dynamics Representations

Limited data has become a major bottleneck in scaling up offline imitation learning (IL). In this paper, we propose enhancing IL performance under limited expert data by introducing a pre-training stage that learns dynamics representations, derived from factorizations of the transition dynamics. We first theoretically justify that the optimal decision variable of offline IL lies in the representation space, significantly reducing the parameters to learn in the downstream IL. Moreover, the dynamics representations can be learned from arbitrary data collected with the same dynamics, allowing the reuse of massive non-expert data and mitigating the limited data issues. We present a tractable loss function inspired by noise contrastive estimation to learn the dynamics representations at the pre-training stage. Experiments on MuJoCo demonstrate that our proposed algorithm can mimic expert policies with as few as a single trajectory. Experiments on real quadrupeds show that we can leverage pre-trained dynamics representations from simulator data to learn to walk from a few real-world demonstrations.

Authors

• Haitong Ma
• Bo Dai 0001
• Zhaolin Ren
• Yebin Wang
• Na Li 0002

Keywords

• Imitation learning
• Heuristic algorithms
• Noise
• Estimation
• Trajectory
• Quadrupedal robots
• Intelligent robots
• Dynamic Representation
• Offline Learning
• Representation Of Space
• Optimization Variables
• Transition Dynamics
• Single Trajectory
• Quadrupedal
• Arbitrary Data
• Pre-training Stage
• Neural Network
• Gaussian Noise
• Kullback-Leibler
• Representation Learning
• Convex Function
• Random Data
• Density Ratio
• Reward Function
• Markov Decision Process
• Auxiliary Data
• Two-stage Algorithm
• Distribution Matching
• Expert Demonstrations
• Behavior Policy
• Inverse Reinforcement Learning
• Fenchel Conjugate
• Locomotion Tasks
• Front Foot
• Minimax Optimization
• Poor Generalization