Learning Transition Models with Time-delayed Causal Relations

This paper introduces an algorithm for discovering implicit and delayed causal relations between events observed by a robot at arbitrary times, with the objective of improving data-efficiency and interpretability of model- based reinforcement learning (RL) techniques. The proposed algorithm initially predicts observations with the Markov assumption, and incrementally introduces new hidden variables to explain and reduce the stochasticity of the observations. The hidden variables are memory units that keep track of pertinent past events. Such events are systematically identified by their information gains. The learned transition and reward models are then used for planning. Experiments on simulated and real robotic tasks show that this method significantly improves over current RL techniques.

Authors

• Junchi Liang
• Abdeslam Boularias

Keywords

• Reinforcement learning
• Markov processes
• Prediction algorithms
• Robot sensing systems
• Tires
• Planning
• Task analysis
• Causal Relationship
• Transition Model
• Model Interpretation
• Information Gain
• Past Events
• Hidden Variables
• Markov Property
• Memory Unit
• Real Robot
• Neural Network
• Learning Models
• Density Estimation
• Recurrent Neural Network
• Attention Mechanism
• Image Object
• Graphical Model
• Object Properties
• Reward Function
• Optimal Path
• Markov Decision Process
• Proximal Policy Optimization
• Dynamic Bayesian Network
• Transition Function
• End Of Episode
• Regions Of The State Space
• Reinforcement Learning Algorithm
• Robot Experiments
• Model-based Reinforcement Learning
• Pre-defined Threshold
• Deep Reinforcement Learning