Bayesian exploration and interactive demonstration in continuous state MAXQ-learning

Deploying robots for service tasks requires learning algorithms that scale to the combinatorial complexity of our daily environment. Inspired by the way humans decompose complex tasks, hierarchical methods for robot learning have attracted significant interest. In this paper, we apply the MAXQ method for hierarchical reinforcement learning to continuous state spaces. By using Gaussian Process Regression for MAXQ value function decomposition, we obtain probabilistic estimates of primitive and completion values for every subtask within the MAXQ hierarchy. From these, we recursively compute probabilistic estimates of state-action values. Based on the expected deviation of these estimates, we devise a Bayesian exploration strategy that balances optimization of expected values and risk from exploring unknown actions. To further reduce risk and to accelerate learning, we complement MAXQ with learning from demonstrations in an interactive way. In every situation and subtask, the system may ask for a demonstration if there is not enough knowledge available to determine a safe action for exploration. We demonstrate the ability of the proposed system to efficiently learn solutions to complex tasks on a box stacking scenario.

Authors

• Kathrin Gräve
• Sven Behnke

Keywords

• Learning (artificial intelligence)
• Robots
• Bayes methods
• Optimization
• Uncertainty
• Approximation methods
• Learning systems
• Continuous State
• Learning Algorithms
• Value Function
• Estimated Values
• State Space
• Gaussian Process
• Kriging
• Efficient Learning
• Hierarchical Method
• Exploration Strategy
• Inverse Reinforcement Learning
• Continuous State Space
• Robot Learning
• Hierarchical Structure
• Actual Values
• Intermediate Levels
• Expert Knowledge
• Workspace
• Level Of Abstraction
• Single Activity
• Imitation Learning
• Levels Of Hierarchy
• Top Level
• Motion Primitives
• Number Of Boxes
• Human Experts
• State-action Pair
• Large State Space
• Negative Reward
• Ways Of Learning