Regularizing Reinforcement Learning with State Abstraction

State abstraction in a discrete reinforcement learning setting clusters states sharing a similar optimal action to yield an easier to solve decision process. In this paper, we generalize the concept of state abstraction to continuous action reinforcement learning by defining an abstract state as a state cluster over which a near-optimal policy of simple shape exists. We propose a hierarchical reinforcement learning algorithm that is able to simultaneously find the state space clustering and the optimal sub-policies in each cluster. The main advantage of the proposed framework is to provide a straightforward way of regularizing reinforcement learning by controlling the behavioral complexity of the learned policy. We apply our algorithm on several benchmark tasks and a robot tactile manipulation task and show that we can match state-of-the-art deep reinforcement learning performance by combining a small number of linear policies.

Authors

• Riad Akrour
• Filipe Veiga
• Jan Peters 0001
• Gerhard Neumann

Keywords

• Reinforcement learning
• Complexity theory
• Convergence
• Shape
• Clustering algorithms
• Task analysis
• Partitioning algorithms
• State Space
• Continuous Action
• Manipulation Tasks
• Deep Reinforcement Learning
• Reinforcement Learning Algorithm
• Number Of Policies
• Complex Policy
• Abstract States
• Continuous Reinforcement
• Neural Network
• Discretion
• Value Function
• Hierarchical Structure
• Transition Probabilities
• Optimal Policy
• Sampling Efficiency
• Markov Decision Process
• Spatial Partitioning
• Discrete Action
• Premature Convergence
• Proximal Policy Optimization
• Dexterous Manipulation
• Policy Update
• Stochastic Policy
• Deep Reinforcement Learning Algorithm
• Reinforcement Learning Framework