Basis Function Construction for Hierarchical Reinforcement Learning

Much past work on solving Markov decision processes (MDPs) using reinforcement learning (RL) has relied on combining parameter estimation methods with hand-designed function approximationarchitectures for representing value functions. Recently, there hasbeen growing interest in a broader framework that combines representation discovery and control learning, where value functions areapproximated using a linear combination of task-dependent basisfunctions learned during the course of solving a particular MDP. This paper introduces an approach to automatic basis function construction for hierarchical reinforcement learning (HRL). Our approach generalizes past work on basis construction to multi-levelaction hierarchies by forming a compressed representation of asemi-Markov decision process (SMDP) at multiple levels of temporal abstraction. The specific approach is based on hierarchicalspectral analysis of graphs induced on an SMDP's state space fromsample trajectories. We present experimental results on benchmarkSMDPs, showing significant speedups when compared to hand-designed approximation architectures.

Authors

• Sarah Osentoski Brown University
• Sridhar Mahadevan Adobe Research

Keywords

• Representation Discovery
• Hierarchical Reinforcement Learning
• Semi-Markov Decision Processes