Safe Hierarchical Policy Optimization using Constrained Return Variance in Options

The standard setting in reinforcement learning (RL) to maximize the mean return does not as- sure a reliable and repeatable behavior of an agent in safety-critical applications like autonomous driving, robotics, and so forth. Often, penalization of the objective function with the variance in return is used to limit the unexpected behavior of the agent shown in the environment. While learning the end-to-end options have been accomplished, in this work, we introduce a novel Bellman style direct approach to estimate the variance in return in hierarchical policies using the option-critic architecture (Bacon et al. (2017)). The penalization of the mean return with the variance enables learning safer trajectories, which avoids inconsis- tently behaving regions. Here, we present the derivation in the policy gradient style method with the new safe objective function which would provide the updates for the option parameters in an online fashion.

Authors

• Arushi Jain
• Doina Precup Mila – Quebec Artificial Intelligence Institute & McGill University

Keywords

No keywords are indexed for this paper.