Philippe Hamel

ICML Conference 2019 Conference Paper

Per-Decision Option Discounting

• Anna Harutyunyan
• Peter Vrancx
• Philippe Hamel
• Ann Nowé
• Doina Precup

In order to solve complex problems an agent must be able to reason over a sufficiently long horizon. Temporal abstraction, commonly modeled through options, offers the ability to reason at many timescales, but the horizon length is still determined by the discount factor of the underlying Markov Decision Process. We propose a modification to the options framework that naturally scales the agent’s horizon with option length. We show that the proposed option-step discount controls a bias-variance trade-off, with larger discounts (counter-intuitively) leading to less estimation variance.

RLDM Conference 2019 Conference Abstract

Per-Decision Option Discounting

• Anna Harutyunyan
• Peter Vrancx
• Philippe Hamel
• Ann Nowe
• Doina Precup

In order to solve complex problems an agent must be able to reason over a sufficiently long horizon. Temporal abstraction, commonly modeled through options, offers the ability to reason at many timescales, but the horizon length is still determined by the discount factor of the underlying Markov Deci- sion Process. We propose a modification to the options framework that allows the agent’s horizon to grow naturally as its actions become more complex and extended in time. We show that the proposed option- step discount controls a bias-variance trade-off, with larger discounts (counter-intuitively) leading to less estimation variance.

NeurIPS Conference 2019 Conference Paper

The Option Keyboard: Combining Skills in Reinforcement Learning

• Andre Barreto
• Diana Borsa
• Shaobo Hou
• Gheorghe Comanici
• Eser Aygün
• Philippe Hamel
• Daniel Toyama
• Jonathan hunt

The ability to combine known skills to create new ones may be crucial in the solution of complex reinforcement learning problems that unfold over extended periods. We argue that a robust way of combining skills is to define and manipulate them in the space of pseudo-rewards (or "cumulants"). Based on this premise, we propose a framework for combining skills using the formalism of options. We show that every deterministic option can be unambiguously represented as a cumulant defined in an extended domain. Building on this insight and on previous results on transfer learning, we show how to approximate options whose cumulants are linear combinations of the cumulants of known options. This means that, once we have learned options associated with a set of cumulants, we can instantaneously synthesise options induced by any linear combination of them, without any learning involved. We describe how this framework provides a hierarchical interface to the environment whose abstract actions correspond to combinations of basic skills. We demonstrate the practical benefits of our approach in a resource management problem and a navigation task involving a quadrupedal simulated robot.

ICML Conference 2009 Conference Paper

Workshop summary: Sparse methods for music audio

• Douglas Eck
• Dan Ellis
• Philippe Hamel