Approximate Policy Iteration with Demonstration Data

We propose an algorithm to solve uncertain sequential decision-making problems that utilizes two different types of data sources. The first is the data available in the conventional reinforcement learning setup: an agent interacts with the environment and receives a sequence of state transition samples alongside the corresponding reward signal. The second data source, which differentiates the setup of this work from the usual reinforcement learning framework, is in the form of expert’s demonstrations, that is, a set of states with the expert’s suggested actions. Benefitting from both sources of data, which are available in many real-world application domains, allows the agent to perform well even with few data points. The algorithm is couched in the framework of Approxi- mate Policy Iteration. Its approximate policy evaluation step is formulated as a convex optimization problem in which the expert demonstration data act as a set of linear constraints. In a real robotic navigation task, we show that the algorithm outperforms both pure approximate policy iteration and supervised learning.

Authors

• Beomjoon Kim
• Amir-massoud Farahmand
• Joelle Pineau
• Doina Precup Mila – Quebec Artificial Intelligence Institute & McGill University

Keywords

No keywords are indexed for this paper.