Using bisimulation for policy transfer in MDPs

Knowledge transfer is a powerful approach to solve Markov Decision Processes. In this paper, we present approaches that use bisimulation-style metrics (Ferns, Panangaden & Precup, 2004) to compute the similarity of states in a large problem to states in smaller problems, which might have already been solved. We propose algorithms that decide what actions to transfer form the small to the large problem, given this information. We also show that this approach can be used, even more successfully, when using temporally extended actions (Sutton, Precup & Singh, 1999). We present theoretical guarantees on the quality of the transferred policy, as well as promising empirical results.

Authors

• Pablo Castro McGill University
• Doina Precup McGill University Mila

Keywords

• Markov Decision Processes
• Bisimulation
• Policy transfer