Efficient Learning of Mixed Observable Predictive State Representations

A key to successful reinforcement learning and planning in partially observable domains is a well built representation with a succinct state information. Recently some progress has been made on learning such representations within the framework of PSRs, specifically applying spectral learning approach. These algorithms guarantee to learn a nearly exact model given enough data, while at the same time keeping the state representation size within some well defined bounds. Nevertheless, in many realistic domains these bounds could be prohibitive from the point of view of RL algorithms, requiring domain specific knowledge and problem structure to make further reduction in the size of the state space. In this work we consider a specific problem structure, termed mixed observability. As opposed to partial observability, some of the observed variables are assumed to be Markovian, resulting in a more compact state representation. Mixed observability setting was found useful in domains as diverse as robotics, compu- tational sustainability and operations research. Motivated by its broad applicability, in this work we develop a PSR–based spectral learning algorithm that leverages this structural assumption. Beyond providing a more compact state representation, the proposed algorithm is faster and more data efficient as compared to the existing spectral learning methods for PSRs. These advantages are supported by theoretical as well as ex- perimental results.

Authors

• Sylvie Ong McGill University
• Yuri Grinberg
• Joelle Pineau

Keywords

No keywords are indexed for this paper.