TD-DeltaPi: A Model-Free Algorithm for Efficient Exploration

We study the problem of finding efficient exploration policies for the case in which an agent is momentarily not concerned with exploiting, and instead tries to compute a policy for later use. We first formally define the Optimal Exploration Problem as one of sequential sampling and show that its solutions correspond to paths of minimum expected length in the space of policies. We derive a model-free, local linear approximation to such solutions and use it to construct efficient exploration policies. We compare our model-free approach to other exploration techniques, including one with the best known PAC bounds, and show that ours is both based on a well-defined optimization problem and empirically efficient.

Authors

• Bruno da Silva University of Massachusetts Amherst
• Andrew Barto University of Massachusetts Amherst

Keywords

No keywords are indexed for this paper.