When is Offline Policy Selection Sample Efficient for Reinforcement Learning?

Offline reinforcement learning algorithms often require careful hyperparametertuning. Beforedeployment, weneedtoselectamongst a set of candidate policies. However, there is limited understanding about the fundamental limits of this offline policy selection (OPS) problem. In this work we provide clarity on when sample efficient OPS is possible, primarily by connecting OPS to off-policy policy evaluation (OPE) and Bellman error (BE) estimation. We first show a hardness result, that in the worst case, OPS is just as hard as OPE, by proving a reduction of OPE to OPS. As a result, no OPS method can be more sample efficient than OPE in the worst case. We then connect BE estimation to the OPS problem, showing how BEcanbeusedasatoolforOPS. WhileBE-basedmethodsgenerally require stronger requirements than OPE, when those conditions are met they can be more sample efficient. Building on this insight, we propose a BE method for OPS, called Identifiable BE Selection (IBES), that has a straightforward method for selecting its own hyperparameters. We conclude with an empirical study comparing OPE and IBES, and by showing the difficulty of OPS on an offline Atari benchmark dataset.

Authors

• Vincent Liu RBC Borealis
• Prabhat Nagarajan University of Alberta
• Andrew Patterson Alberta Machine Intelligence Institute
• Martha White University of Alberta

Keywords

• Offline reinforcement learning
• off-policy policy evaluation