Using Training Regimens to Teach Expanding Function Approximators

In complex real-world environments, traditional (tabular)techniques for solving Reinforcement Learning (RL) do notscale. Function approximation is needed, but unfortunately, existing approaches generally have poor convergence and optimality guarantees. Additionally, for the case of humanenvironments, it is valuable to be able to leverage humaninput. In this paper we introduce Expanding Value Function Approximation (EVFA), a function approximation algorithm that returns the optimal value function given sufficient rounds. To leverage human input, we introduce a newhuman-agent interaction scheme, training regimens, whichallow humans to interact with and improve agent learning inthe setting of a machine learning game. In experiments, weshow EVFA compares favorably to standard value approximation approaches. We also show that training regimensenable humans to further improve EVFA performance. Inour user study, we find that non-experts are able to provideeffective regimens and that they found the game fun.

Authors

• Peng Zang
• Arya Irani
• Peng Zhou Anhui University
• Andrea Thomaz Georgia Institute of Technology
• Charles Isbell

Keywords

• Learning
• Human-robot/agent interaction
• Agents in games and virtual environments