AAAI Conference 1999 Conference Paper
Efficient Exploration for Optimizing Immediate Reward
- Dale Schuurmans
- University of Waterloo
- Lloyd Greenwald
- Drexel University
Weconsider the problemof learning an effective behavior strategy from reward. Althoughmuchstudied, the issue of howto use prior knowledgeto scale optimal behavior learning up to real-world problems remains an important open issue. Weinvestigate the inherent data-complexity of behavior-learning whenthe goal is simply to optimize immediate reward. Although easier than reinforcement learning, whereone must also cope with state dynamics, immediate rewardlearning is still a common problemand is fundamentallyharder than supervised learning. For optimizing immediatereward, prior knowledgecan be expressed either as a bias on the space of possible reward models, or a bias on the space of possible controllers. Weinvestigate the two paradigmatic learning approachesof indirect (reward-model)learning anddirect-control learning, and showthat neither uniformlydominatesthe other in general. Model-based learning has the advantage of generalizing reward experiences across states and actions, but direct-control learning has the advantage of focusing only on potentially optimal actions and avoidinglearning irrelevant worlddetails. Bothstrategies can be strongly advantageousin different circumstances. Weintroduce hybrid learning strategies that combinethe benefits of both approaches, and uniformlyimprovetheir learning efficiency.