Applying Online Search Techniques to Continuous-State Reinforcement Learning

In this paper, wedescribemethods for efficiently computingbetter solutions to control problems in continuousstate spaces. Weprovidealgorithmsthat exploit online search to boost the powerof very approximate valuefunctionsdiscoveredby traditional reinforcement learning techniques. Weexaminelocal searches, where the agentperformsa finite-depth lookahead search, and global searches, wherethe agent performsa searchfor a trajectory all the wayfromthe current state to a goal state. Thekey to the successof the local methods lies in taking a value function, whichgives a roughsolution to the hard problem of finding goodtrajectories fromevery single state, andcombining that withonline search, whichthen gives an accurate solution to the easier problem of finding a goodtrajectory specifically from the current state. Thekey to the success of the global methodslies in using aggressivestate-space search techniquessuchas uniform-cost search and A*, tamedinto a tractable formby exploiting neighborhood relations and trajectory constraints that arise fromcontinuous-space dynamiccontrol.

Authors

• Scott Davies Andrew Y. Ng

Keywords

No keywords are indexed for this paper.