Gamma-nets: Generalizing Value Functions over Timescale

Predictive representations of state connect an agent’s behavior (policy) to observable outcomes, providing a powerful representation for decision making. General value functions (GVFs) represent models of an agent’s world as a collection of predictive questions. A GVF is expressed by: a policy, a prediction target, and a timescale, e. g. , “If a robot drives forward how much current will its motors draw over the next 3s? ” Traditionally, predictions for a given timescale must be specified by the engineer and predictions for each timescale learned independently. Here we present γ-nets, a method for generalizing value function estimation over timescale, allowing a given GVF to be trained and queried for any fixed timescale. The key to our approach is to use timescale as one of the estimator inputs. The prediction target for any fixed timescale is then available at every timestep and we are free to train on any number of timescales. We present preliminary results on a test signal and a robot arm. This work contributes new insights into creating expressive and tractable predictive models for decision-making agents that operate in real-time, long-lived environments.

Authors

• Craig Sherstan
• James MacGlashan
• Patrick M. Pilarski

Keywords

No keywords are indexed for this paper.