Author name cluster

Craig Sherstan

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

9 papers

2 author rows

NeurIPS Conference 2022 Conference Paper

Value Function Decomposition for Iterative Design of Reinforcement Learning Agents

James MacGlashan
Evan Archer
Alisa Devlic
Takuma Seno
Craig Sherstan
Peter Wurman
Peter Stone

Designing reinforcement learning (RL) agents is typically a difficult process that requires numerous design iterations. Learning can fail for a multitude of reasons and standard RL methods provide too few tools to provide insight into the exact cause. In this paper, we show how to integrate \textit{value decomposition} into a broad class of actor-critic algorithms and use it to assist in the iterative agent-design process. Value decomposition separates a reward function into distinct components and learns value estimates for each. These value estimates provide insight into an agent's learning and decision-making process and enable new training methods to mitigate common problems. As a demonstration, we introduce SAC-D, a variant of soft actor-critic (SAC) adapted for value decomposition. SAC-D maintains similar performance to SAC, while learning a larger set of value predictions. We also introduce decomposition-based tools that exploit this information, including a new reward \textit{influence} metric, which measures each reward component's effect on agent decision-making. Using these tools, we provide several demonstrations of decomposition's use in identifying and addressing problems in the design of both environments and agents. Value decomposition is broadly applicable and easy to incorporate into existing algorithms and workflows, making it a powerful tool in an RL practitioner's toolbox.

PDF Details

AAAI Conference 2020 Conference Paper

Gamma-Nets: Generalizing Value Estimation over Timescale

Craig Sherstan
Shibhansh Dohare
James MacGlashan
Johannes Günther
Patrick M. Pilarski

Temporal abstraction is a key requirement for agents making decisions over long time horizons—a fundamental challenge in reinforcement learning. There are many reasons why value estimates at multiple timescales might be useful; recent work has shown that value estimates at different time scales can be the basis for creating more advanced discounting functions and for driving representation learning. Further, predictions at many different timescales serve to broaden an agent's model of its environment. One predictive approach of interest within an online learning setting is general value function (GVFs), which represent models of an agent's world as a collection of predictive questions each defined by a policy, a signal to be predicted, and a prediction timescale. In this paper we present Γ-nets, a method for generalizing value function estimation over timescale, allowing a given GVF to be trained and queried for arbitrary timescales so as to greatly increase the predictive ability and scalability of a GVF-based model. The key to our approach is to use timescale as one of the value estimator's inputs. As a result, the prediction target for any timescale is available at every timestep and we are free to train on any number of timescales. We first provide two demonstrations by 1) predicting a square wave and 2) predicting sensorimotor signals on a robot arm using a linear function approximator. Next, we empirically evaluate Γ-nets in the deep reinforcement learning setting using policy evaluation on a set of Atari video games. Our results show that Γ-nets can be effective for predicting arbitrary timescales, with only a small cost in accuracy as compared to learning estimators for fixed timescales. Γ-nets provide a method for accurately and compactly making predictions at many timescales without requiring a priori knowledge of the task, making it a valuable contribution to ongoing work on model-based planning, representation learning, and lifelong learning algorithms.

PDF Details

RLDM Conference 2019 Conference Abstract

Gamma-nets: Generalizing Value Functions over Timescale

Craig Sherstan
James MacGlashan
Patrick M. Pilarski

Predictive representations of state connect an agent’s behavior (policy) to observable outcomes, providing a powerful representation for decision making. General value functions (GVFs) represent models of an agent’s world as a collection of predictive questions. A GVF is expressed by: a policy, a prediction target, and a timescale, e. g. , “If a robot drives forward how much current will its motors draw over the next 3s? ” Traditionally, predictions for a given timescale must be specified by the engineer and predictions for each timescale learned independently. Here we present γ-nets, a method for generalizing value function estimation over timescale, allowing a given GVF to be trained and queried for any fixed timescale. The key to our approach is to use timescale as one of the estimator inputs. The prediction target for any fixed timescale is then available at every timestep and we are free to train on any number of timescales. We present preliminary results on a test signal and a robot arm. This work contributes new insights into creating expressive and tractable predictive models for decision-making agents that operate in real-time, long-lived environments.

PDF Details

RLDM Conference 2019 Conference Abstract

GVFs: General Value Freebies

Johannes Guenther
Alex K Kearney
Craig Sherstan
Michael Dawson
M. Pilarski

Machine learning offers the ability for machines to learn from data and improve their perfor- mance on a given task. The data used in learning is usually provided either in terms of a predesigned data set or as sampled through interaction with the environment. However, there is another oft-forgotten source of data available for machines to learn from: the learning process itself. As algorithms learn from data and interact with their environment, learning mechanisms produce a continuous stream of data in terms of errors, parameters changes, updates and estimates. These signals have great potential for use in learning and decision making. In this paper, we investigate the utility of such freebie signals that are produced either as the output of learning or due to the act of learning, i. e. , updates to weights and learning rates. Specifi- cally, we implement a prediction learner that models its environment via multiple General Value Functions (GVFs) and deploy it within a robotic setting. The first signal of interest that we study is one known as the Unexpected Demon Error (UDE), which is closely related to the Temporal-Difference (TD) error and can be tied to the notion of surprise. Detecting surprise reveals important information not only about the learning process but also about the environment and the functioning of the agent within its environment. The second type of signal that we investigate is the agent’s learning step size. For this purpose, a vectorized step-size adaptation algorithm is used to update the step sizes over the course of learning. Observing the step-size distribution over time appears to allow a system to automatically detect and characterise common sensor failures in the physical system. We suggest that by adding introspective signals such as UDE and step sizes analysis to the available data, autonomous and long-lived agents can become better aware of their interactions with the environment, resulting in a superior ability to make decisions.

PDF Details

IROS Conference 2018 Conference Paper

Accelerating Learning in Constructive Predictive Frameworks with the Successor Representation

Craig Sherstan
Marlos C. Machado
Patrick M. Pilarski

We propose using the Successor Representation (SR) to accelerate learning in a constructive knowledge system based on General Value Functions (GVFs). In real-world settings, like robotics for unstructured and dynamic environments, it is impossible to model all meaningful aspects of a system and its environment by hand. Instead, robots must learn and adapt to changes in their environment and task, incrementally constructing models from their own experience. GVFs, taken from the field of reinforcement learning (RL), are a way of modeling the world as predictive questions. One approach to such models proposes a massive network of interconnected and interdependent GVFs, which are incrementally added over time. It is reasonable to expect that new, incrementally added predictions can be learned more swiftly if the learning process leverages knowledge gained from past experience. The SR provides a means of capturing regularities that can be reused across multiple GVFs by separating the dynamics of the world from the prediction targets. As a primary contribution of this work, we show that using the SR can improve sample efficiency and learning speed of GVFs in a continual learning setting where new predictions are incrementally added and learned over time. We analyze our approach in a grid-world and then demonstrate its potential on data from a physical robot arm.

Details

UAI Conference 2018 Conference Paper

Comparing Direct and Indirect Temporal-Difference Methods for Estimating the Variance of the Return

Craig Sherstan
Dylan R. Ashley
Brendan Bennett
Kenny Young
Adam White 0001
Martha White
Richard S. Sutton

Temporal-difference (TD) learning methods are widely used in reinforcement learning to estimate the expected return for each state, without a model, because of their significant advantages in computational and data efficiency. For many applications involving risk mitigation, it would also be useful to estimate the variance of the return by TD methods. In this paper, we describe a way of doing this that is substantially simpler than those proposed by Tamar, Di Castro, and Mannor in 2012, or those proposed by White and White in 2016. We show that two TD learners operating in series can learn expectation and variance estimates. The trick is to use the square of the TD error of the expectation learner as the reward of the variance learner, and the square of the expectation learner’s discount rate as the discount rate of the variance learner. With these two modifications, the variance learning problem becomes a conventional TD learning problem to which standard theoretical results can be applied. Our formal results are limited to the table lookup case, for which our method is still novel, but the extension to function approximation is immediate, and we provide some empirical results for the linear function approximation case. Our experimental results show that our direct method behaves just as well as a comparable indirect method, but is generally more robust.

Details

RLDM Conference 2017 Conference Abstract

Confident Decision Making with General Value Functions

Craig Sherstan
Patrick Pilarski

This work builds on the idea of encoding knowledge as temporally extended predictions through the use of general value functions. Prior work has focused on learning predictions about externally derived signals about a task or environment (e. g. a robot’s battery level or joint positions). Here we advocate that an agent should also predict internally generated signals regarding its own learning process—for example, an agent’s confidence in its learned predictions. We suggest how such information would be beneficial in creating an introspective agent that is able to learn to make good decisions in a complex, changing world. Finally, we provide two experimental examples, one simulated and one robotic, showing multiple confidence measures that suggest how this principal might be used in practice.

PDF Details

RLDM Conference 2017 Conference Abstract

Direct Estimation of the Variance of the Return with Temporal-Difference Methods

Craig Sherstan

Using predictive confidence—measures of trust in the outcome of a prediction—are very benefi- cial in decision making. While temporal-difference (TD) methods are commonly used to estimate reward in reinforcement learning, they provide no inherent measure of confidence. Here we present a simple method for directly estimating the variance of the return—the sum of future rewards—by using a network of two temporal-difference nodes. This is accomplished by using the squared TD error of the value estimator (the first moment of the return) as the meta-reward for a second TD node. Other extant methods have estimated the variance indirectly by estimating the second moment of the return. The direct method introduced here, which is arguably simpler, has been empirically evaluated in the tabular setting and found to work just as well as the indirect methods, but is in general more stable. This extended abstract represents a small portion of a larger work currently in preparation.

PDF Details

RLDM Conference 2015 Conference Abstract

Progress Toward the Shared Control of a Prosthetic Arm

Ann Edwards
Michael Dawson
Jacqueline Hebert
Craig Sherstan
Richard Sutton
K Chan
Patrick Pilarski

State-of-the-art myoelectric prostheses typically have a greater number of functions than the pos- sible number of control signals, requiring amputees to manually switch through a fixed list of options to select the desired function. For this reason, the control of powered prosthetic arms is often considered com- plex and non-intuitive. Previous studies have demonstrated that techniques from reinforcement learning, and in particular General Value Functions (GVFs), can be applied to develop temporally extended predic- tions about signals related to prosthetic arm movement. In particular, we have shown that it is possible to learn and maintain predictions about which joint of a robotic arm a user intends to use next, and use this information to create and update an adaptive switching list. In this work, we extend previous studies by demonstrating the real-time use of adaptive switching by an amputee in a simple control task with a myo- electric arm. We also present results from a non-amputee subject controlling a myoelectric arm in a more complex task, providing evidence for the scalability of the learning system. Our results suggest that, com- pared with a fixed-list switching method, adaptive switching can significantly decrease the amount of time and the number of switches required for the control of a robotic arm and potentially reduce the cognitive burden on myoelectric arm users. Furthermore, we anticipate the future blending of human and machine decision making for the shared control of a robotic arm. Given high enough prediction certainty, a robotic arm could begin to switch autonomously between joints, reducing the time and effort required by amputees to activate complex prosthetic motions.

PDF Details