Author name cluster

Travis Mandel

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

7 papers

1 author row

AAAI Conference 2021 Conference Paper

AI-Assisted Scientific Data Collection with Iterative Human Feedback

Travis Mandel
James Boyd
Sebastian J. Carter
Randall H. Tanaka
Taishi Nammoto

Although artificial intelligence has revolutionized data analysis, significantly less work has focused on using AI to improve scientific data collection. Past work in AI for data collection has typically assumed the objective function is welldefined by humans before starting an experiment; however, this is a poor fit for scientific domains where new discoveries and insights are made as data is being collected. In this paper we present a new framework to allow AI systems to work together with humans (e. g. scientists) to collect data more effectively in simple scientific domains. We present a novel algorithm, TESA, which seeks to achieve good performance by learning from past human behavior how to direct data to places that are likely to become scientifically interesting in the future. We analyze the problem theoretically, defining a novel notion of regret in this setting and showing that TESA is zero regret. Next, we show that TESA outperforms other related algorithms in simulations using real data drawn from three diverse domains (economics, mental health, and cognitive psychology). Finally, we run experiments with human subjects across these scientific domains to compare our iterative human-in-the-loop process to a (more standard) workflow in which information is communicated to the AI a priori.

PDF Details

AAAI Conference 2017 Conference Paper

Where to Add Actions in Human-in-the-Loop Reinforcement Learning

Travis Mandel
Yun-En Liu
Emma Brunskill
Zoran Popovi_

In order for reinforcement learning systems to learn quickly in vast action spaces such as the space of all possible pieces of text or the space of all images, leveraging human intuition and creativity is key. However, a human-designed action space is likely to be initially imperfect and limited; furthermore, humans may improve at creating useful actions with practice or new information. Therefore, we propose a framework in which a human adds actions to a reinforcement learning system over time to boost performance. In this setting, however, it is key that we use human effort as efﬁciently as possible, and one signiﬁcant danger is that humans waste effort adding actions at places (states) that aren’t very important. Therefore, we propose Expected Local Improvement (ELI), an automated method which selects states at which to query humans for a new action. We evaluate ELI on a variety of simulated domains adapted from the literature, including domains with over a million actions and domains where the simulated experts change over time. We ﬁnd ELI demonstrates excellent empirical performance, even in settings where the synthetic “experts” are quite poor.

PDF Details

IJCAI Conference 2016 Conference Paper

Efficient Bayesian Clustering for Reinforcement Learning

Travis Mandel
Yun-En Liu
Emma Brunskill
Zoran Popovic

A fundamental artificial intelligence challenge is how to design agents that intelligently trade off exploration and exploitation while quickly learning about an unknown environment. However, in order to learn quickly, we must somehow generalize experience across states. One promising approach is to use Bayesian methods to simultaneously cluster dynamics and control exploration; unfortunately, these methods tend to require computationally intensive MCMC approximation techniques which lack guarantees. We propose Thompson Clustering for Reinforcement Learning (TCRL), a family of Bayesian clustering algorithms for reinforcement learning that leverage structure in the state space to remain computationally efficient while controlling both exploration and generalization. TCRL-Theoretic achieves near-optimal Bayesian regret bounds while consistently improving over a standard Bayesian exploration approach. TCRL-Relaxed is guaranteed to converge to acting optimally, and empirically outperforms state-of-the-art Bayesian clustering algorithms across a variety of simulated domains, even in cases where no states are similar.

PDF Details

AAAI Conference 2016 Conference Paper

Offline Evaluation of Online Reinforcement Learning Algorithms

Travis Mandel
Yun-En Liu
Emma Brunskill
Zoran Popović

In many real-world reinforcement learning problems, we have access to an existing dataset and would like to use it to evaluate various learning approaches. Typically, one would prefer not to deploy a ﬁxed policy, but rather an algorithm that learns to improve its behavior as it gains more experience. Therefore, we seek to evaluate how a proposed algorithm learns in our environment, meaning we need to evaluate how an algorithm would have gathered experience if it were run online. In this work, we develop three new evaluation approaches which guarantee that, given some history, algorithms are fed samples from the distribution that they would have encountered if they were run online. Additionally, we are the ﬁrst to propose an approach that is provably unbiased given ﬁnite data, eliminating bias due to the length of the evaluation. Finally, we compare the sample-efﬁciency of these approaches on multiple datasets, including one from a real-world deployment of an educational game.

PDF Details

RLDM Conference 2015 Conference Abstract

Nonstationary Evaluation for Reinforcement Learning

Travis Mandel
Yun-En Liu
Emma Brunskill
Zoran Popović

In many real-world reinforcement learning problems, we have access to an existing dataset and would like to use it to evaluate various decision making approaches. Typically one uses offline policy evaluation techniques, where the goal is to evaluate how a fixed policy would perform using the available data. However, one rarely deploys a fixed policy, but instead deploys an algorithm that learns to improve its behavior as it gains experience. Therefore, we seek to evaluate how a proposed algorithm learns in our envi- ronment, or in other words, evaluate a policy that changes over time in response to data, a problem known as nonstationary evaluation. This problem has received significant attention in the bandit and contextual bandit frameworks, however no unbiased nonstationary estimators have been proposed for the more general case of reinforcement learning. In this work, we develop two new unbiased nonstationary evaluation approaches for reinforcement learning, discuss their trade-offs, and compare their data-efficiency on a real educational game dataset.

PDF Details

AAAI Conference 2015 Conference Paper

The Queue Method: Handling Delay, Heuristics, Prior Data, and Evaluation in Bandits

Travis Mandel
Yun-En Liu
Emma Brunskill
Zoran Popović

Current algorithms for the standard multi-armed bandit problem have good empirical performance and optimal regret bounds. However, real-world problems often differ from the standard formulation in several ways. First, feedback may be delayed instead of arriving immediately. Second, the real world often contains structure which suggests heuristics, which we wish to incorporate while retaining strong theoretical guarantees. Third, we may wish to make use of an arbitrary prior dataset without negatively impacting performance. Fourth, we may wish to efficiently evaluate algorithms using a previously collected dataset. Surprisingly, these seemingly-disparate problems can be addressed using algorithms inspired by a recently-developed queueing technique. We present the Stochastic Delayed Bandits (SDB) algorithm as a solution to these four problems, which takes black-box bandit algorithms (including heuristic approaches) as input while achieving good theoretical guarantees. We present empirical results from both synthetic simulations and real-world data drawn from an educational game. Our results show that SDB outperforms state-of-the-art approaches to handling delay, heuristics, prior data, and evaluation.

PDF Details

IJCAI Conference 2015 Conference Paper

Towards More Practical Reinforcement Learning

Travis Mandel

Reinforcement Learning is beginning to be applied outside traditional domains such as robotics, and into human-centric domains such as healthcare and education. In these domains, two problems are critical to address: We must be able to evaluate algorithms with a collection of prior data if one is available, and we must devise algorithms that carefully trade off exploration and exploitation in such a way that they are guaranteed to converge to optimal behavior quickly, while retaining very good performance with limited data. In this thesis, I examine these two problems, with an eye towards applications to educational games.

PDF Details