David L. Roberts

AAMAS Conference 2026 Conference Paper

General Flexible f -divergence for Challenging Offline RL Datasets with Low Stochasticity and Diverse Behavior Policies

• Jianxun Wang
• Grant C. Forbes
• Leonardo Villalobos-Arias
• David L. Roberts

Offline RL algorithms aim to improve upon the behavior policy that produces the collected data while constraining the learned policy to be within the support of the dataset. However, practical offline datasetsoftencontainexampleswithlittlediversityorlimitedexploration of the environment, and from multiple behavior policies with diverse expertise levels. Limited exploration can impair the offline RL algorithm’s ability to estimate Q or V values, while constraining towards diverse behavior policies can be overly conservative. Such datasets call for a balance between the RL objective and behavior policy constraints. We first identify the connection between 𝑓 -divergence and optimization constraint on the Bellman residual through a more general Linear Programming form for RL and the convex conjugate. Following this, we introduce the general flexible function formulation for the 𝑓 -divergence to incorporate an adaptive constraint on algorithms’ learning objectives based on the offline training dataset. Results from experiments on the MuJoCo, Fetch, and AdroitHand environments show the correctness of the proposed LP form and the potential of the flexible 𝑓 -divergence in improving performance for learning from a challenging dataset when applied to a compatible constrained optimization algorithm.

AAMAS Conference 2026 Conference Paper

Preventing Process Reward Model Hacking When Training Large Language Models on Verifiable Rewards

• Grant C. Forbes
• Leonardo Villalobos-Arias
• Jianxun Wang
• Arnav Jhala
• David L. Roberts

Alignment of Large Language Models to human preferences is an active and important field of study. Recent work in Reinforcement Learning with Verifiable Rewards (RLVR) aims to bypass the need for costly and imprecise human preference reward data by training LLMs specifically in domains wherein a simple, known solution exists. As the RLVR signal is sparse, however, it is often supplemented with a Process Reward Model (PRM) reward, which provides a dense reward for each token, or step in the chain of thought, for an agent to learn from. Using the VersaPRM extension to the MMLU-Pro dataset, we demonstrate that PRMs are susceptible to reward hacking behavior, wherein the model is incentivized to produce particularly long, plausible-seeming chains of thought that do not result in the correct response. We also develop a theoretical framework and a suite of methods for preventing this reward hacking while still utilizing PRMs effectively, based on recent work in potential-based and optimality-preserving reward shaping. We both prove theoretically and demonstrate practically that these methods prevent PRMs from altering the optimal policy, and thus from being optimized at the expense of the RLVR signal.

AAMAS Conference 2024 Conference Paper

Potential-Based Reward Shaping for Intrinsic Motivation

• Grant C. Forbes
• Nitish Gupta
• Leonardo Villalobos-Arias
• Colin M. Potts
• Arnav Jhala
• David L. Roberts

Recently there has been a proliferation of intrinsic motivation (IM) reward-shaping methods to learn in complex and sparse-reward environments. These methods can often inadvertently change the set of optimal policies in an environment, leading to suboptimal behavior. Previous work on mitigating the risks of reward shaping, particularly through potential-based reward shaping (PBRS), has not been applicable to many IM methods, as they are often complex, trainable functions themselves, and therefore dependent on a wider set of variables than the traditional reward functions that PBRS was developed for. We present an extension to PBRS that we prove preserves the set of optimal policies under a more general set of functions than has been previously proven. We also present Potential-Based Intrinsic Motivation (PBIM), a method for converting IM rewards into a potential-based form that is useable without altering the set of optimal policies. Testing in the Mini- Grid DoorKey and Cliff Walking environments, we demonstrate that PBIM successfully prevents the agent from converging to a suboptimal policy and can speed up training.

AAAI Conference 2024 Short Paper

Potential-Based Reward Shaping for Intrinsic Motivation (Student Abstract)

• Grant C. Forbes
• David L. Roberts

Recently there has been a proliferation of intrinsic motivation (IM) reward shaping methods to learn in complex and sparse-reward environments. These methods can often inadvertently change the set of optimal policies in an environment, leading to suboptimal behavior. Previous work on mitigating the risks of reward shaping, particularly through potential-based reward shaping (PBRS), has not been applicable to many IM methods, as they are often complex, trainable functions themselves, and therefore dependent on a wider set of variables than the traditional reward functions that PBRS was developed for. We present an extension to PBRS that we show preserves the set of optimal policies under a more general set of functions than has been previously demonstrated. We also present Potential-Based Intrinsic Motivation (PBIM), a method for converting IM rewards into a potential-based form that are useable without altering the set of optimal policies. Testing in the MiniGrid DoorKey environment, we demonstrate that PBIM successfully prevents the agent from converging to a suboptimal policy and can speed up training.

AAMAS Conference 2017 Conference Paper

Curriculum Design for Machine Learners in Sequential Decision Tasks

• Bei Peng
• James MacGlashan
• Robert Loftin
• Michael L. Littman
• David L. Roberts
• Matthew E. Taylor

Existing machine-learning work has shown that algorithms can benefit from curricula—learning first on simple examples before moving to more difficult examples. While most existing work on curriculum learning focuses on developing automatic methods to iteratively select training examples with increasing difficulty tailored to the current ability of the learner, relatively little attention has been paid to the ways in which humans design curricula. We argue that a better understanding of the human-designed curricula could give us insights into the development of new machinelearning algorithms and interfaces that can better accommodate machine- or human-created curricula. Our work addresses this emerging and vital area empirically, taking an important step to characterize the nature of human-designed curricula relative to the space of possible curricula and the performance benefits that may (or may not) occur.

AAMAS Conference 2016 Conference Paper

A Need for Speed: Adapting Agent Action Speed to Improve Task Learning from Non-Expert Humans

• Bei Peng
• James MacGlashan
• Robert Loftin
• Michael L. Littman
• David L. Roberts
• Matthew E. Taylor

As robots become pervasive in human environments, it is important to enable users to effectively convey new skills without programming. Most existing work on Interactive Reinforcement Learning focuses on interpreting and incorporating non-expert human feedback to speed up learning; we aim to design a better representation of the learning agent that is able to elicit more natural and effective communication between the human trainer and the learner, while treating human feedback as discrete communication that depends probabilistically on the trainer’s target policy. This work entails a user study where participants train a virtual agent to accomplish tasks by giving reward and/or punishment in a variety of simulated environments. We present results from 60 participants to show how a learner can ground natural language commands and adapt its action execution speed to learn more efficiently from human trainers. The agent’s action execution speed can be successfully modulated to encourage more explicit feedback from a human trainer in areas of the state space where there is high uncertainty. Our results show that our novel adaptive speed agent dominates different fixed speed agents on several measures of performance. Additionally, we investigate the impact of instructions on user performance and user preference in training conditions.

JAAMAS Journal 2015 Journal Article

Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning

• Robert Loftin
• Bei Peng
• David L. Roberts

Abstract For real-world applications, virtual agents must be able to learn new behaviors from non-technical users. Positive and negative feedback are an intuitive way to train new behaviors, and existing work has presented algorithms for learning from such feedback. That work, however, treats feedback as numeric reward to be maximized, and assumes that all trainers provide feedback in the same way. In this work, we show that users can provide feedback in many different ways, which we describe as “training strategies. ” Specifically, users may not always give explicit feedback in response to an action, and may be more likely to provide explicit reward than explicit punishment, or vice versa, such that the lack of feedback itself conveys information about the behavior. We present a probabilistic model of trainer feedback that describes how a trainer chooses to provide explicit reward and/or explicit punishment and, based on this model, develop two novel learning algorithms (SABL and I-SABL) which take trainer strategy into account, and can therefore learn from cases where no feedback is provided. Through online user studies we demonstrate that these algorithms can learn with less feedback than algorithms based on a numerical interpretation of feedback. Furthermore, we conduct an empirical analysis of the training strategies employed by users, and of factors that can affect their choice of strategy.

IS Journal 2014 Journal Article

Toward Cyber-Enhanced Working Dogs for Search and Rescue

• Alper Bozkurt
• David L. Roberts
• Barbara L. Sherman
• Rita Brugarolas
• Sean Mealin
• John Majikes
• Pu Yang
• Robert Loftin

The authors introduce the fundamental building blocks for a cyber-enabled, computer-mediated communication platform to connect human and canine intelligence to achieve a new generation of Cyber-Enhanced Working Dog (CEWD). The use of monitoring technologies provides handlers with real-time information about the behavior and emotional state of their CEWDs and the environments they're working in for a more intelligent canine-human collaboration. From handler to dog, haptic feedback and auditory cues are integrated to provide remote command and feedback delivery. From dog to handler, multiple inertial measurement units strategically located on a harness are used to accurately detect posture and behavior, and concurrent noninvasive photoplethysmogram and electrocardiogram for physiological monitoring. The authors also discuss how CEWDs would be incorporated with a variety of other robotic and autonomous technologies to create next-generation intelligent emergency response systems. Using cyber-physical systems to supplement and augment the two-way information exchange between human handlers and dogs would amplify the remarkable sensory capacities of search and rescue dogs and help them save more lives.

AAAI Conference 2008 Short Paper

Computational Influence for Training and Entertainment

• David L. Roberts

AAMAS Conference 2007 Conference Paper

A Globally Optimal Algorithm for TTD-MDPs

• Sooraj Bhat
• David L. Roberts
• Mark J. Nelson
• Charles L. Isbell
• Michael Mateas

In this paper, we discuss the use of Targeted Trajectory Distribution Markov Decision Processes (TTD-MDPs)–a variant of MDPs in which the goal is to realize a specified distribution of trajectories through a state space–as a general agent-coordination framework.

AAAI Conference 2007 Conference Paper

Authorial Idioms for Target Distributions in TTD-MDPs

• David L. Roberts
• Kenneth St. Clair

In designing Markov Decision Processes (MDP), one must define the world, its dynamics, a set of actions, and a reward function. MDPs are often applied in situations where there is a clear choice of reward functions and in these cases significant care must be taken to construct a reward function that induces the desired behavior. In this paper, we consider an analogous design problem: crafting a target distribution in Targeted Trajectory Distribution MDPs (TTD-MDPs). TTD-MDPs produce probabilistic policies that minimize divergence from a target distribution of trajectories from an underlying MDP. They are an extension of MDPs that provide variety of experience during repeated execution. Here, we present a brief overview of TTD-MDPs with approaches for constructing target distributions. Then we present a novel authorial idiom for creating target distributions using prototype trajectories. We evaluate these approaches on a drama manager for an interactive game.

AAMAS Conference 2007 Conference Paper

Autonomous Nondeterministic Tour Guides: Improving Quality of Experience with TTD-MDPs

• Andrew S. Cantino
• David L. Roberts
• Charles L. Isbell

In this paper, we address the problem of building a system of autonomous agents for a complex environment, in our case, a museum with many visitors. Visitors may have varying preferences for types of art or may wish to visit different exhibits on multiple visits. Often, these goals conict. For example, many visitors may wish to see the museum's most popular work, but that could cause congestion, ruining the experience. Thus, our task is to build a set of agents that can satisfy their visitors' goals, while simultaneously providing high quality experiences for all.

AAAI Conference 2006 Conference Paper

Targeting Specific Distributions of Trajectories in MDPs

• David L. Roberts
• Charles L. Isbell
• Michael Mateas

We define TTD-MDPs, a novel class of Markov decision processes where the traditional goal of an agent is changed from finding an optimal trajectory through a state space to realizing a specified distribution of trajectories through the space. After motivating this formulation, we show how to convert a traditional MDP into a TTD-MDP. We derive an algorithm for finding non-deterministic policies by constructing a trajectory tree that allows us to compute locally-consistent policies. We specify the necessary conditions for solving the problem exactly and present a heuristic algorithm for constructing policies when an exact answer is impossible or impractical. We present empirical results for our algorithm in two domains: a synthetic grid world and stories in an interactive drama or game.