Arrow Research search

Author name cluster

Peter R. Wurman

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

15 papers
2 author rows

Possible papers

15

AAAI Conference 2026 Conference Paper

Out-of-Distribution Generalization with a SPARC: Racing 100 Unseen Vehicles with a Single Policy

  • Bram Grooten
  • Patrick MacAlpine
  • Kaushik Subramanian
  • Peter Stone
  • Peter R. Wurman

Generalization to unseen environments is a significant challenge in the field of robotics and control. In this work, we focus on contextual reinforcement learning, where agents act within environments with varying contexts, such as self-driving cars or quadrupedal robots that need to operate in different terrains or weather conditions than they were trained for. We tackle the critical task of generalizing to out-of-distribution (OOD) settings, without access to explicit context information at test time. Recent work has addressed this problem by training a context encoder and a history adaptation module in separate stages. While promising, this two-phase approach is cumbersome to implement and train. We simplify the methodology and introduce SPARC: single-phase adaptation for robust control. We test SPARC on varying contexts within the high-fidelity racing simulator Gran Turismo 7 and wind-perturbed MuJoCo environments, and find that it achieves reliable and robust OOD generalization.

PDF Details DOI

EWRL Workshop 2025 Workshop Paper

Generalization with a SPARC: Single-Phase Adaptation for Reinforcement Learning in Contextual Environments

  • Bram Grooten
  • Patrick MacAlpine
  • Kaushik Subramanian
  • Peter R. Wurman
  • Peter Stone

Generalization to unseen environments is a significant challenge in the field of robotics and control. In this work, we focus on contextual reinforcement learning, where the agent acts within environments with varying contexts, such as self-driving cars or quadrupedal robots that need to operate in different terrains or weather conditions than they were trained for. We tackle the critical task of generalizing to out-of-distribution (OOD) contexts, without access to explicit context information at test time. Recent work has addressed this problem by training a context encoder and a history adaptation module in separate stages. While promising, this two-phase approach is cumbersome to implement and train. We simplify the methodology and introduce SPARC, a single-phase adaptation method for reinforcement learning in contextual environments. We evaluate SPARC on varying contexts within MuJoCo environments and the high-fidelity racing simulator Gran Turismo 7 and find that it achieves competitive or superior performance on OOD generalization.

PDF

RLC Conference 2025 Conference Paper

ProtoCRL: Prototype-based Network for Continual Reinforcement Learning

  • Michela Proietti
  • Peter R. Wurman
  • Peter Stone
  • Roberto Capobianco

The purpose of continual reinforcement learning is to train an agent on a sequence of tasks such that it learns the ones that appear later in the sequence while retaining the ability to perform the tasks that appeared earlier. Experience replay is a popular method used to make the agent remember previous tasks, but its effectiveness strongly relies on the selection of experiences to store. Kompella et al. (2023) proposed organizing the experience replay buffer into partitions, each storing transitions leading to a rare but crucial event, such that these key experiences get revisited more often during training. However, the method is sensitive to the manual selection of event states. To address this issue, we introduce ProtoCRL, a prototype-based architecture leveraging a variational Gaussian mixture model to automatically discover effective event states and build the associated partitions in the experience replay buffer. The proposed approach is tested on a sequence of MiniGrid environments, demonstrating the agent's ability to adapt and learn new skills incrementally.

PDF Details

RLJ Journal 2025 Journal Article

ProtoCRL: Prototype-based Network for Continual Reinforcement Learning

  • Michela Proietti
  • Peter R. Wurman
  • Peter Stone
  • Roberto Capobianco

The purpose of continual reinforcement learning is to train an agent on a sequence of tasks such that it learns the ones that appear later in the sequence while retaining the ability to perform the tasks that appeared earlier. Experience replay is a popular method used to make the agent remember previous tasks, but its effectiveness strongly relies on the selection of experiences to store. Kompella et al. (2023) proposed organizing the experience replay buffer into partitions, each storing transitions leading to a rare but crucial event, such that these key experiences get revisited more often during training. However, the method is sensitive to the manual selection of event states. To address this issue, we introduce ProtoCRL, a prototype-based architecture leveraging a variational Gaussian mixture model to automatically discover effective event states and build the associated partitions in the experience replay buffer. The proposed approach is tested on a sequence of MiniGrid environments, demonstrating the agent's ability to adapt and learn new skills incrementally.

PDF Details

ICLR Conference 2025 Conference Paper

SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning

  • Hojoon Lee
  • Dongyoon Hwang
  • Donghu Kim
  • Hyunseung Kim
  • Jun Jet Tai
  • Kaushik Subramanian
  • Peter R. Wurman
  • Jaegul Choo

Recent advances in CV and NLP have been largely driven by scaling up the number of network parameters, despite traditional theories suggesting that larger networks are prone to overfitting. These large networks avoid overfitting by integrating components that induce a simplicity bias, guiding models toward simple and generalizable solutions. However, in deep RL, designing and scaling up networks have been less explored. Motivated by this opportunity, we present SimBa, an architecture designed to scale up parameters in deep RL by injecting a simplicity bias. SimBa consists of three components: (i) an observation normalization layer that standardizes inputs with running statistics, (ii) a residual feedforward block to provide a linear pathway from the input to output, and (iii) a layer normalization to control feature magnitudes. By scaling up parameters with SimBa, the sample efficiency of various deep RL algorithms—including off-policy, on-policy, and unsupervised methods—is consistently improved. Moreover, solely by integrating SimBa architecture into SAC, it matches or surpasses state-of-the-art deep RL methods with high computational efficiency across DMC, MyoSuite, and HumanoidBench. These results demonstrate SimBa's broad applicability and effectiveness across diverse RL algorithms and environments.

Details

RLC Conference 2024 Conference Paper

A Super-human Vision-based Reinforcement Learning Agent for Autonomous Racing in Gran Turismo

  • Miguel Vasco
  • Takuma Seno
  • Kenta Kawamoto
  • Kaushik Subramanian
  • Peter R. Wurman
  • Peter Stone

Racing autonomous cars faster than the best human drivers has been a longstanding grand challenge for the fields of Artificial Intelligence and robotics. Recently, an end-to-end deep reinforcement learning agent met this challenge in a high-fidelity racing simulator, Gran Turismo. However, this agent relied on global features that require instrumentation external to the car. This paper introduces, to the best of our knowledge, the first super-human car racing agent whose sensor input is purely local to the car, namely pixels from an ego-centric camera view and quantities that can be sensed from on-board the car, such as the car's velocity. By leveraging global features only at training time, the learned agent is able to outperform the best human drivers in time trial (one car on the track at a time) races using only local input features. The resulting agent is evaluated in Gran Turismo 7 on multiple tracks and cars. Detailed ablation experiments demonstrate the agent's strong reliance on visual inputs, making it the first vision-based super-human car racing agent.

PDF Details

RLJ Journal 2024 Journal Article

A Super-human Vision-based Reinforcement Learning Agent for Autonomous Racing in Gran Turismo

  • Miguel Vasco
  • Takuma Seno
  • Kenta Kawamoto
  • Kaushik Subramanian
  • Peter R. Wurman
  • Peter Stone

Racing autonomous cars faster than the best human drivers has been a longstanding grand challenge for the fields of Artificial Intelligence and robotics. Recently, an end-to-end deep reinforcement learning agent met this challenge in a high-fidelity racing simulator, Gran Turismo. However, this agent relied on global features that require instrumentation external to the car. This paper introduces, to the best of our knowledge, the first super-human car racing agent whose sensor input is purely local to the car, namely pixels from an ego-centric camera view and quantities that can be sensed from on-board the car, such as the car's velocity. By leveraging global features only at training time, the learned agent is able to outperform the best human drivers in time trial (one car on the track at a time) races using only local input features. The resulting agent is evaluated in Gran Turismo 7 on multiple tracks and cars. Detailed ablation experiments demonstrate the agent's strong reliance on visual inputs, making it the first vision-based super-human car racing agent.

PDF Details

NeurIPS Conference 2024 Conference Paper

Discovering Creative Behaviors through DUPLEX: Diverse Universal Features for Policy Exploration

  • Borja G. Leon
  • Francesco Riccio
  • Kaushik Subramanian
  • Peter R. Wurman
  • Peter Stone

The ability to approach the same problem from different angles is a cornerstone of human intelligence that leads to robust solutions and effective adaptation to problem variations. In contrast, current RL methodologies tend to lead to policies that settle on a single solution to a given problem, making them brittle to problem variations. Replicating human flexibility in reinforcement learning agents is the challenge that we explore in this work. We tackle this challenge by extending state-of-the-art approaches to introduce DUPLEX, a method that explicitly defines a diversity objective with constraints and makes robust estimates of policies’ expected behavior through successor features. The trained agents can (i) learn a diverse set of near-optimal policies in complex highly-dynamic environments and (ii) exhibit competitive and diverse skills in out-of-distribution (OOD) contexts. Empirical results indicate that DUPLEX improves over previous methods and successfully learns competitive driving styles in a hyper-realistic simulator (i. e. , GranTurismo ™ 7) as well as diverse and effective policies in several multi-context robotics MuJoCo simulations with OOD gravity forces and height limits. To the best of our knowledge, our method is the first to achieve diverse solutions in complex driving simulators and OOD robotic contexts. DUPLEX agents demonstrating diverse behaviors can be found at https: //ai. sony/publications/Discovering-Creative-Behaviors-through-DUPLEX-Diverse-Universal-Features-for-Policy-Exploration/.

PDF Details DOI

UAI Conference 2023 Conference Paper

Composing Efficient, Robust Tests for Policy Selection

  • Dustin Morrill
  • Thomas J. Walsh 0001
  • Daniel Hernandez
  • Peter R. Wurman
  • Peter Stone 0001

Modern reinforcement learning systems produce many high-quality policies throughout the learning process. However, to choose which policy to actually deploy in the real world, they must be tested under an intractable number of environmental conditions. We introduce RPOSST, an algorithm to select a small set of test cases from a larger pool based on a relatively small number of sample evaluations. RPOSST treats the test case selection problem as a two-player game and optimizes a solution with provable $k$-of-$N$ robustness, bounding the error relative to a test that used all the test cases in the pool. Empirical results demonstrate that RPOSST finds a small set of test cases that identify high quality policies in a toy one-shot game, poker datasets, and a high-fidelity racing simulator.

Details

TMLR Journal 2023 Journal Article

Event Tables for Efficient Experience Replay

  • Varun Raj Kompella
  • Thomas Walsh
  • Samuel Barrett
  • Peter R. Wurman
  • Peter Stone

Experience replay (ER) is a crucial component of many deep reinforcement learning (RL) systems. However, uniform sampling from an ER buffer can lead to slow convergence and unstable asymptotic behaviors. This paper introduces Stratified Sampling from Event Tables (SSET), which partitions an ER buffer into Event Tables, each capturing important subsequences of optimal behavior. We prove a theoretical advantage over the traditional monolithic buffer approach and combine SSET with an existing prioritized sampling strategy to further improve learning speed and stability. Empirical results in challenging MiniGrid domains, benchmark RL environments, and a high-fidelity car racing simulator demonstrate the advantages and versatility of SSET over existing ER buffer sampling

PDF Details

IS Journal 2022 Journal Article

Challenges and Opportunities of Applying Reinforcement Learning to Autonomous Racing

  • Peter R. Wurman
  • Peter Stone
  • Michael Spranger

Simulated motorsports are an exciting environment in which to explore the power and limitations of deep reinforcement learning. Racing requires precise control of a vehicle that is operating at its traction limits while competing wheel-to-wheel with other drivers. We recently demonstrated an agent that can beat the best drivers in the world at the racing game Gran Turismo. In this article, we briefly discuss some of the lessons learned and some of the remaining open research challenges.

Details DOI

JAIR Journal 2021 Journal Article

Agent-Based Markov Modeling for Improved COVID-19 Mitigation Policies

  • Roberto Capobianco
  • Varun Kompella
  • James Ault
  • Guni Sharon
  • Stacy Jong
  • Spencer Fox
  • Lauren Meyers
  • Peter R. Wurman

The year 2020 saw the covid-19 virus lead to one of the worst global pandemics in history. As a result, governments around the world have been faced with the challenge of protecting public health while keeping the economy running to the greatest extent possible. Epidemiological models provide insight into the spread of these types of diseases and predict the effects of possible intervention policies. However, to date, even the most data-driven intervention policies rely on heuristics. In this paper, we study how reinforcement learning (RL) and Bayesian inference can be used to optimize mitigation policies that minimize economic impact without overwhelming hospital capacity. Our main contributions are (1) a novel agent-based pandemic simulator which, unlike traditional models, is able to model fine-grained interactions among people at specific locations in a community; (2) an RLbased methodology for optimizing fine-grained mitigation policies within this simulator; and (3) a Hidden Markov Model for predicting infected individuals based on partial observations regarding test results, presence of symptoms, and past physical contacts. This article is part of the special track on AI and COVID-19.

PDF Details DOI

ICRA Conference 2021 Conference Paper

Efficient Real-Time Inference in Temporal Convolution Networks

  • Piyush Khandelwal
  • James MacGlashan
  • Peter R. Wurman
  • Peter Stone 0001

It has been recently demonstrated that Temporal Convolution Networks (TCNs) provide state-of-the-art results in many problem domains where the input data is a time-series. TCNs typically incorporate information from a long history of inputs (the receptive field) into a single output using many convolution layers. Real-time inference using a trained TCN can be challenging on devices with limited compute and memory, especially if the receptive field is large. This paper introduces the RT-TCN algorithm that reuses the output of prior convolution operations to minimize the computational requirements and persistent memory footprint of a TCN during real-time inference. We also show that when a TCN is trained using time slices of the input time-series, it can be executed in realtime continually using RT-TCN. In addition, we provide TCN architecture guidelines that ensure that real-time inference can be performed within memory and computational constraints.

Details

UAI Conference 1996 Conference Paper

Optimal Factory Scheduling using Stochastic Dominance A*

  • Peter R. Wurman
  • Michael P. Wellman

We examine a standard factory scheduling problem with stochastic processing and setup times, minimizing the expectation of the weighted number of tardy jobs. Because the costs of operators in the schedule are stochastic and sequence dependent, standard dynamic programming algorithms such as A* may fail to find the optimal schedule. The SDA* (Stochastic Dominance A*) algorithm remedies this difficulty by relaxing the pruning condition. We present an improved state-space search formulation for these problems and discuss the conditions under which stochastic scheduling problems can be solved optimally using SDA*. In empirical testing on randomly generated problems, we found that in 70%, the expected cost of the optimal stochastic solution is lower than that of the solution derived using a deterministic approximation, with comparable search effort.

Details

AAAI Conference 1996 Short Paper

Optimal Factory Scheduling Using Stochastic Dominance A*

  • Peter R. Wurman

Generating optimal production schedules for manufacturing facilities is an area of great theoretical and practical importance. During the last decade, an effort has been made to reconcile the techniques developed by the AI and OR communities. The work described here aims to continue in this vein by showing how a class of well-defined stochastic scheduling problems can be mapped into a general search procedure. This approach improves upon other methods by handling the general case of multidimensional stochastic costs.

PDF Details
v2026.09.13