Author name cluster

Bei Peng

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

13 papers

2 author rows

AAMAS Conference 2026 Conference Paper

Heuristic Transformer: Belief Augmented In-Context Reinforcement Learning

Oliver Dippel
Bei Peng
Alexei Lisitsa

Transformers have recently shown that reinforcement learning can be reframed as an in-context prediction problem, allowing agents to adapt to new tasks without updating their parameters. However, existing in-context RL approaches rely solely on past trajectories as prompts, leaving the model to implicitly infer uncertainty about the reward structure from raw experience alone. We introduce the Heuristic Transformer (HT), an in-context RL method that explicitly augments the prompt with a learned belief over rewards. A low-dimensional stochastic latent variable captures the posterior distribution over rewards and is provided to the transformer alongside trajectories and query states. This enables the model to reason directly over uncertainty rather than inferring it indirectly from data. Across Darkroom, Miniworld, and MuJoCo benchmarks, HT consistently outperforms existing in-context RL baselines in both performance and generalization, particularly in stochastic settings. Our results show that combining belief-based representations with transformer policies is a powerful mechanism for improving in-context decision-making, and suggest a new direction for integrating probabilistic reasoning into transformer-based RL.

PDF

EAAI Journal 2026 Journal Article

Modeling cross-regional local government cooperation in construction and demolition waste resource utilization through a small-world network framework

Xingwei Li
Yuxi Zou
Sijing Liu
Bei Peng
Weihong Chen

Cooperative governance among local governments is essential for addressing the challenges of construction and demolition waste resource utilization. However, research on how cooperation behaviors diffuse among local governments in construction and demolition waste cross-regional resource utilization projects remains scarce. This study employs agent-based modeling methods from the field of artificial intelligence, combined with complex adaptive systems theory, to simulate the effects of decision-making preferences, cooperative benefits, central government subsidies, and penalties on the cooperative behavior of local governments. This study constructed a small-world network model comprising 40 entities. After 100 iterations of evolution, the results showed that the diffusion of positive cooperation behaviors is more effective when the proportion of local governments with stubborn decision-making personalities is greater. Conversely, an increase in governments with mutable decision-making personalities fosters the spread of negative cooperation behaviors. In addition, both the increased cooperation benefits and central government regulations lead to a wider diffusion of positive cooperation behaviors among local governments. In terms of engineering application, these results provide valuable insights for advancing collaborative governance in other regions or sectors facing similar challenges.

Details DOI

IROS Conference 2025 Conference Paper

LSTM-MHSA-Enhanced Deep Reinforcement Learning for Accurate Gait Control in Human Musculoskeletal Model

Shiyu Mao
Zihao Tang
Fanny Ficuciello
Bei Peng
Dunwen Wei

Modeling and controlling the musculoskeletal system are crucial for understanding human motor functions, optimizing human-robot interaction, and developing embodied intelligence. However, existing musculoskeletal models are mainly limited to specific body parts and muscle groups, and still face challenges in large-scale muscle coordination and the generation of diverse movements. In this study, we propose a musculoskeletal deep reinforcement learning (DRL) control model. This model integrates a Long Short-Term Memory (LSTM) network and a Multi-Head Self-Attention (MHSA) mechanism into the Proximal Policy Optimization (PPO) algorithm. The LSTM-MHSA-enhanced PPO control approach generates accurate muscle activation, motion trajectories, and torque control strategies to precisely control and replicate diverse human gaits based on target joint movements. Experimental results demonstrate that this LSTM-MHSA-enhanced PPO algorithm significantly improves the model accuracy compared to the traditional PPO algorithm, with a 43. 75% and 34. 14% reduction in Mean Absolute Error (MAE) for walking and running tasks, respectively. Furthermore, for complex tasks such as striking and dancing, the MAE decreases by 46. 97% and 41. 78%, respectively. These findings highlight that integrating LSTM and MHSA into PPO algorithm not only enhances gait simulation accuracy but also improves the model’s generalization capability, particularly for complex motion patterns. This research provides an efficient tool for motion simulation and gait analysis, advancing the development of human musculoskeletal control systems.

Details

NeurIPS Conference 2025 Conference Paper

MACS: Multi-Agent Reinforcement Learning for Optimization of Crystal Structures

Elena Zamaraeva
Christopher Collins
George Darling
Matthew S Dyer
Bei Peng
Rahul Savani
Dmytro Antypov
Vladimir Gusev

Geometry optimization of atomic structures is a common and crucial task in computational chemistry and materials design. Following the learning to optimize paradigm, we propose a new multi-agent reinforcement learning method called Multi-Agent Crystal Structure optimization (MACS) to address the problem of periodic crystal structure optimization. MACS treats geometry optimization as a partially observable Markov game in which atoms are agents that adjust their positions to collectively discover a stable configuration. We train MACS across various compositions of reported crystalline materials to obtain a policy that successfully optimizes structures from the training compositions as well as structures of larger sizes and unseen compositions, confirming its excellent scalability and zero-shot transferability. We benchmark our approach against a broad range of state-of-the-art optimization methods and demonstrate that MACS optimizes periodic crystal structures significantly faster, with fewer energy calculations, and the lowest failure rate.

PDF Details

NeurIPS Conference 2021 Conference Paper

FACMAC: Factored Multi-Agent Centralised Policy Gradients

Bei Peng
Tabish Rashid
Christian Schroeder de Witt
Pierre-alexandre Kamienny
Philip Torr
Wendelin Boehmer
Shimon Whiteson

We propose FACtored Multi-Agent Centralised policy gradients (FACMAC), a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces. Like MADDPG, a popular multi-agent actor-critic method, our approach uses deep deterministic policy gradients to learn policies. However, FACMAC learns a centralised but factored critic, which combines per-agent utilities into the joint action-value function via a non-linear monotonic function, as in QMIX, a popular multi-agent $Q$-learning algorithm. However, unlike QMIX, there are no inherent constraints on factoring the critic. We thus also employ a nonmonotonic factorisation and empirically demonstrate that its increased representational capacity allows it to solve some tasks that cannot be solved with monolithic, or monotonically factored critics. In addition, FACMAC uses a centralised policy gradient estimator that optimises over the entire joint action space, rather than optimising over each agent's action space separately as in MADDPG. This allows for more coordinated policy changes and fully reaps the benefits of a centralised critic. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks. Empirical results demonstrate FACMAC's superior performance over MADDPG and other baselines on all three domains.

PDF Details

NeurIPS Conference 2021 Conference Paper

Regularized Softmax Deep Multi-Agent Q-Learning

Ling Pan
Tabish Rashid
Bei Peng
Longbo Huang
Shimon Whiteson

Tackling overestimation in $Q$-learning is an important problem that has been extensively studied in single-agent reinforcement learning, but has received comparatively little attention in the multi-agent setting. In this work, we empirically demonstrate that QMIX, a popular $Q$-learning algorithm for cooperative multi-agent reinforcement learning (MARL), suffers from a more severe overestimation in practice than previously acknowledged, and is not mitigated by existing approaches. We rectify this with a novel regularization-based update scheme that penalizes large joint action-values that deviate from a baseline and demonstrate its effectiveness in stabilizing learning. Furthermore, we propose to employ a softmax operator, which we efficiently approximate in a novel way in the multi-agent setting, to further reduce the potential overestimation bias. Our approach, Regularized Softmax (RES) Deep Multi-Agent $Q$-Learning, is general and can be applied to any $Q$-learning based MARL algorithm. We demonstrate that, when applied to QMIX, RES avoids severe overestimation and significantly improves performance, yielding state-of-the-art results in a variety of cooperative multi-agent tasks, including the challenging StarCraft II micromanagement benchmarks.

PDF Details

JMLR Journal 2020 Journal Article

Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey

Sanmit Narvekar
Bei Peng
Matteo Leonetti
Jivko Sinapov
Matthew E. Taylor
Peter Stone

Reinforcement learning (RL) is a popular paradigm for addressing sequential decision tasks in which the agent has only limited environmental feedback. Despite many advances over the past three decades, learning in many domains still requires a large amount of interaction with the environment, which can be prohibitively expensive in realistic scenarios. To address this problem, transfer learning has been applied to reinforcement learning such that experience gained in one task can be leveraged when starting to learn the next, harder task. More recently, several lines of research have explored how tasks, or data samples themselves, can be sequenced into a curriculum for the purpose of learning a problem that may otherwise be too difficult to learn from scratch. In this article, we present a framework for curriculum learning (CL) in reinforcement learning, and use it to survey and classify existing CL methods in terms of their assumptions, capabilities, and goals. Finally, we use our framework to find open problems and suggest directions for future RL curriculum learning research. [abs] [ pdf ][ bib ] &copy JMLR 2020. ( edit, beta )

PDF Details

NeurIPS Conference 2020 Conference Paper

Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Tabish Rashid
Gregory Farquhar
Bei Peng
Shimon Whiteson

QMIX is a popular $Q$-learning algorithm for cooperative MARL in the centralised training and decentralised execution paradigm. In order to enable easy decentralisation, QMIX restricts the joint action $Q$-values it can represent to be a monotonic mixing of each agent's utilities. However, this restriction prevents it from representing value functions in which an agent's ordering over its actions can depend on other agents' actions. To analyse this representational limitation, we first formalise the objective QMIX optimises, which allows us to view QMIX as an operator that first computes the $Q$-learning targets and then projects them into the space representable by QMIX. This projection returns a representable $Q$-value that minimises the unweighted squared error across all joint actions. We show in particular that this projection can fail to recover the optimal policy even with access to $Q^*$, which primarily stems from the equal weighting placed on each joint action. We rectify this by introducing a weighting into the projection, in order to place more importance on the better joint actions. We propose two weighting schemes and prove that they recover the correct maximal action for any joint action $Q$-values, and therefore for $Q^*$ as well. Based on our analysis and results in the tabular setting we introduce two scalable versions of our algorithm, Centrally-Weighted (CW) QMIX and Optimistically-Weighted (OW) QMIX and demonstrate improved performance on both predator-prey and challenging multi-agent StarCraft benchmark tasks (Samvelyan et al. , 2019).

PDF Details

AAMAS Conference 2017 Conference Paper

Curriculum Design for Machine Learners in Sequential Decision Tasks

Bei Peng
James MacGlashan
Robert Loftin
Michael L. Littman
David L. Roberts
Matthew E. Taylor

Existing machine-learning work has shown that algorithms can benefit from curricula—learning first on simple examples before moving to more difficult examples. While most existing work on curriculum learning focuses on developing automatic methods to iteratively select training examples with increasing difficulty tailored to the current ability of the learner, relatively little attention has been paid to the ways in which humans design curricula. We argue that a better understanding of the human-designed curricula could give us insights into the development of new machinelearning algorithms and interfaces that can better accommodate machine- or human-created curricula. Our work addresses this emerging and vital area empirically, taking an important step to characterize the nature of human-designed curricula relative to the space of possible curricula and the performance benefits that may (or may not) occur.

PDF

AAMAS Conference 2017 Conference Paper

How Do Humans Teach: On Curriculum Design for Machine Learners

Bei Peng

Existing machine-learning work has shown that algorithms can benefit from curricula—learning first on simple examples before moving to more difficult examples. While most existing work on curriculum learning focuses on developing automatic methods to iteratively select training examples with increasing difficulty tailored to the current ability of the learner, relatively little attention has been paid to the ways in which humans design curricula. This thesis aims to better understand the curriculum-design strategies followed by non-experts when teaching the agent, and leverage the findings to develop new machine-learning algorithms and interfaces that better accommodate natural tendencies of human trainers. We discuss completed work on this topic, including the definition of a curriculum-design problem in the context of sequential decision tasks, analysis of how different curricula affect agent learning in a Sokoban-like domain, and results of a user study that explores whether non-experts generate such curricula. Finally, we also present directions for future work.

PDF

AAMAS Conference 2016 Conference Paper

A Need for Speed: Adapting Agent Action Speed to Improve Task Learning from Non-Expert Humans

Bei Peng
James MacGlashan
Robert Loftin
Michael L. Littman
David L. Roberts
Matthew E. Taylor

As robots become pervasive in human environments, it is important to enable users to effectively convey new skills without programming. Most existing work on Interactive Reinforcement Learning focuses on interpreting and incorporating non-expert human feedback to speed up learning; we aim to design a better representation of the learning agent that is able to elicit more natural and effective communication between the human trainer and the learner, while treating human feedback as discrete communication that depends probabilistically on the trainer’s target policy. This work entails a user study where participants train a virtual agent to accomplish tasks by giving reward and/or punishment in a variety of simulated environments. We present results from 60 participants to show how a learner can ground natural language commands and adapt its action execution speed to learn more efficiently from human trainers. The agent’s action execution speed can be successfully modulated to encourage more explicit feedback from a human trainer in areas of the state space where there is high uncertainty. Our results show that our novel adaptive speed agent dominates different fixed speed agents on several measures of performance. Additionally, we investigate the impact of instructions on user performance and user preference in training conditions.

PDF

JAAMAS Journal 2015 Journal Article

Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning

Robert Loftin
Bei Peng
David L. Roberts

Abstract For real-world applications, virtual agents must be able to learn new behaviors from non-technical users. Positive and negative feedback are an intuitive way to train new behaviors, and existing work has presented algorithms for learning from such feedback. That work, however, treats feedback as numeric reward to be maximized, and assumes that all trainers provide feedback in the same way. In this work, we show that users can provide feedback in many different ways, which we describe as “training strategies. ” Specifically, users may not always give explicit feedback in response to an action, and may be more likely to provide explicit reward than explicit punishment, or vice versa, such that the lack of feedback itself conveys information about the behavior. We present a probabilistic model of trainer feedback that describes how a trainer chooses to provide explicit reward and/or explicit punishment and, based on this model, develop two novel learning algorithms (SABL and I-SABL) which take trainer strategy into account, and can therefore learn from cases where no feedback is provided. Through online user studies we demonstrate that these algorithms can learn with less feedback than algorithms based on a numerical interpretation of feedback. Furthermore, we conduct an empirical analysis of the training strategies employed by users, and of factors that can affect their choice of strategy.

Details DOI

AAAI Conference 2014 Conference Paper

A Strategy-Aware Technique for Learning Behaviors from Discrete Human Feedback

Robert Loftin
James MacGlashan
Bei Peng
Matthew Taylor
Michael Littman
Jeff Huang
David Roberts

This paper introduces two novel algorithms for learning behaviors from human-provided rewards. The primary novelty of these algorithms is that instead of treating the feedback as a numeric reward signal, they interpret feedback as a form of discrete communication that depends on both the behavior the trainer is trying to teach and the teaching strategy used by the trainer. For example, some human trainers use a lack of feedback to indicate whether actions are correct or incorrect, and interpreting this lack of feedback accurately can significantly improve learning speed. Results from user studies show that humans use a variety of training strategies in practice and both algorithms can learn a contextual bandit task faster than algorithms that treat the feedback as numeric. Simulated trainers are also employed to evaluate the algorithms in both contextual bandit and sequential decision-making tasks with similar results.

PDF Details