Elad Liebman

NeurIPS Conference 2024 Conference Paper

N-agent Ad Hoc Teamwork

• Caroline Wang
• Arrasy Rahman
• Ishan Durugkar
• Elad Liebman
• Peter Stone

Current approaches to learning cooperative multi-agent behaviors assume relatively restrictive settings. In standard fully cooperative multi-agent reinforcement learning, the learning algorithm controls *all* agents in the scenario, while in ad hoc teamwork, the learning algorithm usually assumes control over only a *single* agent in the scenario. However, many cooperative settings in the real world are much less restrictive. For example, in an autonomous driving scenario, a company might train its cars with the same learning algorithm, yet once on the road, these cars must cooperate with cars from another company. Towards expanding the class of scenarios that cooperative learning methods may optimally address, we introduce $N$*-agent ad hoc teamwork* (NAHT), where a set of autonomous agents must interact and cooperate with dynamically varying numbers and types of teammates. This paper formalizes the problem, and proposes the *Policy Optimization with Agent Modelling* (POAM) algorithm. POAM is a policy gradient, multi-agent reinforcement learning approach to the NAHT problem, that enables adaptation to diverse teammate behaviors by learning representations of teammate behaviors. Empirical evaluation on tasks from the multi-agent particle environment and StarCraft II shows that POAM improves cooperative task returns compared to baseline approaches, and enables out-of-distribution generalization to unseen teammates.

AAAI Conference 2023 Conference Paper

DM²: Decentralized Multi-Agent Reinforcement Learning via Distribution Matching

• Caroline Wang
• Ishan Durugkar
• Elad Liebman
• Peter Stone

Current approaches to multi-agent cooperation rely heavily on centralized mechanisms or explicit communication protocols to ensure convergence. This paper studies the problem of distributed multi-agent learning without resorting to centralized components or explicit communication. It examines the use of distribution matching to facilitate the coordination of independent agents. In the proposed scheme, each agent independently minimizes the distribution mismatch to the corresponding component of a target visitation distribution. The theoretical analysis shows that under certain conditions, each agent minimizing its individual distribution mismatch allows the convergence to the joint policy that generated the target distribution. Further, if the target distribution is from a joint policy that optimizes a cooperative task, the optimal policy for a combination of this task reward and the distribution matching reward is the same joint policy. This insight is used to formulate a practical algorithm (DM^2), in which each individual agent matches a target distribution derived from concurrently sampled trajectories from a joint expert policy. Experimental validation on the StarCraft domain shows that combining (1) a task reward, and (2) a distribution matching reward for expert demonstrations for the same task, allows agents to outperform a naive distributed baseline. Additional experiments probe the conditions under which expert demonstrations need to be sampled to obtain the learning benefits.

IJCAI Conference 2020 Conference Paper

Balancing Individual Preferences and Shared Objectives in Multiagent Reinforcement Learning

• Ishan Durugkar
• Elad Liebman
• Peter Stone

In multiagent reinforcement learning scenarios, it is often the case that independent agents must jointly learn to perform a cooperative task. This paper focuses on such a scenario in which agents have individual preferences regarding how to accomplish the shared task. We consider a framework for this setting which balances individual preferences against task rewards using a linear mixing scheme. In our theoretical analysis we establish that agents can reach an equilibrium that leads to optimal shared task reward even when they consider individual preferences which aren't fully aligned with this task. We then empirically show, somewhat counter-intuitively, that there exist mixing schemes that outperform a purely task-oriented baseline. We further consider empirically how to optimize the mixing scheme.

RLDM Conference 2019 Conference Abstract

Balancing Individual Preferences with Shared Objectives in Multiagent Co- operation

• Ishan P Durugkar
• Elad Liebman

Much of the multiagent research literature has considered the challenges of jointly learning how to perform shared tasks, particularly tasks that require agents to communicate and coordinate in order to succeed. When the agents must cooperate towards a shared objective without any prior coordination, this problem falls within the ad hoc teamwork scenario. This paper considers a particular cooperative scenario in which the agents still do not get to coordinate a priori, but in addition start off with their own individual preferences. In this paper we consider whether we can leverage these individual preferences, and propose a balancing scheme to do so. We analyze the effects of individual preferences with this balancing scheme towards the shared objective. Our experiments show that having individual preferences may prove beneficial for overall shared task performance in certain contexts.

RLDM Conference 2019 Conference Abstract

Utilizing Background Music in Person-Agent Interaction

• Elad Liebman
• Peter Stone

Numerous studies have demonstrated that mood affects emotional and cognitive processing. Previous work has established that music-induced mood can measurably alter people’s behavior in different contexts. Recent work suggests that this impact also holds in social and cooperative settings. In this study we further establish how background information (and specifically music) can affect people’s decision making in inter-social tasks, and show that this information can be effectively incorporated in an agent’s world representation in order to better predict people’s behavior. For this purpose, we devised an experiment in which people drove a simulated car through an intersection while listening to music. The intersection was not empty, as another simulated vehicle, controlled autonomously, was also crossing the intersection in a different direction. Our results corroborate that music indeed alters people’s behavior with respect to this social task. Furthermore, we show that explicitly modeling this impact is possible, and can lead to improved performance of the autonomous agent.

AAMAS Conference 2018 Conference Paper

A Stitch in Time - Autonomous Model Management via Reinforcement Learning

• Elad Liebman
• Eric Zavesky
• Peter Stone

Concept drift - a change, either sudden or gradual, in the underlying properties of data - is one of the most prevalent challenges to maintaining high-performing learned models over time in autonomous systems. In the face of concept drift, one can hope that the old model is sufficiently representative of the new data despite the concept drift, one can discard the old data and retrain a new model with (often limited) new data, or one can use transfer learning methods to combine the old data with the new to create an updated model. Which of these three options is chosen affects not only near-term decisions, but also future needs to transfer or retrain. In this paper, we thus model response to concept drift as a sequential decision making problem and formally frame it as a Markov Decision Process. Our reinforcement learning approach to the problem shows promising results on one synthetic and two real-world datasets.

AAAI Conference 2018 Short Paper

Sequential Decision Making in Artificial Musical Intelligence

• Elad Liebman

My main research motivation is to develop complete autonomous agents that interact with people socially. For an agent to be social with respect to humans, it needs to be able to parse and process the human cultural experience. That in itself gives rise to many fascinating learning problems. Music, as a general target domain, serves as an excellent testbed for these research ideas. Musical skills involve extremely advanced knowledge representation and problem solving tools. Creating agents that can interact richly with people in the music domain is a challenge that will advance social agents research and contribute important and broadly applicable AI knowledge. This belief is fueled not just by my background in computer science and artificial intelligence, but also by my deep passion for music as well as my extensive musical training. One key aspect of musical intelligence which hasn’t been sufficiently studied is that of sequential decision-making. My thesis strives to answer the following question: How can a sequential decision making perspective guide us in the creation of better music agents, and social agents in general? More specifically, this thesis focuses on two aspects of musical intelligence: music recommendation and multiagent interaction in the context of music.

AAMAS Conference 2017 Conference Paper

Autonomous Model Management via Reinforcement Learning - Extended Abstract

• Elad Liebman
• Eric Zavesky
• Peter Stone

ICML Conference 2016 Conference Paper

On the Analysis of Complex Backup Strategies in Monte Carlo Tree Search

• Piyush Khandelwal
• Elad Liebman
• Scott Niekum
• Peter Stone 0001

Over the past decade, Monte Carlo Tree Search (MCTS) and specifically Upper Confidence Bound in Trees (UCT) have proven to be quite effective in large probabilistic planning domains. In this paper, we focus on how values are backpropagated in the MCTS tree, and apply complex return strategies from the Reinforcement Learning (RL) literature to MCTS, producing 4 new MCTS variants. We demonstrate that in some probabilistic planning benchmarks from the International Planning Competition (IPC), selecting a MCTS variant with a backup strategy different from Monte Carlo averaging can lead to substantially better results. We also propose a hypothesis for why different backup strategies lead to different performance in particular environments, and manipulate a carefully structured grid-world domain to provide empirical evidence supporting our hypothesis.

IS Journal 2016 Journal Article

UT Austin Villa: Project-Driven Research in AI and Robotics

• Katie Genter
• Patrick MacAlpine
• Jacob Menashe
• Josiah Hannah
• Elad Liebman
• Sanmit Narvekar
• Ruohan Zhang
• Peter Stone

UT Austin Villa is a robot soccer team that has competed in the annual RoboCup soccer competitions since 2003. The team has won several championships and has inspired research contributions spanning many topics in robotics and artificial intelligence. This article summarizes some of these research contributions and provides a snapshot into the current development status of the team. Educational uses of the team's code bases are also presented.

RLDM Conference 2015 Conference Abstract

Decision Mechanisms Underlying Mood-Congruent Emotional Classification

• Elad Liebman
• Peter Stone
• Corey White

Numerous studies have demonstrated that an individual’s mood can affect their emotional pro- cessing. The goal of the present study was to use a sequential sampling model of simple decisions, the drift- diffusion model (DDM), to explore which components of the decision process underlie mood-congruent bias in emotional decision making. DDM assumes that decisions are made by a noisy process that accumu- lates information over time from a starting point toward one of response criteria or boundaries. This model can be fitted to response times and choice probabilities to determine whether classification bias reflects a change in the emotional evaluation of the stimuli, or rather a change in a priori bias for one response over the other. In our experiment, participants decided whether words were emotionally positive or negative while listening to music that was chosen to induce positive or negative mood. The behavioral results show that the music manipulation was effective, as participants were biased to label words positive in the positive music condition. The drift-diffusion model shows that this bias was driven by a change in the starting point of evi- dence accumulation, which indicates an a priori response bias. In contrast, there was no evidence that music affected how participants evaluated the emotional content of the stimuli, which would have been reflected by a change in the drift rates. This result has implications for future studies of emotional classification and mood, which we discuss.

RLDM Conference 2013 Conference Abstract

DJ-MC: A Reinforcement-Learning Framework for a Music Playlist Recommender System (Ex-

• Elad Liebman
• Peter Stone

In recent years, there has been growing focus on the study of automated recommender systems. Music recommendation systems serve as a prominent domain for such works, both from an academic and a commercial perspective. To our knowledge, most of these systems focus on predicting the preference of individual songs independently based on a learned model of a listener. However, a relatively well known fact in music cognition is that music is experienced in temporal context and in sequence. In this work we present a reinforcement-learning based framework for music recommendation that does not recommend songs individually but rather song sequences, or playlists, based on a learned model of preferences for both individual songs and song transitions. To reduce exploration time, we initialize a model based on user feedback. This model is subsequently updated by reinforcement. We show our algorithm outperforms a more naive approach both on synthetic data and on a real song database.