Author name cluster

Roxana Rădulescu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

16 papers

1 author row

AAMAS Conference 2026 Conference Paper

MORL4Water: A Modular Multi-Objective Reinforcement Learning Toolkit for Water Resource Management

Zuzanna Osika
Roxana Rădulescu
Jazmin Zatarain-Salazar
Frans A. Oliehoek
Pradeep K. Murukannaiah

Many real-world decision problems involve conflicting objectives. Multi-objective reinforcement learning (MORL) extends standard RLtooptimizemultipleobjectivessimultaneously, producingpolicy setsthatcapturedifferenttrade-offs. However, MORLresearchoften relies on simplified benchmarks with limited real-world relevance. We present MORL4Water, a modular toolkit for creating realistic MORL environments in water resource management. Built on MO- Gymnasium, MORL4Water enables scenario construction from real data and systematic evaluation of MORL methods. We illustrate its use on the Nile and Susquehanna rivers, benchmarking several MORL algorithms against EMODPS, a domain-specific baseline. Beyond standard performance metrics, we analyze solution sets to reveal differences in exploration, scalability, and trade-off diversity. Our results show that most state-of-the-art MORL algorithms underperform relative to EMODPS, especially in higher-dimensional settings, and highlight the value of solution-set analysis for robust, real-world applications.

PDF

AIJ Journal 2026 Journal Article

Multi-objective reinforcement learning for provably incentivising alignment with value systems

Manel Rodriguez-Soto
Roxana Rădulescu
Filippo Bistaffa
Oriol Ricart
Arnau Mayoral-Macau
Maite Lopez-Sanchez
Juan A. Rodriguez-Aguilar
Ann Nowé

This paper addresses the problem of ensuring that autonomous learning agents align with multiple moral values. Specifically, we present the theoretical principles and algorithmic tools necessary for creating an environment where we ensure that the agent learns a behaviour aligned with multiple moral values while striving to achieve its individual objective. To address this value alignment problem, we adopt the Multi-Objective Reinforcement Learning framework and propose a novel algorithm that combines techniques from Multi-Objective Reinforcement Learning and Linear Programming. In addition, we illustrate our value alignment process with an example involving an autonomous vehicle. Here, we demonstrate that the agent learns to behave in alignment with the ethical values of safety, achievement, and comfort, with achievement representing the agent’s individual objective. Such ethical behaviour differs depending on the ordering between values. We also use a synthetic multi-objective environment to evaluate the computational costs of guaranteeing ethical learning as the number of values increases.

Details DOI

AAMAS Conference 2025 Conference Paper

Multi-Objective Reinforcement Learning for Water Management

Zuzanna Osika
Roxana Rădulescu
Jazmin Zatarain-Salazar
Frans A. Oliehoek
Pradeep K. Murukannaiah

Many real-world problems (e. g. , resource management, autonomous driving, drug discovery) require optimizing multiple, conflicting objectives. Multi-objective reinforcement learning (MORL) extends classic reinforcement learning to handle multiple objectives simultaneously, yielding a set of policies that capture various trade-offs. However, the MORL field lacks complex, realistic environments and benchmarks. We introduce a water resource (Nile river basin) management case study and model it as a MORL environment. We then benchmark existing MORL algorithms on this task. Our results show that specialized water management methods outperform state-ofthe-art MORL approaches, underscoring the scalability challenges MORL algorithms face in real-world scenarios.

PDF

EWRL Workshop 2024 Workshop Paper

Divide and Conquer: Provably Unveiling the Pareto Front with Multi-Objective Reinforcement Learning

Willem Röpke
Mathieu Reymond
Patrick Mannion
Diederik M Roijers
Ann Nowe
Roxana Rădulescu

A notable challenge in multi-objective reinforcement learning is obtaining a Pareto front of policies to attain optimal performance under different preferences. We introduce Iterated Pareto Referent Optimisation (IPRO), which decomposes finding the Pareto front into a sequence of constrained single-objective problems. This enables us to guarantee convergence while providing an upper bound on the distance to undiscovered Pareto optimal solutions at each step. Empirical evaluations demonstrate that IPRO matches or outperforms methods that require additional assumptions. Furthermore, IPRO is a general-purpose multi-objective optimisation method, making it applicable to domains beyond reinforcement learning.

PDF

AAMAS Conference 2024 Conference Paper

Emergent Cooperation under Uncertain Incentive Alignment

Nicole Orzan
Erman Acar
Davide Grossi
Roxana Rădulescu

Understanding the emergence of cooperation in systems of computational agents is crucial for the development of effective cooperative AI. Interaction among individuals in real-world settings are often sparse and occur within a broad spectrum of incentives, which often are only partially known. In this work, we explore how cooperation can arise among reinforcement learning agents in scenarios characterised by infrequent encounters, and where agents face uncertainty about the alignment of their incentives with those of others. To do so, we train the agents under a wide spectrum of environments ranging from fully competitive, to fully cooperative, to mixed-motives. Under this type of uncertainty we study the effects of mechanisms, such as reputation and intrinsic rewards, that have been proposed in the literature to foster cooperation in mixed-motives environments. Our findings show that uncertainty substantially lowers the agents’ ability to engage in cooperative behaviour, when that would be the best course of action. In this scenario, the use of effective reputation mechanisms and intrinsic rewards boosts the agents’ capability to act nearly-optimally in cooperative environments, while greatly enhancing cooperation in mixed-motive environments as well.

PDF

AAMAS Conference 2024 Conference Paper

Utility-Based Reinforcement Learning: Unifying Single-objective and Multi-objective Reinforcement Learning

Peter Vamplew
Cameron Foale
Conor F. Hayes
Patrick Mannion
Enda Howley
Richard Dazeley
Scott Johnson
Johan Källström

Research in multi-objective reinforcement learning (MORL) has introduced the utility-based paradigm, which makes use of both environmental rewards and a function that defines the utility derived by the user from those rewards. In this paper we extend this paradigm to the context of single-objective reinforcement learning (RL), and outline multiple potential benefits including the ability to perform multi-policy learning across tasks relating to uncertain objectives, risk-aware RL, discounting, and safe RL. We also examine the algorithmic implications of adopting a utility-based approach.

PDF

AAMAS Conference 2023 Conference Paper

A Brief Guide to Multi-Objective Reinforcement Learning and Planning

Conor F. Hayes
Roxana Rădulescu
Eugenio Bargiacchi
Johan Källström
Matthew Macfarlane
Mathieu Reymond
Timothy Verstraeten
Luisa M. Zintgraf

Real-world sequential decision-making tasks are usually complex, and require trade-offs between multiple – often conflicting – objectives. However, the majority of research in reinforcement learning (RL) and decision-theoretic planning assumes a single objective, or that multiple objectives can be handled via a predefined weighted sum over the objectives. Such approaches may oversimplify the underlying problem, and produce suboptimal results. This extended abstract outlines the limitations of using a semi-blind iterative process to solve multi-objective decision making problems. Our extended paper [4], serves as a guide for the application of explicitly multi-objective methods to difficult problems.

PDF

AAMAS Conference 2023 Conference Paper

A Study of Nash Equilibria in Multi-Objective Normal-Form Games

Willem Röpke
Diederik M. Roijers
Ann Nowé
Roxana Rădulescu

We present a detailed analysis of Nash equilibria in multi-objective normal-form games, which are normal-form games with vectorial payoffs. Our approach is based on modelling each player’s utility using a utility function that maps a vector to a scalar utility. For mixed strategies, we can apply the utility function before calculating the expectation of the payoff vector as well as after, resulting in two distinct optimisation criteria. We show that when computing the utility from the expected payoff, a Nash equilibrium can be guaranteed when players have quasiconcave utility functions. In addition, we show that when players have quasiconvex utility functions, pure strategy Nash equilibria are equal under both optimisation criteria. We extend this to settings where some players optimise for one criterion, while others optimise for the second. We combine these results and formulate an algorithm that computes all pure strategy Nash equilibria given quasiconvex utility functions.

PDF

AAMAS Conference 2023 Conference Paper

Bridging the Gap Between Single and Multi Objective Games

Willem Röpke
Carla Groenland
Roxana Rădulescu
Ann Nowé
Diederik M. Roijers

A classic model to study strategic decision making in multi-agent systems is the normal-form game. This model can be generalised to allow for an infinite number of pure strategies leading to continuous games. Multi-objective normal-form games are another generalisation that model settings where players receive separate payoffs in more than one objective. We bridge the gap between the two models by providing a theoretical guarantee that a game from one setting can always be transformed to a game in the other. We extend the theoretical results to include guaranteed equivalence of Nash equilibria. The mapping makes it possible to apply algorithms from one field to the other. We demonstrate this by introducing a fictitious play algorithm for multi-objective games and subsequently applying it to two well-known continuous games. We believe the equivalence relation will lend itself to new insights by translating the theoretical guarantees from one formalism to another. Moreover, it may lead to new computational approaches for continuous games when a problem is more naturally solved in the succinct format of multi-objective games.

PDF

EWRL Workshop 2023 Workshop Paper

Cooperative Foraging Behaviour Through Multi-Agent Reinforcement Learning with Graph-Based Communication

Hicham Azmani
Andries Rosseau
Ann Nowe
Roxana Rădulescu

According to the social intelligence hypothesis, cooperation is considered a key component of intelligence and is required to solve a wide range of problems, from everyday challenges like scheduling meetings to global challenges like mitigating climate change and providing humanitarian aid. Extending the ability for artificial intelligence (AI) to cooperate well is critical as AI becomes more prevalent in our lives. In recent years, multi-agent reinforcement learning (MARL) has emerged as a powerful approach to model and analyse the problem of cooperation among artificial agents. In this paper, we investigate the impact of communication on cooperation among reinforcement learning agents in social dilemmas. These are settings in which the short-term individual interests are in conflict with the long-term collective ones, thus each individual profits from defecting, but the overall group would benefit if everyone cooperates. We particularly focus on a temporally and spatially extended Stag-Hunt-like social dilemma that models animal foraging behaviour using principles from Optimal Foraging Theory. We propose a method for communication that combines a graph-based attention mechanism with deep reinforcement learning methods. Additionally, we examine several facets of communication, including the effects of the communication topology and the communication range. We find that greater cooperative behaviour can be achieved through graph-based communication using reinforcement learning in social dilemmas. Additionally, we find that during foraging, local communication promotes better cooperation than long-distance communication. Finally, we visualise and investigate the learned attention weights and explain how agents process communications from other agents.

PDF

AAMAS Conference 2023 Conference Paper

Scalar Reward is Not Enough

Peter Vamplew
Benjamin J. Smith
Johan Källström
Gabriel Ramos
Roxana Rădulescu
Diederik M. Roijers
Conor F. Hayes
Friedrik Hentz

Silver et al. [14] posit that scalar reward maximisation is sufficient to underpin all intelligence and provides a suitable basis for artificial general intelligence (AGI). This extended abstract summarises the counter-argument from our JAAMAS paper[19].

PDF

JAAMAS Journal 2022 Journal Article

A practical guide to multi-objective reinforcement learning and planning

Conor F. Hayes
Roxana Rădulescu
Diederik M. Roijers

Abstract Real-world sequential decision-making tasks are generally complex, requiring trade-offs between multiple, often conflicting, objectives. Despite this, the majority of research in reinforcement learning and decision-theoretic planning either assumes only a single objective, or that multiple objectives can be adequately handled via a simple linear combination. Such approaches may oversimplify the underlying problem and hence produce suboptimal results. This paper serves as a guide to the application of multi-objective methods to difficult problems, and is aimed at researchers who are already familiar with single-objective reinforcement learning and planning methods who wish to adopt a multi-objective perspective on their research, as well as practitioners who encounter multi-objective decision problems in practice. It identifies the factors that may influence the nature of the desired solution, and illustrates by example how these influence the design of multi-objective decision-making systems for complex problems.

Details DOI

JAAMAS Journal 2022 Journal Article

On nash equilibria in normal-form games with vectorial payoffs

Willem Röpke
Diederik M. Roijers
Roxana Rădulescu

Abstract We provide an in-depth study of Nash equilibria in multi-objective normal-form games (MONFGs), i. e. , normal-form games with vectorial payoffs. Taking a utility-based approach, we assume that each player’s utility can be modelled with a utility function that maps a vector to a scalar utility. In the case of a mixed strategy, it is meaningful to apply such a scalarisation both before calculating the expectation of the payoff vector as well as after. This distinction leads to two optimisation criteria. With the first criterion, players aim to optimise the expected value of their utility function applied to the payoff vectors obtained in the game. With the second criterion, players aim to optimise the utility of expected payoff vectors given a joint strategy. Under this latter criterion, it was shown that Nash equilibria need not exist. Our first contribution is to provide a sufficient condition under which Nash equilibria are guaranteed to exist. Secondly, we show that when Nash equilibria do exist under both criteria, no equilibrium needs to be shared between the two criteria, and even the number of equilibria can differ. Thirdly, we contribute a study of pure strategy Nash equilibria under both criteria. We show that when assuming quasiconvex utility functions for players, the sets of pure strategy Nash equilibria under both optimisation criteria are equivalent. This result is further extended to games in which players adhere to different optimisation criteria. Finally, given these theoretical results, we construct an algorithm to compute all pure strategy Nash equilibria in MONFGs where players have a quasiconvex utility function.

Details DOI

KER Journal 2020 Journal Article

A utility-based analysis of equilibria in multi-objective normal-form games

Roxana Rădulescu
Patrick Mannion
Yijie Zhang
Diederik M. Roijers
Ann Nowé

Abstract In multi-objective multi-agent systems (MOMASs), agents explicitly consider the possible trade-offs between conflicting objective functions. We argue that compromises between competing objectives in MOMAS should be analyzed on the basis of the utility that these compromises have for the users of a system, where an agent’s utility function maps their payoff vectors to scalar utility values. This utility-based approach naturally leads to two different optimization criteria for agents in a MOMAS: expected scalarized returns (ESRs) and scalarized expected returns (SERs). In this article, we explore the differences between these two criteria using the framework of multi-objective normal-form games (MONFGs). We demonstrate that the choice of optimization criterion (ESR or SER) can radically alter the set of equilibria in a MONFG when nonlinear utility functions are used.

Details DOI

KER Journal 2020 Journal Article

Toll-based reinforcement learning for efficient equilibria in route choice

Gabriel de O. Ramos
Bruno C. da Silva
Roxana Rădulescu
Ana L. C. Bazzan
Ann Nowé

Abstract The problem of traffic congestion incurs numerous social and economical repercussions and has thus become a central issue in every major city in the world. For this work we look at the transportation domain from a multiagent system perspective, where every driver can be seen as an autonomous decision-making agent. We explore how learning approaches can help achieve an efficient outcome, even when agents interact in a competitive environment for sharing common resources. To this end, we consider the route choice problem, where self-interested drivers need to independently learn which routes minimise their expected travel costs. Such a selfish behaviour results in the so-called user equilibrium, which is inefficient from the system’s perspective. In order to mitigate the impact of selfishness, we present Toll-based Q-learning (TQ-learning, for short). TQ-learning employs the idea of marginal-cost tolling (MCT), where each driver is charged according to the cost it imposes on others. The use of MCT leads agents to behave in a socially desirable way such that the is attainable. In contrast to previous works, however, our tolling scheme is distributed (i.e., each agent can compute its own toll), is charged a posteriori (i.e., at the end of each trip), and is fairer (i.e., agents pay exactly their marginal costs). Additionally, we provide a general formulation of the toll values for univariate, homogeneous polynomial cost functions. We present a theoretical analysis of TQ-learning, proving that it converges to a system-efficient equilibrium (i.e., an equilibrium aligned to the system optimum) in the limit. Furthermore, we perform an extensive empirical evaluation on realistic road networks to support our theoretical findings, showing that TQ-learning indeed converges to the optimum, which translates into a reduction of the congestion levels by 9.1%, on average.

Details DOI

JAAMAS Journal 2019 Journal Article

Multi-objective multi-agent decision making: a utility-based analysis and survey

Roxana Rădulescu
Patrick Mannion
Ann Nowé

Abstract The majority of multi-agent system implementations aim to optimise agents’ policies with respect to a single objective, despite the fact that many real-world problem domains are inherently multi-objective in nature. Multi-objective multi-agent systems (MOMAS) explicitly consider the possible trade-offs between conflicting objective functions. We argue that, in MOMAS, such compromises should be analysed on the basis of the utility that these compromises have for the users of a system. As is standard in multi-objective optimisation, we model the user utility using utility functions that map value or return vectors to scalar values. This approach naturally leads to two different optimisation criteria: expected scalarised returns (ESR) and scalarised expected returns (SER). We develop a new taxonomy which classifies multi-objective multi-agent decision making settings, on the basis of the reward structures, and which and how utility functions are applied. This allows us to offer a structured view of the field, to clearly delineate the current state-of-the-art in multi-objective multi-agent decision making approaches and to identify promising directions for future research. Starting from the execution phase, in which the selected policies are applied and the utility for the users is attained, we analyse which solution concepts apply to the different settings in our taxonomy. Furthermore, we define and discuss these solution concepts under both ESR and SER optimisation criteria. We conclude with a summary of our main findings and a discussion of many promising future research directions in multi-objective multi-agent systems.

Details DOI