Ana Bazzan

NeurIPS Conference 2025 Conference Paper

Constructing an Optimal Behavior Basis for the Option Keyboard

• Lucas N. Alegre
• Ana Bazzan
• Andre Barreto
• Bruno Silva

Multi-task reinforcement learning aims to quickly identify solutions for new tasks with minimal or no additional interaction with the environment. Generalized Policy Improvement (GPI) addresses this by combining a set of base policies to produce a new one that is at least as good—though not necessarily optimal—as any individual base policy. Optimality can be ensured, particularly in the linear-reward case, via techniques that compute a Convex Coverage Set (CCS). However, these are computationally expensive and do not scale to complex domains. The Option Keyboard (OK) improves upon GPI by producing policies that are at least as good—and often better. It achieves this through a learned meta-policy that dynamically combines base policies. However, its performance critically depends on the choice of base policies. This raises a key question: is there an optimal set of base policies—an optimal behavior basis —that enables zero-shot identification of optimal solutions for any linear tasks? We solve this open problem by introducing a novel method that efficiently constructs such an optimal behavior basis. We show that it significantly reduces the number of base policies needed to ensure optimality in new tasks. We also prove that it is strictly more expressive than a CCS, enabling particular classes of non-linear tasks to be solved optimally. We empirically evaluate our technique in challenging domains and show that it outperforms state-of-the-art approaches, increasingly so as task complexity increases.

NeurIPS Conference 2023 Conference Paper

A Toolkit for Reliable Benchmarking and Research in Multi-Objective Reinforcement Learning

• Florian Felten
• Lucas N. Alegre
• Ann Nowe
• Ana Bazzan
• El Ghazali Talbi
• Grégoire Danoy
• Bruno C. da Silva

Multi-objective reinforcement learning algorithms (MORL) extend standard reinforcement learning (RL) to scenarios where agents must optimize multiple---potentially conflicting---objectives, each represented by a distinct reward function. To facilitate and accelerate research and benchmarking in multi-objective RL problems, we introduce a comprehensive collection of software libraries that includes: (i) MO-Gymnasium, an easy-to-use and flexible API enabling the rapid construction of novel MORL environments. It also includes more than 20 environments under this API. This allows researchers to effortlessly evaluate any algorithms on any existing domains; (ii) MORL-Baselines, a collection of reliable and efficient implementations of state-of-the-art MORL algorithms, designed to provide a solid foundation for advancing research. Notably, all algorithms are inherently compatible with MO-Gymnasium; and(iii) a thorough and robust set of benchmark results and comparisons of MORL-Baselines algorithms, tested across various challenging MO-Gymnasium environments. These benchmarks were constructed to serve as guidelines for the research community, underscoring the properties, advantages, and limitations of each particular state-of-the-art method.

NeurIPS Conference 2023 Conference Paper

Multi-Step Generalized Policy Improvement by Leveraging Approximate Models

• Lucas N. Alegre
• Ana Bazzan
• Ann Nowe
• Bruno C. da Silva

We introduce a principled method for performing zero-shot transfer in reinforcement learning (RL) by exploiting approximate models of the environment. Zero-shot transfer in RL has been investigated by leveraging methods rooted in generalized policy improvement (GPI) and successor features (SFs). Although computationally efficient, these methods are model-free: they analyze a library of policies---each solving a particular task---and identify which action the agent should take. We investigate the more general setting where, in addition to a library of policies, the agent has access to an approximate environment model. Even though model-based RL algorithms can identify near-optimal policies, they are typically computationally intensive. We introduce $h$-GPI, a multi-step extension of GPI that interpolates between these extremes---standard model-free GPI and fully model-based planning---as a function of a parameter, $h$, regulating the amount of time the agent has to reason. We prove that $h$-GPI's performance lower bound is strictly better than GPI's, and show that $h$-GPI generally outperforms GPI as $h$ increases. Furthermore, we prove that as $h$ increases, $h$-GPI's performance becomes arbitrarily less susceptible to sub-optimality in the agent's policy library. Finally, we introduce novel bounds characterizing the gains achievable by $h$-GPI as a function of approximation errors in both the agent's policy library and its (possibly learned) model. These bounds strictly generalize those known in the literature. We evaluate $h$-GPI on challenging tabular and continuous-state problems under value function approximation and show that it consistently outperforms GPI and state-of-the-art competing methods under various levels of approximation errors.

AAMAS Conference 2019 Conference Paper

A Context-aware Convention Formation Framework for Large-Scale Networks

• Mohammad Rashedul Hasan
• Anita Raja
• Ana Bazzan

Conventions can serve as a useful mechanism for deciding the dominant coordination strategy and facilitating consensus in a multiagent system (MAS). In this paper, we present a decentralized convention formation framework that harnesses the structural properties and diversity of the network for creating social conventions within large and open multiagent convention spaces. We validate our convention formation framework using a language coordination problem in which agents in a MAS construct a common lexicon in a decentralized fashion on various networks. Experimentation results indicate that our approach is both effective (able to converge into a large majority convention state with more than 90% agents sharing a high-quality lexicon) and efficient (faster) as compared to state-of-the-art approaches for social conventions in large convention spaces.

JAAMAS Journal 2018 Journal Article

A context-aware convention formation framework for large-scale networks

• Mohammad Rashedul Hasan
• Anita Raja
• Ana Bazzan

Abstract In this article, we present a decentralized convention formation framework for creating social conventions within large multiagent convention spaces. We study the role of the topological characteristics of the network in forming conventions with an emphasis on scale-free topologies. We hypothesize that contextual knowledge encapsulated in the topology can help improve both the quality of the emergent convention and the speed of forming such a convention. We also investigate the influence of network diversity. While recent research on diversity indicates that it improves organizational productivity, we observe that not all diversity is equally useful and identify the necessary conditions to maximize the benefit of diversity. We validate our convention formation framework using a language coordination problem in which agents in a multiagent system construct a common lexicon in a decentralized fashion. Agent interactions are modeled using a language game where every agent repeatedly plays with its neighbors. Each agent stochastically updates its lexicon based on the utility values of the lexicons received from its immediate neighbors. We introduce a novel context-aware utility computation mechanism and equip the agents with the ability to reorganize their neighborhood based on this utility estimate to expedite the convention formation process. A key idea behind our approach is the ability of socially influential high-utility-lexicon agents to bias their neighbors towards accepting their lexicons. Extensive experimentation results indicate that our proposed solution is both effective (able to converge into a large majority convention state with more than 90% agents sharing a high-quality lexicon) and efficient (faster) as compared to state-of-the-art approaches for social conventions in large convention spaces.

AAAI Conference 2015 Conference Paper

Fast Convention Formation in Dynamic Networks Using Topological Knowledge

• Mohammad Hasan
• Anita Raja
• Ana Bazzan

In this paper, we design a distributed mechanism that is able to create a social convention within a large convention space for multiagent systems (MAS) operating on various topologies. Specifically, we investigate a language coordination problem in which agents in a dynamic MAS construct a common lexicon in a decentralized fashion. Agent interactions are modeled using a language game where every agent repeatedly plays with its neighbors. Each agent stochastically updates its lexicons based on the utility values of the received lexicons from its immediate neighbors. We present a novel topology-aware utility computation mechanism and equip the agents with the ability to reorganize their neighborhood based on this utility estimate to expedite the convention formation process. Extensive simulation results indicate that our proposed mechanism is both effective (able to converge into a large majority convention state with more than 90% agents sharing a high-quality lexicon) and efficient (faster) as compared to state-of-the-art approaches for social conventions in large convention spaces.

AAMAS Conference 2012 Conference Paper

Agent-based simulation of mobility in real-world transportation networks

• Maicon Amarante
• Ana Bazzan

Macroscopic and equilibrium-based models for traffic assignment and simulation disregard many details of traffic movement. For some applications, one needs to understand and analyze microscopic properties. This paper discusses an agent-based simulation of route choice under different conditions of demand generation, number, and types of travelers. The effects of en-route decision-making and vehicle-tovehicle communication were tested in a real-world scenario. The analysis has considered different classes of travelers, which is only possible if a microscopic, agent-based simulation is used. The main conclusion is that for travelers whose trips are long, there is a benefit of using communication and replan en-route, depending on the demand volume.

AAMAS Conference 2012 Conference Paper

Break with agents who listen to too many others (at least when making Boolean decisions! )

• Daniel Epstein
• Ana Bazzan
• Andr
• eacute; Machado

In multiagent scenarios where decision-makers have to coordinate actions (e. g. , minority and congestion games), previous works have shown that agents may reach coordination mostly by looking at past decisions. Not many works consider the structure behind agents’ connections. When structure is indeed considered, it assumes some kind of random network with a given, fixed connectivity degree. The present paper departs from this approach mainly as follows. First, it considers network topologies based on preferential attachments (especially useful in social networks). Second, the formalism of random Boolean networks is used to allow agents to consider their acquaintances. Our results using preferential attachments and random Boolean networks show that an efficient equilibrium can be achieved, provided agents do experimentation. Also, we show that influential agents tend to consider few inputs in their Boolean functions.

AAMAS Conference 2012 Conference Paper

Combining Independent and Joint Learning: a Negotiation based Approach

• Reinaldo Bianchi
• Ana Bazzan

This work presents a new class of multiagent reinforcement learning algorithms that takes advantage of negotiation in order to improve the process of action selection. In this class of algorithms, agents use communication to cooperate and negotiate over the joint actions, thus enhancing the process of action selection. In this paper a new algorithm in this class is proposed: the Negotiation-based Q-Learning (NQL), which uses negotiation in the context of the Q-Learning algorithm. Results show that allowing negotiation between agents significantly enhances the performance of the multiagent learning process.

AAMAS Conference 2010 Conference Paper

Distributed Clustering for Group Formation and Task Allocation

• Daniela Santos
• Ana Bazzan

Most clustering methods rely on central data structures and/or cannot cope with dynamically changing settings. However, issues related to the current use of Internet resources (distribution of data, privacy, etc. ) require new ways of dealing with data clustering. Inmultiagent systems this is also becoming an issue as one wishesto group agents in an efficient way and according to some featuresof the environment. In this paper we briefly discuss how a distributed clustering algorithm that is inspired by swarm intelligencetechniques is used in problems of task allocation.

AAMAS Conference 2008 Conference Paper

Evaluating the Performance of DCOP Algorithms in a Real World, Dynamic Problem

• Robert Junges
• Ana Bazzan

Complete algorithms have been proposed to solve problems modelled as distributed constraint optimization (DCOP). However, there are only few attempts to address real world scenarios using this formalism, mainly because of the complexity associated with those algorithms. In the present work we compare three complete algorithms for DCOP, aiming at studying how they perform in complex and dynamic scenarios of increasing sizes. In order to assess their performance we measure not only standard quantities such as number of cycles to arrive to a solution, size and quantity of exchanged messages, but also computing time and quality of the solution which is related to the particular domain we use. This study can shed light in the issues of how the algorithms perform when applied to problems other than those reported in the literature (graph coloring, meeting scheduling, and distributed sensor network).

AAMAS Conference 2008 Conference Paper

Simulating the Effects of Sanction for the Emergence of Cooperation in a Public Goods Game

• Ana Bazzan
• Silvio Dahmen
• Alexandre Baraviera

Several explanations have been proposed in order to explain why, in public goods games, cooperation does not collapse. In these games free-riders enjoy the benefits of other individuals who contribute in benefit of a community. In the present work we address a public goods game where individuals have the choice between contributing to a sanctioning institution and to a sanction-free one. In the former there is a possible sanction for those who do not contribute. Our results show that individuals who contribute to a sanctioning institution are better off after several repetitions of the game, despite the costs associated with sanctioning. This reproduces results found in experiments with human subjects, which point to advantages of sanctioning measures as a factor for the stabilization of cooperation.

AAMAS Conference 2006 Conference Paper

A General, Agent-Based Model of Pedestrian Dynamics: from Lane Formation to Auditorium Evacuation

• Marcelo Toyama
• Ana Bazzan
• Roberto da Silva

AAMAS Conference 2006 Conference Paper

Congestion Tolls as Utility Alignment Between Agent and System Optimum

• Ana Bazzan
• Robert Junges

AAMAS Conference 2005 Conference Paper

Emerging Collective Behavior in a Simple Artificial Financial Market

• Roberto da Silva
• Ana Bazzan
• Alexandre Baraviera
• S
• iacute; lvio Dahmen

AAMAS Conference 2005 Conference Paper

Using Cooperative Mediation to Coordinate Traffic Lights: a Case Study

• Denise de Oliveira
• Ana Bazzan
• Victor Lesser