Natalia Criado

AAMAS Conference 2026 Conference Paper

Decentralized Value Systems Agreements

• Arturo Hernández-Sánchez
• Natalia Criado
• Stella Heras
• Miguel Rebollo
• Jose Such

One of the biggest challenges of value-based decision-making is dealing with the subjective nature of values. The relative importance of a value for a particular decision varies between individuals, and people may also have different interpretations of what aligning with a value means in a given situation. While members of a society are likely to share a set of principles or values, their value systems—that is, how they interpret these values and the relative importance they give to them—have been found to differ significantly. This work proposes a novel method for aggregating value systems, generating distinct value agreements that accommodate the inherent differences within these systems. Unlike existing work, which focuses on finding a single value agreement, the proposed approach may be more suitable for a realistic and heterogeneous society. In our solution, the agents indicate their value systems and the extent to which they are willing to concede. Then, a set of agreements is found, taking a decentralized optimization approach. Our work has been applied to identify value agreements in two real-world scenarios using data from a Participatory Value Evaluation process and a European Value Survey. These case studies illustrate the different aggregations that can be obtained with our method and compare them with those obtained using existing value system aggregation techniques. In both cases, the results showed a substantialimprovementinindividualutilitiescomparedtoexisting alternatives.

AAAI Conference 2024 Conference Paper

Moral Uncertainty and the Problem of Fanaticism

• Jazon Szabo
• Natalia Criado
• Jose Such
• Sanjay Modgil

While there is universal agreement that agents ought to act ethically, there is no agreement as to what constitutes ethical behaviour. To address this problem, recent philosophical approaches to `moral uncertainty' propose aggregation of multiple ethical theories to guide agent behaviour. However, one of the foundational proposals for aggregation - Maximising Expected Choiceworthiness (MEC) - has been criticised as being vulnerable to fanaticism; the problem of an ethical theory dominating agent behaviour despite low credence (confidence) in said theory. Fanaticism thus undermines the `democratic' motivation for accommodating multiple ethical perspectives. The problem of fanaticism has not yet been mathematically defined. Representing moral uncertainty as an instance of social welfare aggregation, this paper contributes to the field of moral uncertainty by 1) formalising the problem of fanaticism as a property of social welfare functionals and 2) providing non-fanatical alternatives to MEC, i.e. Highest k-trimmed Mean and Highest Median.

AAMAS Conference 2024 Conference Paper

Multi-user Norm Consensus

• Marc Serramia
• Natalia Criado
• Michael Luck

Many agents act in environments with multiple human users, from care robots to smart assistants. When interacting in multi-user environments it is paramount that these agents act as all users expect. However, it is not always possible to have well-defined collective preferences, nor to easily infer them from individual preferences. This is especially true in fast changing environments, like a device placed in a public space where users can enter and exit freely. In response, this paper proposes a model to represent individual preferences about the behaviour of an agent and a mechanism to find multi-user consensuses over these preferences. Norms can then be generated to ensure that when the agent follows them it will act according to the preferences of all users. We formalise what a consensus norm is and what properties the set of consensus norms should satisfy (i. e. generate the minimum number of norms while maximising the coverage of user preferences). We provide an optimisation approach to find this set of norms and show that our approach satisfies the aforementioned properties.

JAIR Journal 2023 Journal Article

A Model to Support Collective Reasoning: Formalization, Analysis and Computational Assessment

• Jordi Ganzer
• Natalia Criado
• Maite Lopez-Sanchez
• Simon Parsons
• Juan A. Rodriguez-Aguilar

In this paper we propose a new model to represent human debates and methods to obtain collective conclusions from them. This model overcomes two drawbacks of existing approaches. First, our model does not assume that participants agree on the structure of the debate. It does this by allowing participants to express their opinion about all aspects of the debate. Second, our model does not assume that participants’ opinions are rational, an assumption that significantly limits current approaches. Instead, we define a weaker notion of rationality that characterises coherent opinions, and we consider different scenarios based on the coherence of individual opinions and the level of consensus. We provide a formal analysis of different opinion aggregation functions that compute a collective decision based on the individual opinions and the debate structure. In particular, we demonstrate that aggregated opinions can be coherent even if there is a lack of consensus and individual opinions are not coherent. We conclude with an empirical evaluation demonstrating that collective opinions can be computed efficiently for real-sized debates.

AAMAS Conference 2023 Conference Paper

Predicting Privacy Preferences for Smart Devices as Norms

• Marc Serramia
• William Seymour
• Natalia Criado
• Michael Luck

Smart devices, such as smart speakers, are becoming ubiquitous, and users expect these devices to act in accordance with their preferences. In particular, since these devices gather and manage personal data, users expect them to adhere to their privacy preferences. However, the current approach of gathering these preferences consists in asking the users directly, which usually triggers automatic responses failing to capture their true preferences. In response, in this paper we present a collaborative filtering approach to predict user preferences as norms. These preference predictions can be readily adopted or can serve to assist users in determining their own preferences. Using a dataset of privacy preferences of smart assistant users, we test the accuracy of our predictions.

EUMAS Conference 2022 Conference Paper

Integrating Quantitative and Qualitative Reasoning for Value Alignment

• Jazon Szabo
• Jose M. Such
• Natalia Criado
• Sanjay Modgil

Abstract Agents that focus only on achieving their own goals may cause significant harm to society. As a result, when deciding which actions to perform, agents have to consider societal values and how their actions impact these values—the ‘value alignment problem’. There is therefore a need to integrate quantitative machine reasoning with an ability to reason about qualitative human values. In this paper, we present a novel framework for value-based reasoning that aims to bridge the gap between these two modes of reasoning. In particular, our framework extends the theory of grading to model how societal values can trade off with each other or with the agent’s goals. Furthermore, our framework introduces the use of hyperreal numbers to represent both quantitative and qualitative aspects of reasoning and help address the value alignment problem.

EUMAS Conference 2020 Conference Paper

Understanding the Role of Values and Norms in Practical Reasoning

• Jazon Szabo
• Jose M. Such
• Natalia Criado

Abstract Mistrust poses a significant threat to the widespread adoption of intelligent agents. Guarantees about the behaviour of intelligent systems could help foster trust. In this paper, we investigate mechanisms to integrate value-based reasoning in practical reasoning as a way to ensure that agents actions align not only with society norms but also with user’s values. In particular, we expand a normative BDI agent architecture with an explicit representation of values.

JAIR Journal 2018 Journal Article

Resource-bounded Norm Monitoring In Multi-agent Systems

• Natalia Criado

Norms allow system designers to specify the desired behaviour of a sociotechnical system. In this way, norms regulate what the social and technical agents in a sociotechnical system should (not) do. In this context, a vitally important question is the development of mechanisms for monitoring whether these agents comply with norms. Proposals on norm monitoring often assume that monitoring has no costs and/or that monitors have unlimited resources to observe the environment and the actions performed by agents. In this paper, we challenge this assumption and propose the first practical resource-bounded norm monitor. Our monitor is capable of selecting the resources to be deployed and use them to check norm compliance with incomplete information about the actions performed and the state of the world. We formally demonstrate the correctness and soundness of our norm monitor and study its complexity. We also demonstrate in randomised simulations and benchmark experiments that our monitor can select monitored resources effectively and efficiently, detecting more norm violations and fulfilments than other tractable optimization approaches and obtaining slightly worse results than intractable optimal approaches.

AAMAS Conference 2017 Conference Paper

A Practical Resource-Constrained Norm Monitor

• Natalia Criado

IS Journal 2016 Journal Article

Intelligent Cybersecurity Agents [Guest editors' introduction]

• Jose M. Such
• Natalia Criado
• Laurent Vercouter
• Martin Rehak

The cross-fertilization between agent technologies and cybersecurity is giving rise to a number of intelligent approaches that can help mitigate the ever-growing cyber threats individuals and organizations face online. The guest editors outline some examples of agent technologies and how they can be useful for (or are already being applied to) the cybersecurity domain.

IJCAI Conference 2016 Conference Paper

Selective Norm Monitoring

• Natalia Criado
• Jose M. Such

Real-world norm monitors have limited capabilities for observing agents. This paper proposes a novel mechanism to take full advantage of limited observation capabilities by selecting the agents to be monitored. Our evaluation shows this significantly increases the number of violations detected.

JAAMAS Journal 2015 Journal Article

A coherence maximisation process for solving normative inconsistencies

• Natalia Criado
• Elizabeth Black
• Michael Luck

Abstract Norms can be used in multi-agent systems for defining patterns of behaviour in terms of permissions, prohibitions and obligations that are addressed to agents playing a specific role. Agents may play different roles during their execution and they may even play different roles simultaneously. As a consequence, agents may be affected by inconsistent norms; e. g. , an agent may be simultaneously obliged and forbidden to reach a given state of affairs. Dealing with this type of inconsistency is one of the main challenges of normative reasoning. Existing approaches tackle this problem by using a static and predefined order that determines which norm should prevail in the case where two norms are inconsistent. One main drawback of these proposals is that they allow only pairwise comparison of norms; it is not clear how agents may use the predefined order to select a subset of norms to abide by from a set of norms containing multiple inconsistencies. Furthermore, in dynamic and non-deterministic environments it can be difficult or even impossible to specify an order that resolves inconsistencies satisfactorily in all potential situations. In response to these two problems, we propose a mechanism with which an agent can dynamically compute a preference order over subsets of its competing norms by considering the coherence of its cognitive and normative elements. Our approach allows flexible resolution of normative inconsistencies, tailored to the current circumstances of the agent. Moreover, our solution can be used to determine norm prevalence among a set of norms containing multiple inconsistencies.

AAMAS Conference 2012 Conference Paper

Determining the Willingness to Comply With Norms

• Natalia Criado
• Estefan
• iacute; a Argente
• Pablo Noriega
• Vicent Botti

In this paper, we propose that agents make decisions about norm compliance based on three different factors: self-interest, enforcement mechanisms and internalised emotions. Different agent personalities can be defined according to the importance given to each factor.

AAMAS Conference 2011 Conference Paper

Reasoning About Norm Compliance

• Natalia Criado
• Estefania Argente
• Vicente Botti
• Pablo Noriega

This paper proposes a reasoning process to allow agents to decide when and how norms should be violated or obeyed. The coherence-based reasoning mechanism proposed in this paper, allows norm aware agents to confront the norm compliance dilemma and build alternatives for such normative decisions.

AAMAS Conference 2011 Conference Paper

Reasoning About Norms Within Uncertain Environments

• Natalia Criado

The main aim of my thesis is the development of agents capable of reasoning about norms given that they are situated in an uncertain environment. The n-BDI agent architecture developed in my thesis is aimed at allowing agents to determine which and how norms will be obeyed and supporting agents when facing with norm violations.

AAMAS Conference 2010 Conference Paper

A BDI Architecture for Normative Decision Making

• Natalia Criado
• Estefan
• iacute; a Argente
• Vicente Botti

Norms, to become effective, must be recognised as normsby agents. These agents must be able to accept normsbut maintaining their autonomy. In this paper, the multi-context BDI agent architecture has been extended with arecognition and a normative context in order to allow agentsto acquire norms from their environment and consider normsin their decision making processes.

AAMAS Conference 2010 Conference Paper

THOMAS: A Service-Oriented Framework For Virtual Organizations

• Elena Del Val
• Natalia Criado
• Carlos Carrascosa
• Vicente Julian
• Miguel Rebollo
• Estefania Argente
• Vicente Botti

This paper describes the THOMAS framework, a useful framework for the development of virtual organizations, on thebasis of a service-based approach.