Mehdi Dastani

CLeaR Conference 2026 Conference Paper

Abstractions in Causal Models and Game Structures

• Sylvia S. Kerkhove
• Natasha Alechina
• Mehdi Dastani

We investigate abstractions in causal and strategic models of multi-agent systems, exploiting the relationship between these models. The paper contains two main results. The first one demonstrates that abstraction in causal models is a faithful correspondent of abstraction in strategic models, i. e. , for a given causal model, if we generate a corresponding strategic model and abstract this model, we will obtain the same model as when we abstract the causal model first and then generate its corresponding strategic model. The second result is that a causal dependency in an abstract (high-level) model entails a causal dependency in the original (low-level) model. This allows us to reason about causes in a simpler abstract model and derive conclusions about causality in the much larger low-level model. These results set the stage for studying and using abstractions of causal models in multi-agent settings.

AAMAS Conference 2026 Conference Paper

Neuro-symbolic Action Masking for Deep Reinforcement Learning

• Shuai Han
• Mehdi Dastani
• Shihan Wang

Deep reinforcement learning (DRL) may explore infeasible actions during training and execution. Existing approaches assume a symbol grounding function that maps high-dimensional states to consistent symbolic representations and a manually specified action masking techniques to constrain actions. In this paper, we propose Neuro-symbolic Action Masking (NSAM), a novel framework that automatically learn symbolic models, which are consistent with givendomainconstraintsofhigh-dimensionalstates, inaminimally supervised manner during the DRL process. Based on the learned symbolic model of states, NSAM learns action masks that rules out infeasible actions. NSAM enables end-to-end integration of symbolic reasoning and deep policy optimization, where improvements in symbolic grounding and policy learning mutually reinforce each other. We evaluate NSAM on multiple domains with constraints, and experimental results demonstrate that NSAM significantly improves sample efficiency of DRL agent while substantially reducing constraint violations.

AAAI Conference 2026 Conference Paper

Rational Revision of Group Intentions

• Nima Motamed
• Natasha Alechina
• Mehdi Dastani
• Dragan Doder

In systems such as group calendars or collaborative platforms, agents make group commitments to future actions that must adapt as new facts or constraints emerge. We develop a formal framework for revising such group intentions in systems where coalitions adopt shared, temporally extended intentions represented in a logic based on Alternating-Time Temporal Logic with strategy contexts. After formulating coherence criteria for systems of group intentions, we establish representation theorems in the style of Katsuno and Mendelzon, showing that revision operators satisfy rationality postulates precisely when they can be represented by preorders on strategy profiles. These results extend classical revision theory by covering non-total preorders and a logic of higher expressive power. Altogether, the framework lays the groundwork for principled revision of group intentions in systems where both coordination and change are essential.

AAMAS Conference 2026 Conference Paper

Reasoning About Responsibility for Taking Risks

• Maksim Gladyshev
• Natasha Alechina
• Mehdi Dastani
• Dragan Doder

Tracing responsibility and assigning blame to decision-making actorsinmulti-agentsystemshasdrawnattentionofMAScommunity in recent years. Existing approaches have proposed several definitions for multi-agent responsibility, but while these definitions differ in details, most of them agree that responsibility for some event may be allocated to agents only if the event is actually realized. In this paper, we argue that in many scenarios this restriction is too strong and that an undesirable outcome may be understood as a (high) risk of some event, and not necessarily as the realization of the event. To cover such cases, we propose a logical framework for reasoning about various notions of multi-agent responsibility for taking risks. The proposed logic contains primitives for probability, strategic power, time and knowledge modalities, which, as we demonstrate, allow to express various notions of group responsibility in probabilistic settings. As the main result we prove that the proposed logic has a complete axiomatization, a decidable satisfiability problem, and an efficient model-checking procedure.

JAIR Journal 2026 Journal Article

Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning

• Giovanni Varricchione
• Natasha Alechina
• Mehdi Dastani
• Brian Logan

Reward machines have recently been proposed as a means of encoding team tasks in cooperative multi-agent reinforcement learning. The resulting multi-agent reward machine is then decomposed into individual reward machines, one for each member of the team, allowing agents to learn in a decentralised manner while still achieving the team task. In this paper, we show how multi-agent reward machines for team tasks can be synthesised automatically from an abstraction of the environment in which the agents act and a high-level specification of the desired team behaviour expressed in a fragment of Alternating-time Temporal Logic. We present results from a number of benchmarks which suggest that our automated approach performs as well or better than reward machines in the literature.

AAMAS Conference 2025 Conference Paper

Causes and Strategies in Multiagent Systems

• Sylvia S. Kerkhove
• Natasha Alechina
• Mehdi Dastani

Causality plays an important role in daily processes, human reasoning, and artificial intelligence. There has however not been much research on causality in multi-agent strategic settings. In this work, we introduce a systematic way to build a multi-agent system model, represented as a concurrent game structure, for a given structural causal model. In the obtained so-called causal concurrent game structure, transitions correspond to interventions on agent variables of the given causal model. The Halpern and Pearl framework of causality is used to determine the effects of a certain value for an agent variable on other variables. The causal concurrent game structure allows us to analyse and reason about causal effects of agents’ strategic decisions. We formally investigate the relation between causal concurrent game structures and the original structural causal models.

EWRL Workshop 2025 Workshop Paper

Learning Reward Structure with Subtasks in Reinforcement Learning

• Shuai Han
• Mehdi Dastani
• Shihan Wang

Improving sample efficiency of Reinforcement Learning (RL) in sparse-reward environments poses a significant challenge. In scenarios where the reward structure is complex, accurate action evaluation often relies heavily on precise information about past achieved subtasks and their order. Previous approaches have often failed or proved inefficient in constructing and leveraging such intricate reward structures. In this work, we propose an RL algorithm that can automatically structure the reward function for sample efficiency, given a set of labels that signify subtasks. Given such minimal knowledge about the task, we train a high-level policy that selects optimal subtasks in each state together with a low-level policy that efficiently learns to complete each sub-task. We evaluate our algorithm in a variety of sparse-reward environments. The experiment results show that our method significantly outperforms the state-of-art baselines as the difficulty of the task increases.

KR Conference 2025 Conference Paper

Pushdown Reward Machines for Reinforcement Learning

• Giovanni Varricchione
• Toryn Q. Klassen
• Natasha Alechina
• Mehdi Dastani
• Brian Logan
• Sheila A. McIlraith

Reward machines (RMs) are automata structures that encode (non-Markovian) reward functions for reinforcement learning (RL). RMs can reward any behaviour representable in regular languages and, when paired with RL algorithms that exploit RM structure, have been shown to significantly improve sample efficiency in many domains. In this work, we present pushdown reward machines (pdRMs), an extension of reward machines based on deterministic pushdown automata. pdRMs can recognise and reward temporally extended behaviours representable in deterministic context-free languages, making them more expressive than reward machines. We introduce two variants of pdRM-based policies, one which has access to the entire stack of the pdRM, and one which can only access the top k symbols (for a given constant k) of the stack. We propose a procedure to check when the two kinds of policies (for a given environment, pdRM, and constant k) achieve the same optimal state values. We then provide theoretical results establishing the expressive power of pdRMs, and space complexity results for the proposed learning problems. Lastly, we propose an approach for off-policy RL algorithms that exploits counterfactual experiences with pdRMs. We conclude by providing experimental results showing how agents can be trained to perform tasks representable in deterministic context-free languages using pdRMs.

I&C Journal 2025 Journal Article

Reasoning about group responsibility for exceeding risk threshold in one-shot games

• Maksim Gladyshev
• Natasha Alechina
• Mehdi Dastani
• Dragan Doder

Tracing and analysing the responsibility for unsafe outcomes of actors' decisions in multi-agent settings have been studied in recent years. These studies often focus on deterministic scenarios and assume that the unsafe outcomes for which actors can be held responsible are actually realized. This paper considers a broader notion of responsibility where unsafe outcomes are not necessarily realized, but their probabilities are unacceptably high. We present a logic combining strategic, probabilistic and temporal primitives designed to express concepts such as the risk of an undesirable outcome and being responsible for exceeding a risk threshold in one-shot games. We demonstrate that the proposed logic is (weakly) complete, decidable and has an efficient model-checking procedure. Finally, we define a probabilistic notion of responsibility and study its formal properties in the proposed logic setting.

AAMAS Conference 2025 Conference Paper

Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learning

• Changxi Zhu
• Mehdi Dastani
• Shihan Wang

In decentralized multi-agent deep reinforcement learning (MADRL), communication can help agents to gain a better understanding of the environment to better coordinate their behaviors. Nevertheless, communication may involve uncertainty, which potentially introduces variance to the learning of decentralized agents. In this extended abstract, we report on our research that focuses on a specific decentralized MADRL setting with communication and a theoretical analysis to study the variance caused by communication in policy gradients. We argue for modular techniques to reduce the variance in policy gradients during training. We show a pseudo algorithm to illustrate the integration of the modular techniques into existing decentralized MADRL with communication methods.

AAAI Conference 2025 Conference Paper

Temporal Causal Reasoning with (Non-Recursive) Structural Equation Models

• Maksim Gladyshev
• Natasha Alechina
• Mehdi Dastani
• Dragan Doder
• Brian Logan

Structural equation models (SEM) are a standard approach to representing causal dependencies between variables. In this paper we propose a new interpretation of existing formalisms in the field of Actual Causality in which SEM's are viewed as mechanisms transforming the dynamics of exogenous variables into the dynamics of endogenous variables. This allows us to combine counterfactual causal reasoning with existing temporal logic formalizms, and to introduce a temporal logic, CPLTL, for causal reasoning about such structures. Then, we demonstrate that the standard restriction to so-called recursive models (with no cycles in the dependency graphs) is not necessary in our approach. This fact provides us extra tools for reasoning about mutually dependent processes and feedback loops. Finally, we introduce the notions of model equivalence for temporal causal models and show that CPLTL has an efficient model-checking procedure.

UAI Conference 2025 Conference Paper

The Causal Information Bottleneck and Optimal Causal Variable Abstractions

• Francisco Nunes Ferreira Quialheiro Simoes
• Mehdi Dastani
• Thijs van Ommen

To effectively study complex causal systems, it is often useful to construct abstractions of parts of the system by discarding irrelevant details while preserving key features. The Information Bottleneck (IB) method is a widely used approach to construct variable abstractions by compressing random variables while retaining predictive power over a target variable. Traditional methods like IB are purely statistical and ignore underlying causal structures, making them ill-suited for causal tasks. We propose the Causal Information Bottleneck (CIB), a causal extension of the IB, which compresses a set of chosen variables while maintaining causal control over a target variable. This method produces abstractions of (sets of) variables which are causally interpretable, give us insight about the interactions between the abstracted variables and the target variable, and can be used when reasoning about interventions. We present experimental results demonstrating that the learned abstractions accurately capture causal relations as intended.

JAAMAS Journal 2024 Journal Article

A survey of multi-agent deep reinforcement learning with communication

• Changxi Zhu
• Mehdi Dastani
• Shihan Wang

Abstract Communication is an effective mechanism for coordinating the behaviors of multiple agents, broadening their views of the environment, and to support their collaborations. In the field of multi-agent deep reinforcement learning (MADRL), agents can improve the overall learning performance and achieve their objectives by communication. Agents can communicate various types of messages, either to all agents or to specific agent groups, or conditioned on specific constraints. With the growing body of research work in MADRL with communication (Comm-MADRL), there is a lack of a systematic and structural approach to distinguish and classify existing Comm-MADRL approaches. In this paper, we survey recent works in the Comm-MADRL field and consider various aspects of communication that can play a role in designing and developing multi-agent reinforcement learning systems. With these aspects in mind, we propose 9 dimensions along which Comm-MADRL approaches can be analyzed, developed, and compared. By projecting existing works into the multi-dimensional space, we discover interesting trends. We also propose some novel directions for designing future Comm-MADRL systems through exploring possible combinations of the dimensions.

AAMAS Conference 2024 Conference Paper

A Survey of Multi-Agent Deep Reinforcement Learning with Communication

• Changxi Zhu
• Mehdi Dastani
• Shihan Wang

Communication is an effective mechanism for coordinating the behaviors of multiple agents, broadening their views of the environment, and to support their collaborations. In the field of multi-agent deep reinforcement learning (MADRL), agents can improve the overall learning performance and achieve their objectives through communication. Agents can communicate various types of messages, either to all agents or to specific agent groups, or conditioned on specific constraints. With the growing body of research work in MADRL with communication (Comm-MADRL), there is a lack of a systematic and structural approach to distinguish and classify existing Comm-MADRL approaches. In this paper, we survey recent works in the Comm-MADRL field and consider various aspects of communication that can play a role in designing and developing multi-agent reinforcement learning systems. With these aspects in mind, we propose 9 dimensions along which Comm-MADRL approaches can be analyzed, developed, and compared. By projecting existing works into the multi-dimensional space, we discover interesting trends. We also propose some novel directions for designing future Comm-MADRL systems through exploring possible combinations of the dimensions.

AAMAS Conference 2024 Conference Paper

Actual Trust in Multiagent Systems

• Michael Akintunde
• Vahid Yazdanpanah
• Asieh Salehi Fathabadi
• Corina Cirstea
• Mehdi Dastani
• Luc Moreau

We study how trust can be established in multiagent systems where human and AI agents collaborate. We propose a computational notion of actual trust, emphasising the modelling of trust based on agents’ capacity to deliver tasks in prospect. Unlike reputationbased trust, we consider the specific setting in which agents interact and model a forward-looking notion of trust. We provide a conceptual analysis of actual trust’s characteristics and highlight relevant trust verification tools. By advancing the understanding and verification of trust in collaborative systems, we contribute to responsible and trustworthy human-AI interactions, enhancing reliability in various domains.

AAMAS Conference 2024 Conference Paper

Bootstrapped Policy Learning: Goal Shaping for Efficient Task-oriented Dialogue Policy Learning

• Yangyang Zhao
• Mehdi Dastani
• Shihan Wang

Reinforcement Learning (RL) shows promise in optimizing taskoriented dialogue policies, but addressing the challenge of reward sparsity remains challenging. Curriculum learning offers an effective solution by strategically training dialogue policies from simple to complex, facilitating a smooth knowledge transition across varied goal complexities. However, these methods typically assume that goal difficulty will increase gradually to adapt to difficult goals over time. In complex environments lacking intermediate goals, attaining smooth knowledge transitions becomes tricky. This paper proposes a novel Bootstrapped Policy Learning (BPL) framework that adaptively tailors a curriculum for each complex goal through goal shaping, which consists of progressively challenging subgoals. Goal shaping comprises goal decomposition and evolution, breaking complex goals into solvable subgoals and progressively increasing subgoal difficulty as the policy improves. BPL harmoniously combines these aspects, enabling smooth knowledge transitions from simple to complex goals, thereby enhancing task-oriented dialogue policy learning efficiency. Our experiments demonstrate the effectiveness of BPL in two complex dialogue environments.

JAAMAS Journal 2024 Journal Article

Correction: A survey of multi-agent deep reinforcement learning with communication

• Changxi Zhu
• Mehdi Dastani
• Shihan Wang

ECAI Conference 2024 Conference Paper

Learning Reward Structure with Subtasks in Reinforcement Learning

• Shuai Han
• Mehdi Dastani
• Shihan Wang 0001

Improving sample efficiency of Reinforcement Learning (RL) in sparse-reward environments poses a significant challenge. In scenarios where the reward structure is complex, accurate action evaluation often relies heavily on precise information about past achieved subtasks and their order. Previous approaches have often failed or proved inefficient in constructing and leveraging such intricate reward structures. In this work, we propose an RL algorithm that can automatically structure the reward function for sample efficiency, given a set of labels that signify subtasks. Given such minimal knowledge about the task, we train a high-level policy that selects optimal subtasks in each state together with a low-level policy that efficiently learns to complete each sub-task. We evaluate our algorithm in a variety of sparse-reward environments. The experiment results show that our method significantly outperforms the state-of-art baselines as the difficulty of the task increases.

ECAI Conference 2024 Conference Paper

Maximally Permissive Reward Machines

• Giovanni Varricchione
• Natasha Alechina
• Mehdi Dastani
• Brian Logan 0001

Reward machines allow the definition of rewards for temporally extended tasks and behaviors. Specifying “informative” reward machines can be challenging. One way to address this is to generate reward machines from a high-level abstract description of the learning environment, using techniques such as AI planning. However, previous planning-based approaches generate a reward machine based on a single (sequential or partial-order) plan, and do not allow maximum flexibility to the learning agent. In this paper we propose a new approach to synthesising reward machines which is based on the set of partial order plans for a goal. We prove that learning using such “maximally permissive” reward machines results in higher rewards than learning using RMs based on a single plan. We present experimental results which support our theoretical claims by showing that our approach obtains higher rewards than the single-plan approach in practice.

EWRL Workshop 2024 Workshop Paper

Model-based Sparse Communication in Multi-agent Reinforcement Learning

• Shuai Han
• Mehdi Dastani
• Shihan Wang

Learning to communicate efficiently is central to multi-agent reinforcement learning (MARL). Existing methods often require agents to exchange messages intensively, which abuses communication channels and leads to high communication overhead. Only a few methods target on learning sparse communication, but they allow limited information to be shared, which affects the efficiency of policy learning. In this work, we propose model-based communication (MBC), a learning framework with a decentralized communication scheduling process. The MBC framework enables multiple agents to make decisions with sparse communication. In particular, the MBC framework introduces a model-based message estimator to estimate the up-to-date global messages using past local data. A decentralized message scheduling mechanism is also proposed to determine whether a message shall be sent based on the estimation. We evaluated our method in a variety of mixed cooperative-competitive environments. The experiment results show that the MBC method shows better performance and lower channel overhead than the state-of-art baselines.

AAAI Conference 2024 Conference Paper

Pure-Past Action Masking

• Giovanni Varricchione
• Natasha Alechina
• Mehdi Dastani
• Giuseppe De Giacomo
• Brian Logan
• Giuseppe Perelli

We present Pure-Past Action Masking (PPAM), a lightweight approach to action masking for safe reinforcement learning. In PPAM, actions are disallowed (“masked”) according to specifications expressed in Pure-Past Linear Temporal Logic (PPLTL). PPAM can enforce non-Markovian constraints, i.e., constraints based on the history of the system, rather than just the current state of the (possibly hidden) MDP. The features used in the safety constraint need not be the same as those used by the learning agent, allowing a clear separation of concerns between the safety constraints and reward specifications of the (learning) agent. We prove formally that an agent trained with PPAM can learn any optimal policy that satisfies the safety constraints, and that they are as expressive as shields, another approach to enforce non-Markovian constraints in RL. Finally, we provide empirical results showing how PPAM can guarantee constraint satisfaction in practice.

IJCAI Conference 2024 Conference Paper

Revising Beliefs and Intentions in Stochastic Environments

• Nima Motamed
• Natasha Alechina
• Mehdi Dastani
• Dragan Doder

The development of autonomous agents operating in dynamic and stochastic environments requires theories and models of how beliefs and intentions are revised while taking their interplay into account. In this paper, we initiate the study of belief and intention revision in stochastic environments, where an agent's beliefs and intentions are specified in a decidable probabilistic temporal logic. We then provide general Katsuno & Mendelzon-style representation theorems for both belief and intention revision, giving clear semantic characterizations of revision methods.

IJCAI Conference 2023 Conference Paper

Data-Driven Revision of Conditional Norms in Multi-Agent Systems (Extended Abstract)

• Davide Dell'Anna
• Natasha Alechina
• Fabiano Dalpiaz
• Mehdi Dastani
• Brian Logan

In multi-agent systems, norm enforcement is a mechanism for steering the behavior of individual agents in order to achieve desired system-level objectives. Due to the dynamics of multi-agent systems, however, it is hard to design norms that guarantee the achievement of the objectives in every operating context. Also, these objectives may change over time, thereby making previously defined norms ineffective. In this paper, we investigate the use of system execution data to automatically synthesise and revise conditional prohibitions with deadlines, a type of norms aimed at preventing agents from exhibiting certain patterns of behaviors. We propose DDNR (Data-Driven Norm Revision), a data-driven approach to norm revision that synthesises revised norms with respect to a data set of traces describing the behavior of the agents in the system. We evaluate DDNR using a state-of-the-art, off-the-shelf urban traffic simulator. The results show that DDNR synthesises revised norms that are significantly more accurate than the original norms in distinguishing adequate and inadequate behaviors for the achievement of the system-level objectives.

ECAI Conference 2023 Conference Paper

Dynamic Causality

• Maksim Gladyshev
• Natasha Alechina
• Mehdi Dastani
• Dragan Doder
• Brian Logan 0001

There have been a number of attempts to develop a formal definition of causality that accords with our intuitions about what constitutes a cause. Perhaps the best known is the “modified” definition of actual causality, HPm, due to Halpern. In this paper, we argue that HPm gives counterintuitive results for some simple causal models. We propose Dynamic Causality (DC), an alternative semantics for causal models that leads to an alternative definition of causes. DC ascribes the same causes as HPm on the examples of causal models widely discussed in the literature and ascribes intuitive causes for the kinds of causal models we consider. Moreover, we show that the complexity of determining a cause under the DC definition is lower than for the HPm definition.

KR Conference 2023 Conference Paper

Group Responsibility for Exceeding Risk Threshold

• Maksim Gladyshev
• Natasha Alechina
• Mehdi Dastani
• Dragan Doder

The need for tools and techniques to formally analyze and trace the responsibility for unsafe outcomes to decision-making actors is urgent. Existing formal approaches assume that the unsafe outcomes for which actors can be held responsible are actually realized. This paper considers a broader notion of responsibility where unsafe outcomes are not necessarily realized, but their probabilities are unacceptably high. We present a logic combining strategic, probabilistic and temporal primitives designed to express concepts such as the risk of an undesirable outcome and being responsible for exceeding a risk threshold. We demonstrate that the proposed logic is complete and decidable.

AAMAS Conference 2023 Conference Paper

Model-based Sparse Communication in Multi-agent Reinforcement Learning

• Shuai Han
• Mehdi Dastani
• Shihan Wang

Learning to communicate efficiently is central to multi-agent reinforcement learning (MARL). Existing methods often require agents to exchange messages intensively, which abuses communication channels and leads to high communication overhead. Only a few methods target on learning sparse communication, but they allow limited information to be shared, which affects the efficiency of policy learning. In this work, we propose model-based communication (MBC), a learning framework with a decentralized communication scheduling process. The MBC framework enables multiple agents to make decisions with sparse communication. In particular, the MBC framework introduces a model-based message estimator to estimate the up-to-date global messages using past local data. A decentralized message scheduling mechanism is also proposed to determine whether a message shall be sent based on the estimation. We evaluated our method in a variety of mixed cooperative-competitive environments. The experiment results show that the MBC method shows better performance and lower channel overhead than the state-of-art baselines.

PRL Workshop 2023 Workshop Paper

Preemptive Restraining Bolts

• Giovanni Varricchione
• Natasha Alechina
• Mehdi Dastani
• Giuseppe De Giacomo
• Brian Logan
• Giuseppe Perelli

We present preemptive restraining bolts (PRBs), a new approach to safe reinforcement learning which uses non-Markovian action masking, i.e., actions are masked (disallowed) based on the history of the system, rather than just the current state. PRBs are expressed in Pure Past Linear Temporal Logic and have minimal overhead (linear in the size of the state) compared to Markovian action masking, while having the same expressive power as other non-Markovian approaches such as shields (that can express any safety Linear Temporal Logic property). As with restraining bolts, the language in which safety properties are expressed does not have to be the same as the language specifying the features of the state for the learning agent. Critically, PRBs can be applied in the learning process to learn an optimal safe policy while using only safe actions during learning. As a result, PRBs can be used to provide general safety guarantees, without compromising efficiency.

IJCAI Conference 2023 Conference Paper

Probabilistic Temporal Logic for Reasoning about Bounded Policies

• Nima Motamed
• Natasha Alechina
• Mehdi Dastani
• Dragan Doder
• Brian Logan

To build a theory of intention revision for agents operating in stochastic environments, we need a logic in which we can explicitly reason about their decision-making policies and those policies' uncertain outcomes. Towards this end, we propose PLBP, a novel probabilistic temporal logic for Markov Decision Processes that allows us to reason about policies of bounded size. The logic is designed so that its expressive power is sufficient for the intended applications, whilst at the same time possessing strong computational properties. We prove that the satisfiability problem for our logic is decidable, and that its model checking problem is PSPACE-complete. This allows us to e. g. algorithmically verify whether an agent's intentions are coherent, or whether a specific policy satisfies safety and/or liveness properties.

LAMAS&SR Workshop 2023 Workshop Paper

Reasoning about Exceeding Risk Threshold

• Maksim Gladyshev
• Natasha Alechina
• Mehdi Dastani
• Dragan Doder

The problem of tracing the responsibility for unsafe outcomes to decision-making actors in multi-agent systems is urgent. While all existing approaches focus on deterministic outcomes, assuming that (a group of) agents can be held responsible for φ only if φ actually happens and agents could act differently to prevent φ, we find this notion of responsibility insufficient in many scenarios. In this work we combine coalition ability operator [G] from [12] with a probabilistic operator Lα from [5] that allow us to reason about probabilities and their changes. This approach allows us to claim that a group of agents can be held responsible for the unsafe outcome even if this outcome does not actually happen, but the group has caused its probability to be increased to an (unacceptably) high level. The proposed logic could be useful for analysing and assigning responsibility to groups of agents for their risky and unsafe behaviors. Finally, we establish (weak) completeness and decidability results for the proposed logic.

EUMAS Conference 2023 Conference Paper

Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning

• Giovanni Varricchione
• Natasha Alechina
• Mehdi Dastani
• Brian Logan 0001

Abstract Reward machines have recently been proposed as a means of encoding team tasks in cooperative multi-agent reinforcement learning. The resulting multi-agent reward machine is then decomposed into individual reward machines, one for each member of the team, allowing agents to learn in a decentralised manner while still achieving the team task. However, current work assumes the multi-agent reward machine to be given. In this paper, we show how reward machines for team tasks can be synthesised automatically from an Alternating-Time Temporal Logic specification of the desired team behaviour and a high-level abstraction of the agents’ environment. We present results suggesting that our automated approach has comparable, if not better, sample efficiency than reward machines generated by hand for multi-agent tasks.

AAMAS Conference 2023 Conference Paper

What Do You Care About: Inferring Values from Emotions

• Jieting Luo
• Mehdi Dastani
• Thomas Studer
• Beishui Liao

Observers can glean information from others’ emotional expressions through the act of drawing inferences from another individual’s emotional expressions. It is important for socially aware artificial systems to be capable of doing that as it can facilitate social interaction among agents, and is particularly important in humanrobot interaction for supporting a more personalized treatment of users. In this short paper, we propose a methodology for developing a formal model that allows agents to infer another agent’s values from her emotion expressions.

LAMAS&SR Workshop 2022 Workshop Paper

A Probabilistic Finite Temporal Logic for Policies and Intentions

• Nima Motamed
• Natasha Alechina
• Mehdi Dastani
• Dragan Doder

JAIR Journal 2022 Journal Article

Data-Driven Revision of Conditional Norms in Multi-Agent Systems

• Davide Dell'Anna
• Natasha Alechina
• Fabiano Dalpiaz
• Mehdi Dastani
• Brian Logan

In multi-agent systems, norm enforcement is a mechanism for steering the behavior of individual agents in order to achieve desired system-level objectives. Due to the dynamics of multi-agent systems, however, it is hard to design norms that guarantee the achievement of the objectives in every operating context. Also, these objectives may change over time, thereby making previously defined norms ineffective. In this paper, we investigate the use of system execution data to automatically synthesise and revise conditional prohibitions with deadlines, a type of norms aimed at prohibiting agents from exhibiting certain patterns of behaviors. We propose DDNR (Data-Driven Norm Revision), a data-driven approach to norm revision that synthesises revised norms with respect to a data set of traces describing the behavior of the agents in the system. We evaluate DDNR using a state-of-the-art, off-the-shelf urban traffic simulator. The results show that DDNR synthesises revised norms that are significantly more accurate than the original norms in distinguishing adequate and inadequate behaviors for the achievement of the system-level objectives.

AAMAS Conference 2022 Conference Paper

Modeling Affective Reaction in Multi-agent Systems

• Jieting Luo
• Mehdi Dastani

Affective reaction is a mechanism that is triggered by a perceived emotion and results in the observer’s emotion, subsequently steering the behavior of the observer. It is crucial to understand this mechanism as it constitutes a mediator between the emotional expressions of one agent and the (behavioral) responses of another thus facilitating social interaction among agents. This paper reports on a formal specification of affective reaction based on its psychological analysis using values and value systems as agents’ appraisal standards.

LAMAS&SR Workshop 2022 Workshop Paper

Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning

• Giovanni Varricchione
• Natasha Alechina
• Mehdi Dastani
• Brian Logan

AAMAS Conference 2021 Conference Paper

Responsibility Research for Trustworthy Autonomous Systems

• Vahid Yazdanpanah
• Enrico H. Gerding
• Sebastian Stein
• Mehdi Dastani
• Catholijn M. Jonker
• Timothy J. Norman

To develop and effectively deploy Trustworthy Autonomous Systems (TAS), we face various social, technological, legal, and ethical challenges in which different notions of responsibility can play a key role. In this work, we elaborate on these challenges, discuss research gaps, and show how the multidimensional notion of responsibility can play a role to bridge them. We argue that TAS requires operational tools to represent and reason about responsibilities of humans as well as AI agents. We review major challenges to which responsibility reasoning can contribute, highlight open research problems, and argue for the application of multiagent responsibility models in a variety of TAS domains.

ECAI Conference 2020 Conference Paper

An Ideal Team Is More than a Team of Ideal Agents

• Can Kurtan
• Pinar Yolum
• Mehdi Dastani

The problem of building a team to perform a complex task is often more than an optimal assignment of subtasks to agents based on individual performances. Subtasks may have subtle dependencies and relations that affect the overall performance of the formed team. This paper investigates the dependencies between subtasks and introduces some desired qualities of teams, such as preserving privacy or fairness. It proposes algorithms to analyze and build teams by taking into account the dependencies of assigned subtasks and agent performances. The performance of the algorithms are evaluated experimentally based on a multiagent system that is developed to answer complex queries. We show that by improving an initial team iteratively, the algorithm obtains teams with higher performance.

AIJ Journal 2020 Journal Article

Intention as commitment toward time

• Marc van Zee
• Dragan Doder
• Leendert van der Torre
• Mehdi Dastani
• Thomas Icard
• Eric Pacuit

In this paper we address the interplay among intention, time, and belief in dynamic environments. The first contribution is a logic for reasoning about intention, time and belief, in which assumptions of intentions are represented by preconditions of intended actions. Intentions and beliefs are coherent as long as these assumptions are not violated, i. e. as long as intended actions can be performed such that their preconditions hold as well. The second contribution is the formalization of what-if scenarios: what happens with intentions and beliefs if a new (possibly conflicting) intention is adopted, or a new fact is learned? An agent is committed to its intended actions as long as its belief-intention database is coherent. We conceptualize intention as commitment toward time and we develop AGM-based postulates for the iterated revision of belief-intention databases, and we prove a Katsuno-Mendelzon-style representation theorem.

EUMAS Conference 2020 Conference Paper

Multiagent Task Coordination as Task Allocation Plus Task Responsibility

• Vahid Yazdanpanah
• Mehdi Dastani
• Shaheen Fatima
• Nicholas R. Jennings
• Devrim Murat Yazan
• W. Henk M. Zijm

Abstract In this work, we present a dynamic Task Coordination framework ( ) for multiagent systems. Here task coordination refers to a twofold problem where an exogenously imposed state of affairs should be satisfied by a multiagent system. To address this problem the involved agents or agent groups need to be assigned tasks to fulfill (task allocation) and the behavior of these agents needs to be monitored to evaluate whether their tasks are fulfilled so that responsibility for dismissing tasks can be determined (task responsibility). We believe the allocation of tasks should regard both the strategic abilities of agents and their epistemic limitations. To date, however, existing work on the application of logical strategic reasoning for task allocation assumes perfect information for agents (dismissing imperfect information settings) and allocates tasks to individual agents (dismissing task allocation to agent groups). In, we address this gap by modeling task allocation using imperfect information semantics for strategic reasoning and integrate it with a notion of task responsibility. We formally verify properties of: on validity as well as stability of task allocations and fairness as well as non-monotonicity of task responsibilities.

JAAMAS Journal 2020 Journal Article

Runtime revision of sanctions in normative multi-agent systems

• Davide Dell’Anna
• Mehdi Dastani
• Fabiano Dalpiaz

Abstract To achieve system-level properties of a multiagent system, the behavior of individual agents should be controlled and coordinated. One way to control agents without limiting their autonomy is to enforce norms by means of sanctions. The dynamicity and unpredictability of the agents’ interactions in uncertain environments, however, make it hard for designers to specify norms that will guarantee the achievement of the system-level objectives in every operating context. In this paper, we propose a runtime mechanism for the automated revision of norms by altering their sanctions. We use a Bayesian Network to learn, from system execution data, the relationship between the obedience/violation of the norms and the achievement of the system-level objectives. By combining the knowledge acquired at runtime with an estimation of the preferences of rational agents, we devise heuristic strategies that automatically revise the sanctions of the enforced norms. We evaluate our heuristics using a traffic simulator and we show that our mechanism is able to quickly identify optimal revisions of the initially enforced norms.

AAMAS Conference 2019 Conference Paper

Decidable Model Checking with Uniform Strategies

• Natasha Alechina
• Mehdi Dastani
• Brian Logan

The logic of strategic ability Resource-Bounded Alternating Time Syntactic Epistemic Logic (RB±ATSEL) has a decidable modelchecking problem for coalition uniform strategies. A strategy is coalition uniform if agents in a coalition select the same joint action in all states where the knowledge of the coalition is the same. However, this presupposes free and unbounded communication between the agents in the coalition before every action selection. In this paper we present a modified version of RB±ATSEL, RB±ATSELc, with explicit (and explicitly costed) communication actions. RB±ATSELc is interpreted on communication models which have an explicit communication step before every action selection. We show that, unlike standard ATL under imperfect information, the model checking problem for RB±ATSELc is decidable under perfect recall uniform strategies. Our decidability result also applies to ATL with imperfect information and perfect recall when interpreted on communication models.

AAMAS Conference 2019 Conference Paper

Runtime Revision of Norms and Sanctions based on Agent Preferences

• Davide Dell'Anna
• Mehdi Dastani
• Fabiano Dalpiaz

To fulfill the overall objectives of a multiagent system, the behavior of individual agents should be controlled and coordinated. Runtime norm enforcement is one way to do so without over-constraining the agents’ autonomy. Due to the dynamicity and uncertainty of the environment, however, it is hard to specify norms that, when enforced, will fulfill the system-level objectives in every operating context. In this paper, we propose a mechanism for the automated revision of norms by altering their sanctions, based on the data monitored during the system execution and on some knowledge about the agents’ preferences. We use a Bayesian Network to learn at runtime the relationship between the obedience/violation of a norm and the achievement of the system objectives. We propose two heuristic strategies that explore the updated Bayesian Network and automatically revise the sanction of an enforced norm. An evaluation of our heuristics using a traffic simulator shows that our mechanisms outperform uninformed heuristics in terms of convergence speed.

AAMAS Conference 2019 Conference Paper

Strategic Responsibility Under Imperfect Information

• Vahid Yazdanpanah
• Mehdi Dastani
• Wojciech Jamroga
• Natasha Alechina
• Brian Logan

A central issue in the specification and verification of autonomous agents and multiagent systems is the ascription of responsibility to individual agents and groups of agents. When designing a (multi)agent system, we must specify which agents or groups of agents are responsible for bringing about a particular state of affairs. Similarly, when verifying a multiagent system, we may wish to determine the responsibility of agents or groups of agents for a particular state of affairs, and the contribution of each agent to bringing about that state of affairs. In this paper, we discuss several aspects of responsibility, including strategic ability of agents, their epistemic properties, and their relationship to the evolution of the system behavior. We introduce a formal framework for reasoning about the responsibility of individual agents and agent groups in terms of the agents’ strategies and epistemic properties, and state some properties of the framework.

FLAP Journal 2018 Journal Article

Modeling Norm Specification and Verification in Multiagent Systems.

• Natasha Alechina
• Brian Logan
• Mehdi Dastani

KER Journal 2018 Journal Article

Monitoring norms: a multi-disciplinary perspective

• Mehdi Dastani
• Paolo Torroni
• Neil Yorke-Smith

Abstract The concept of a norm is found widely across fields including artificial intelligence, biology, computer security, cultural studies, economics, law, organizational behaviour and psychology. The concept is studied with different terminology and perspectives, including individual, social, legal and philosophical. If a norm is an expected behaviour in a social setting, then this article considers how it can be determined whether an individual is adhering to this expected behaviour. We call this process monitoring, and again it is a concept known with different terminology in different fields. Monitoring of norms is foundational for processes of accountability, enforcement, regulation and sanctioning. Starting with a broad focus and narrowing to the multi-agent systems literature, this survey addresses four key questions: what is monitoring, what is monitored, who does the monitoring and how the monitoring is accomplished.

AAMAS Conference 2017 Conference Paper

Other-Condemning Anger = Blaming Accountable Agents for Unattainable Desires

• Mehdi Dastani
• Emiliano Lorini
• John-Jules Meyer
• Alexander Pankov

ECAI Conference 2016 Conference Paper

A Dynamic Logic of Norm Change

• Max Knobbout
• Mehdi Dastani
• John-Jules Ch. Meyer

Norms are effective and flexible means to control and regulate the behaviour of autonomous systems. Adding norms to a system changes its specification which may in turn ensure desirable system properties. As of yet, there is no generally agreed formal methodology to represent and reason about the dynamics of norms and their impacts on system specifications. In this paper, we introduce various types of norms, such as state-based or action-based norms, and gradually develop a dynamic modal logic to characterize the dynamics of such norms in a formal way. The logic can be used to prove various properties of norm dynamics and their impacts on system specification. Moreover, we show that this logic is sound and complete.

AAMAS Conference 2016 Conference Paper

Commitments and Interaction Norms in Organisations (JAAMAS Extended Abstract)

• Mehdi Dastani
• Leendert van der Torre
• Neil Yorke-Smith

ECAI Conference 2016 Conference Paper

Distributed Controllers for Norm Enforcement

• Bas Testerink
• Mehdi Dastani
• Nils Bulling

This paper focuses on computational mechanisms that control the behavior of autonomous systems at runtime without necessarily restricting their autonomy. We build on existing approaches from runtime verification, control automata, and norm-based systems, and define norm-based controllers that enforce norms by modifying system behavior at runtime to make it norm compliant. For many applications, an autonomous system should comply with a set of norms. We extend our approach to a distributed setting, where a set of norm-based controllers jointly modify the runtime behavior of an autonomous system. The norms that a set of norm-based controllers jointly enforce are investigated and characterized in terms of the norms that are enforced by individual norm-based controllers. We show that a set of norm-based controllers is able to modify the runtime behavior of an autonomous system to make it compliant with all norms that the individual norm-based controllers aim at enforcing.

AIJ Journal 2016 Journal Article

Norm-based mechanism design

• Nils Bulling
• Mehdi Dastani

The increasing presence of autonomous (software) systems in open environments in general, and the complex interactions taking place among them in particular, require flexible control and coordination mechanisms to guarantee desirable overall system level properties without limiting the autonomy of the involved systems. In artificial intelligence, and in particular in the multi-agent systems research field, social laws, norms, and sanctions have been widely proposed as flexible means for coordinating the behaviour of autonomous agents in multi-agent settings. Recently, many languages have been proposed to specify and implement norm-based environments where the behaviour of autonomous agents is monitored, evaluated based on norms, and possibly sanctioned if norms are violated. In this paper, we first introduce a formal setting of multi-agent environments based on concurrent game structures which abstracts from concrete specification languages. We extend this formal setting with norms and sanctions, and show how concepts from mechanism design can be used to formally analyse and verify whether a specific behaviour can be enforced (or implemented) if agents follow their subjective preferences. We relate concepts from mechanism design to our setting, where agents' preferences are modelled by linear time temporal logic (LTL) formulae. This proposal bridges the gap between norms and mechanism design allowing us to formally study and analyse the effect of norms and sanctions on the behaviour of rational agents. The proposed machinery can be used to check whether specific norms and sanctions have the designer's expected effect on the rational agents' behaviour or if a set of norms and sanctions that realise the effect exists at all. We investigate the computational complexity of our framework, focusing on its implementation in Nash equilibria and we show that it is located at the second and third level of the polynomial hierarchy. Despite this high complexity, on the positive side, these results are in line with existing complexity results of related problems. Finally, we propose a concrete executable specification language that can be used to implement multi-agent environments. We show that the proposed specification language generates specific concurrent game structures and that the abstract multi-agent environment setting can be applied to study and analyse the behaviour of multi-agent programs with and without norms.

IJCAI Conference 2016 Conference Paper

Verifying Existence of Resource-Bounded Coalition Uniform Strategies

• Natasha Alechina
• Mehdi Dastani
• Brian Logan

We consider the problem of whether a coalition of agents has a knowledge-based strategy to ensure some outcome under a resource bound. We extend previous work on verification of multi-agent systems where actions of agents produce and consume resources, by adding epistemic pre- and postconditions to actions. This allows us to model scenarios where agents perform both actions which change the world, and actions which change their knowledge about the world, such as observation and communication. To avoid logical omniscience and obtain a compact model of the system, our model of agents' knowledge is syntactic. We define a class of coalition-uniform strategies with respect to any (decidable) notion of coalition knowledge. We show that the model-checking problem for the resulting logic is decidable for any notion of coalition-uniform strategies in these classes.

IJCAI Conference 2015 Conference Paper

AGM Revision of Beliefs about Action and Time

• Marc van Zee
• Dragan Doder
• Mehdi Dastani
• Leendert van der Torre

The AGM theory of belief revision is based on propositional belief sets. In this paper we develop a logic for revision of temporal belief bases, containing expressions about temporal propositions (tomorrow it will rain), possibility (it may rain tomorrow), actions (the robot enters the room) and preand post-conditions of these actions. We prove the Katsuno-Mendelzon and the Darwiche-Pearl representation theorems by restricting the logic to formulas representing beliefs up to certain time. We illustrate our belief change model through several examples.

JAAMAS Journal 2015 Journal Article

Commitments and interaction norms in organisations

• Mehdi Dastani
• Leendert van der Torre
• Neil Yorke-Smith

Abstract In an organisational setting such as an online marketplace, an entity called the ‘organisation’ or ‘institution’ defines interaction protocols, monitors agent interaction, and intervenes to enforce the interaction protocols. The organisation might be a software system that thus regulates the marketplace, for example. In this article we abstract over application-specific protocols and consider commitment lifecycles as generic interaction protocols. We model interaction protocols by explicitly-represented norms, such that we can operationalise the enforcement of protocols by means of norm enforcement, and we can analyse the protocols by a logical analysis of the norms. We adopt insights and methods from commitment-based approaches to agent interaction as well as from norm-based approaches to agent behaviour governance. First, we show how to use explicitly-represented norms to model commitment dynamics (lifecycles). Second, we introduce an operational semantics to operationalise norm enforcement. Third, we show how to logically analyse interaction protocols by means of commitment dynamics and norm enforcement. The model, semantics, and analysis are illustrated by a running example from a vehicle insurance domain.

KER Journal 2015 Journal Article

Programming multi-agent systems

• Mehdi Dastani

Abstract With the significant advances in the area of autonomous agents and multi-agent systems in the last decade, promising technologies for the development and engineering of multi-agent systems have emerged. The result is a variety of agent-oriented programming languages, development frameworks, execution platforms, and tools that facilitate building and engineering of multi-agent systems. This paper provides an overview of the multi-agent programming research field and explains the aim and characteristics of various multi-agent programming languages and development frameworks. This overview is complemented with a discussion on the current trends and challenges in this research community.

FLAP Journal 2015 Journal Article

Retalis Language for Information Engineering in Autonomous Robot Software.

• Pouyan Ziafati
• Mehdi Dastani
• John-Jules Ch. Meyer
• Leon van der Torre
• Holger Voos

Robotic information engineering is the processing and management of data to create knowledge of the robot’s environment. It is an essential robotic tech- nique to apply AI methods such as situation awareness, task-level planning and knowledge-intensive task execution. Consequently, information engineering has been identified as a major challenge to make robotic systems more responsive to real-world situations. The Retalis language integrates ELE and SLR, two logic-based languages. Retalis is used to develop information engineering com- ponents of autonomous robots. In such a component, ELE is used for temporal and logical reasoning, and data transformation in flows of data. SLR is used to implement a knowledge base maintaining a history of events. SLR supports state-based representation of knowledge built upon discrete sensory data, man- agement of sensory data in active memories and synchronization of queries over asynchronous sensory data. In this paper, we introduce eight requirements for robotic information engineering, and we show how Retalis unifies and advances the state-of-the-art research on robotic information engineering. Moreover, we evaluate the efficiency of Retalis by implementing an application for a NAO robot. Retalis receives events about the positions of objects with respect to the top camera of NAO robot, the transformation among the coordinate frames of NAO robot, and the location of the NAO robot in the environment. About one thousand and nine hundreds events per second are processed in real-time to calculate the positions of objects in the environment.

JELIA Conference 2014 Conference Paper

Reasoning about Dynamic Normative Systems

• Max Knobbout
• Mehdi Dastani
• John-Jules Ch. Meyer

Abstract The use of normative systems is widely accepted as an effective approach to control and regulate the behaviour of agents in multi-agent systems. When norms are added to a normative system, the behaviour of such a system changes. As of yet, there is no clear formal methodology to model the dynamics of a normative system under addition of various types of norms. In this paper we view the addition of a norm as an update of a normative system, and we provide update semantics to model this process.

AAAI Conference 2013 Conference Paper

Multi-Cycle Query Caching in Agent Programming

• Natasha Alechina
• Tristan Behrens
• Mehdi Dastani
• Koen Hindriks
• Jomi Hubner
• Brian Logan
• Hai Nguyen
• Marc van Zee

In many logic-based BDI agent programming languages, plan selection involves inferencing over some underlying knowledge representation. While context-sensitive plan selection facilitates the development of flexible, declarative programs, the overhead of evaluating repeated queries to the agent’s beliefs and goals can result in poor run time performance. In this paper we present an approach to multi-cycle query caching for logic-based BDI agent programming languages. We extend the abstract performance model presented in (Alechina et al. 2012) to quantify the costs and benefits of caching query results over multiple deliberation cycles. We also present results of experiments with prototype implementations of both single- and multi-cycle caching in three logic-based BDI agent platforms, which demonstrate that significant performance improvements are achievable in practice.

EUMAS Conference 2013 Conference Paper

Norm Enforcement Mechanism

• Mehdi Dastani

IJCAI Conference 2013 Conference Paper

Reasoning about Normative Update

• Natasha Alechina
• Mehdi Dastani
• Brian Logan

We consider the problem of updating a multi-agent system with a set of conditional norms. A norm comes into effect when its condition becomes true, and imposes either an obligation or a prohibition on an agent which remains in force until a state satisfying a deadline condition is reached. If the norm is violated, a sanction is imposed on the agent. We define a notion of a normative update of a multi-agent system by a set of conditional norms, and study the problem of checking whether the agent(s) can bring about a state satisfying a property without incurring a specified number of sanctions.

AAMAS Conference 2012 Conference Paper

A logic of emotions: from appraisal to coping

• Mehdi Dastani
• Emiliano Lorini

Emotions is a cognitive mechanism that directs an agent's thoughts and attentions to what is relevant, important, and significant. Such a mechanism is crucial for the design of resource-bounded agents that must operate in highly-dynamic, semi-predictable environments and which need mechanisms for allocating their computational resources efficiently. The aim of this work is to propose a logical analysis of emotions and their influences on an agent's behaviour. We focus on four emotion types (viz, hope, fear, joy, and distress) and provide their logical characterizations in a model logic framework. As the intensity of emotion is essential for its influence on an agent's behaviour, the logic is devised to represent and reason about graded beliefs, graded goals and intentions. The belief strength and the goal strength determine the intensity of emotions. Emotions trigger different types of coping strategy which are aimed at dealing with emotions either by forming or revising an intention to act in the world, or by changing the agent's interpretation of the situation (by changing beliefs or goals).

AAMAS Conference 2012 Conference Paper

A Programming Approach to Monitoring Communication in an Organisational Environment

• Mehdi Dastani
• Leendert van der Torre
• Neil Yorke-Smith

Agreement technologies [1] achieve coordination among autonomous computational entities, by combining technologies for norms, semantics, organisations, argumentation, negotiation, and trust. We consider how an organisational programming language, such as 2OPL [2], can be extended to monitor communication. Such an extended programming language can be used to facilitate the development of electronic institutions, organisations, or marketplaces that aim at monitoring agent interaction (including both communication and non-communication actions), checking compliance with norms, and enforcing norms by means of sanctions. This abstract reports on specifying an operational semantics for agent interactions within such a setting, distinguishing constitutive norms for monitoring and sanction rules for enforcement of norms.

AAMAS Conference 2012 Conference Paper

Programming Norm-Aware Agents

• Natasha Alechina
• Mehdi Dastani
• Brian Logan

Normative organisations provide a means to coordinate the activities of individual agents in multiagent settings. The coordination is realized at run time by creating obligations and prohibitions (norms) for individual agents. If an agent cannot meet an obligation or violates a prohibition, the organisation imposes a sanction on the agent. In this paper, we consider \emph{norm-aware} agents that deliberate on their goals, norms and sanctions before deciding which plan to select and execute. A norm-aware agent is able to violate norms (accepting the resulting sanctions) if it is in the agent's overall interests to do so, e. g. , if meeting an obligation would result in an important goal of the agent becoming unachievable. Programming norm-aware agents in conventional BDI-based agent programming languages is difficult, as they lack support for deliberating about goals, norms and sanctions and deadlines. We present the norm-aware agent programming language N-2APL. N-2APL is based on 2APL and provides support for beliefs, goals, plans, norms, sanctions and deadlines. We give the syntax and semantics of N-2APL, and show that N-2APL agents are rational in the sense of committing to a set of plans that will achieve the agent's most important goals and obligations by their deadlines while respecting its most important prohibitions.

AAMAS Conference 2012 Conference Paper

Reasoning under Compliance Assumptions in Normative Multiagent Systems

• Max Knobbout
• Mehdi Dastani

The use of norms in multiagent systems has proven to be a successful approach in order to coordinate and regulate the behaviour of participating agents. In such normative systems it is generally assumed that agents can obey or disobey norms. In this paper, we develop a logical framework for normative systems that allows reasoning about agents' abilities under a multitude of norm compliance assumptions. In particular, we investigate different types of norm compliance and propose an extension of Alternating Temporal Logic (ATL) to reason about the abilities of (coalitions of) agents under different types of norm compliance assumptions. For this extension we show that the problem of model-checking remains close to the domain of standard ATL. Finally, we show that some norms can limit an agent's autonomy in the sense that an agent cannot control the violation of these norms. We present and discuss various classes of the so-called self-supporting norms, i. e. , norms for which individual agents have control over their violations

AAMAS Conference 2011 Conference Paper

Belief/Goal Sharing BDI Modules

• Michal Cap
• Mehdi Dastani
• Maaike Harbers

This paper proposes a modularisation framework for BDI based agent programming languages developed from a software engineering perspective. Like other proposals, BDI modules are seen as encapsulations of cognitive components. However, unlike other approaches, modules are here instantiated and manipulated in a similar fashion as objects in object orientation. In particular, an agent's mental state is formed dynamically by instantiating and activating BDI modules. The agent deliberates on its active module instances, which interact by sharing their beliefs and goals.

AAMAS Conference 2011 Conference Paper

Normative Programs and Normative Mechanism Design

• Nils Bulling
• Mehdi Dastani

The environment is an essential component of multi-agent systems, which is often used to coordinate the behaviour of individual agents. Recently many programming languages have been proposed to facilitate the implementation of such environments. This extended abstract is motivated by the emerging programming languages that are designed to implement environments in terms of normative concepts such as norms and sanctions. We propose a formal analysis of normative environment programs from a mechanism design perspective. By doing this we aim at relating normative environment programs to mechanism design, setting the stage for studying formal properties of these programs such as whether a set of norms implements a specific social choice function in a specific equilibria.

JAAMAS Journal 2011 Journal Article

Preface

• Rafael H. Bordini
• Mehdi Dastani
• Amal El Fallah Seghrouchni

AAMAS Conference 2011 Conference Paper

Programming Mental State Abduction

• Michal Sindlar
• Mehdi Dastani
• John-Jules Ch. Meyer

Many multi-agent system applications involve software agents that reason about the behavior of other agents with which they interact in cooperation or competition. In order to design and develop those systems, the employed programming languages should provide tools to facilitate the implementation of agents that can perform such reasoning. This paper focuses on BDI-based programming languages and proposes a nonmonotonic reasoning mechanism that can be incorporated into agents, allowing them to reason about observed behavior to infer others' beliefs or goals. In particular, it is suggested that the behavior-generating rules of agents are translated into a nonmonotonic logic programming framework. A formal analysis of the presented approach is provided and it is shown that it has desirable properties.

TCS Journal 2011 Journal Article

Reasoning about plan revision in BDI agent programs

• Natasha Alechina
• Mehdi Dastani
• Brian Logan
• John-Jules Ch. Meyer

Facilities for handling plan execution failures are essential for agents which must cope with the effects of nondeterministic actions, and some form of failure handling can be found in most mature agent programming languages and platforms. While such features simplify the development of more robust agents, they make it hard to reason about the execution of agent programs, e. g. , to verify their correctness. In this paper, we present an approach to the verification of agent programs which admit exceptional executions. We consider executions of the BDI-based agent programming language 3APL in which plans containing non-executable actions can be revised using plan revision rules, and present a logic for reasoning about normal and exceptional executions of 3APL programs. We provide a complete axiomatization for the logic and, using a simple example, show how to express properties of 3APL programs as formulas of the logic.

AAMAS Conference 2011 Conference Paper

Rich Goal Types in Agent Programming

• Mehdi Dastani
• M. Birna van Riemsdijk
• Michael Winikoff

Goals are central to the design and implementation of intelligent software agents. Much of the literature on goals and reasoning about goals in agent programming frameworks only deals with a limited set of goal types, typically achievement goals, and sometimes maintenance goals. In this paper we extend a previously proposed unifying framework for goals with additional richer goal types that are explicitly represented as Linear Temporal Logic (LTL) formulae. We show that these goal types can be modelled as a combination of achieve and maintain goals. This is done by providing an operationalization of these new goal types, and showing that the operationalization generates computation traces that satisfy the temporal formula.

IJCAI Conference 2011 Conference Paper

Verifying Normative Behaviour via Normative Mechanism Design

• Nils Bulling
• Mehdi Dastani

The environment is an essential component of multi-agent systems and is often used to coordinate the behaviour of individualagents. Recently many languages have been proposed to specify and implement multi-agent environments in terms of social and normative concepts. In this paper, we first introduce a formal setting of multi-agent environment which abstracts from concrete specification languages. We extend this formal setting with norms and sanctions and show how concepts from mechanism design can be used to formally analyse and verify whether specific normative behaviours can be enforced (or implemented) if agents follow their subjective preferences. We also consider complexity issues of associated problems.

ECAI Conference 2010 Conference Paper

A unified interaction-aware goal framework

• Michael Winikoff
• Mehdi Dastani
• M. Birna van Riemsdijk

Goals are central to the design and implementation of intelligent software agents. Much of the literature on goals and reasoning about goals only deals with a limited set of goal types, typically achievement goals, and sometimes maintenance goals; and much of the work on interactions between goals only deals with achievement goals. We aim at extending a previously proposed unifying framework for goals with additional richer goal types, including a combined "achieve and maintain" goal type. We propose to provide an operationalization of these new goal types, proving that the operationalization meets desired properties.

AAMAS Conference 2010 Conference Paper

Emotions to Control Agent Deliberation

• Bas Steunebrink
• Mehdi Dastani
• John-Jules Meyer

The execution of an artificial agent is usually implemented with a sense-reason-act cycle. This cycle includes tasks such as event processing, generating and revising plans, and selecting actions to execute. However, there are typically many choices in the design of such a cycle, which are often hard-coded in the cycle in an ad hoc way. The question of this paper is how one decides, in a principled way, how often and which reasoning rules to apply, how to interleave the execution of plans, or when to start replanning. This paper proposes and formalizes the eliciting conditions of hope, fear, joy, and distress according to a well-known psychological model of human emotion. These conditions are then used to reduce the choices an agent can make in each state. They formalize the idea that emotions focus an agent's attention on what is important in each state.

ECAI Conference 2010 Conference Paper

Mental State Ascription Using Dynamic Logic

• Michal P. Sindlar
• Mehdi Dastani
• John-Jules Ch. Meyer

In situations where the behavior of a system must be interpreted because its state is not accessible, it is useful to explain observed behavior in mentalistic terms. This paper presents a formalism based on propositional dynamic logic to model ascription of beliefs, goals, or plans on grounds of observed actions. The formalism is used to provide semantics for an existing approach to abducing the mental state of an observed agent; in doing so it is shown how behavior-producing rules can be given different explanatory interpretations.

AAMAS Conference 2010 Conference Paper

Programming Norm Change

• Nick Tinnemeier
• Mehdi Dastani
• John-Jules Meyer

To adequately deal with the unpredictable and dynamic environments normative frameworks are typically deployed in, mechanisms for modifying the norms at runtime are crucial. We present the syntax and operational semantics of generic programming constructs to facilitate runtime norm modification, allowing a programmer to specify when and how the norms may be changed by external agents or by the normative framework. The norms take on the form of conditional obligations and prohibitions, which instantiate detached obligations and prohibitions (instances). We present rule-based constructs for runtime modification of the norms and their instances, and a mechanism for automatically updating the instances when their underlying norms change.

AAMAS Conference 2010 Conference Paper

Reasoning about Strategies of Multi-Agent Programs

• Mehdi Dastani
• Wojciech Jamroga

Verification of multi-agent programs is a key problem in agentresearch and development. One may for example be interested inchecking if a specific subset of individual agent programs canachieve a particular state of their shared environment, or if theycan protect the system from entering a "bad" state. This paperfocuses on BDI-based multi-agent programs that consist of a finiteset of BDI-based agent programs executed concurrently. We choosealternating-time temporal logic (ATL) for expressing suchproperties. However, the original ATL is based on a synchronousmodel of multi-agent computation while most (if not all) multi-agentprogramming frameworks use asynchronous semantics where activitiesof different agents are interleaved. Moreover, unlike in ATL, ouragent programs do not have perfect information about the currentglobal state of the system. They are not appropriate subjects ofmodal epistemic logic either (since they do not know the globalmodel of the system). We begin by adapting the semantics of ATL tothe situation at hand; then, we consider the verification problem inthe new setting and present some preliminary results.

AAMAS Conference 2010 Conference Paper

Strategic Executions of Choreographed Timed Normative Multi-Agent Systems

• Lăcrămioara Aștefănoaei
• Frank S. De Boer
• Mehdi Dastani

This paper proposes a combined mechanism for coordinating agents in timed normative multi-agent systems. Timing constraints in a multi-agent system make it possible to force action execution to happen before certain time invariants are violated. In such multi-agent systems we achieve coordination at two orthogonal levels with respect to states and actions. On the one hand, the behaviour of individual agents is regulated by means of social and organisational inspired concepts like norms and sanctions. On the other hand, the behaviour of sets of agents is restricted according to action-based coordination mechanisms called choreographies. In both cases, the resulting behaviour is constrained by time.

JAAMAS Journal 2009 Journal Article

Guest editorial: Special issue on the European Workshop on Multi-Agent Systems (EUMAS)

• Rafael H. Bordini
• Mehdi Dastani

LORI Conference 2009 Conference Paper

Intentions and Assignments

• Emiliano Lorini
• Mehdi Dastani
• Hans van Ditmarsch
• Andreas Herzig
• John-Jules Ch. Meyer

Abstract The aim of this work is propose a logical approach to intention dynamics based on the notion of assignment [3, 7]. The function of an assignment is to associate the truth value of a certain formula ϕ to a propositional atom p. We combine a static modal logic of belief and choice with three kinds of dynamic modalities and corresponding three kinds of assignments: assignments operating on an agent’s beliefs, assignments operating on the agent’s choices and assignments operating on the objective world. An agent’s intention is defined in our approach as the agent’s choice to perform a given action and two basic operations on intentions called intention generation and intention reconsideration are defined as specific kinds of assignments on choices.

AAMAS Conference 2009 Conference Paper

Roles and Norms for Programming Agent Organizations

• Nick Tinnemeier
• Mehdi Dastani
• John-Jules Meyer

We present a programming language for implementing multiagent systems consisting of a set of individual agents that interact with a computational organization specified in terms of roles, norms and sanctions. We provide an operational semantics of the language that can serve as a basis for the implementation of organizational platforms. The view on roles is motivated by four key properties we deem important for an efficient implementation of agent organizations.

JAAMAS Journal 2008 Journal Article

2APL: a practical agent programming language

• Mehdi Dastani

Abstract This article presents a BDI-based agent-oriented programming language, called 2APL (A Practical Agent Programming Language). This programming language facilitates the implementation of multi-agent systems consisting of individual agents that may share and access external environments. It realizes an effective integration of declarative and imperative style programming by introducing and integrating declarative beliefs and goals with events and plans. It also provides practical programming constructs to allow the generation, repair, and (different modes of) execution of plans based on beliefs, goals, and events. The formal syntax and semantics of the programming language are given and its relation with existing BDI-based agent-oriented programming languages is discussed.

ECAI Conference 2008 Conference Paper

A Formal Model of Emotions: Integrating Qualitative and Quantitative Aspects

• Bas R. Steunebrink
• Mehdi Dastani
• John-Jules Ch. Meyer

When constructing a formal model of emotions for intelligent agents, two types of aspects have to be taken into account. First, qualitative aspects pertain to the conditions that elicit emotions. Second, quantitative aspects pertain to the actual experience and intensity of elicited emotions. In this paper, we show how the qualitative aspects of a well-known psychological model of human emotions can be formalized in an agent specification language and how its quantitative aspects can be integrated into this model. Furthermore, we discuss several unspecified details and implicit assumptions in the psychological model that are explicated by this effort.

AAMAS Conference 2008 Conference Paper

Goals in Agent Systems: A Unifying Framework

• M. Birna van Riemsdijk
• Mehdi Dastani
• Michael Winikoff

In the literature on agent systems, the proactive behavior of agents is often modeled in terms of goals that the agents pursue. We review a number of commonly-used existing goal types and propose a simple and general definition of goal, which unifies these goal types. We then give a formal and generic operationalization of goals by defining an abstract goal architecture, which describes the adoption, pursuit, and dropping of goals in a generic way. This operationalization is used to characterize the discussed goal types.

JAAMAS Journal 2008 Journal Article

Goals in conflict: semantic foundations of goals in agent programming

• M. Birna van Riemsdijk
• Mehdi Dastani
• John-Jules Ch. Meyer

Abstract This paper addresses the notion of (declarative) goals as used in agent programming. Goals describe desirable states, and semantics of these goals in an agent programming context can be defined in various ways. We focus in this paper on the representation of conflicting goals. In particular, we define two semantics for goals, one for unconditional goals and one for conditional goals. The first is based on propositional logic, and the latter is based on default logic. We establish relations between and properties of these semantics.

AAMAS Conference 2008 Conference Paper

Integrating Artifact-Based Environments with Heterogeneous Agent Programming Platforms

• Alessandro Ricci
• Michele Piunti
• Lemi Daghan Acay
• Rafael Bordini
• Jomi H
• uuml; bner
• Mehdi Dastani

“Agents and Artifacts” (A&A) and CARTAGO are becoming increasingly popular as, respectively, a general-purpose programming model and a related infrastructure for developing shared computational environments in agent-based software systems. However, so far there has been no work on developing multiagent systems (MAS) where agents implemented and deployed in different agent-programming platforms can interact as part of the same MAS with a shared environment. Due to the generality of CARTAGO environments and its Java-based implementation, we have successfully implemented an open multi-agent system where heterogeneous agents developed with different platforms—namely Jason, 2APL as BDI-based approaches and simpA as an activityoriented approach rather than BDI-based—work together in shared workspaces where they interact and cooperate by dynamically creating and using shared artifacts, analogously to human working environments. This paper shows how this was achieved by first presenting a general model for incorporating a theory of use and observation of artifacts in cognitive agents, then describing a general approach for developing such heterogeneous MAS using CARTAGO integrated with existing agent-oriented programming platforms.

AAMAS Conference 2008 Conference Paper

Reasoning about agent deliberation

• Natasha Alechina
• Mehdi Dastani
• Brian Logan
• John-Jules Meyer

We present a logic for reasoning about properties of agent programs under different agent execution strategies. Using the agent programming language SimpleAPL as an example, we show how safety and liveness properties can be expressed by translating agent programs into expressions of the logic. We give sound and complete axiomatizations of two different program execution strategies for SimpleAPL programs, and, for each of those strategies, prove a correspondence between the operational semantics of SimpleAPL and the models of the corresponding logic.

KR Conference 2008 Conference Paper

Reasoning about Agent Deliberation

• Natasha Alechina
• Mehdi Dastani
• Brian S. Logan
• John-Jules Ch. Meyer

We present a family of sound and complete logics for reasoning about deliberation strategies for SimpleAPL programs. SimpleAPL is a fragment of the agent programming language 3APL designed for the implementation of cognitive agents with beliefs, goals and plans. The logics are variants of PDL, and allow us to prove safety and liveness properties of SimpleAPL agent programs under different deliberation strategies. We show how to axiomatize different deliberation strategies for SimpleAPL programs, and, for each strategy we consider, prove a correspondence between the operational semantics of SimpleAPL and the models of the corresponding logic. We illustrate the utility of our approach with an example in which we show how to verify correctness properties for a simple agent program under different deliberation strategies.

AAMAS Conference 2007 Conference Paper

A Grounded Specification Language for Agent Programs

• Mehdi Dastani
• M. Birna van Riemsdijk
• John-Jules Ch. Meyer

This paper studies the relation between agent specification and agent programming languages. In particular, it shows that an agent programming language obeys some desirable properties expressed in an agent specification language, i. e. , that any agent implemented by the programming language satisfies the desirable property expressed in the specification language. We study this relation by defining and aligning the semantics of an agent specification language and implementation language, and prove that certain properties expressed in the specification language are satisfied by the implementation language.

AAMAS Conference 2007 Conference Paper

Practical Extensions in Agent Programming Languages

• Mehdi Dastani
• Dirk Hobo
• John-Jules Ch. Meyer

This paper proposes programming constructs to improve the practical application of existing BDI-based agent-oriented programming languages that have formal semantics. The proposed programming constructs include operations such as testing, adopting and dropping declarative goals, different execution modes for plans, repairing plans when their execution fail, event and exception handling mechanisms, and interfaces to existing imperative and declarative programming languages.

AAMAS Conference 2006 Conference Paper

Goal Types in Agent Programming

• Mehdi Dastani
• M. Birna van Riemsdijk
• John-Jules Meyer

ECAI Conference 2006 Conference Paper

Goal Types in Agent Programming

• Mehdi Dastani
• M. Birna van Riemsdijk
• John-Jules Ch. Meyer

This paper presents three types of declarative goals: perform goals, achieve goals, and maintain goals. The integration of these goal types in a simple but extendable logic-based agent-oriented programming language is discussed and motivated. The computational semantics for each goal type is presented by means of a transition system. It is shown that the presented semantics of the goal types ensure some desirable and expected properties.

AAMAS Conference 2006 Conference Paper

Goal-Oriented Modularity in Agent Programming

• M. Birna van Riemsdijk
• Mehdi Dastani
• John-Jules Meyer
• Frank de Boer

ECAI Conference 2006 Conference Paper

Programming Agents with Emotions

• Mehdi Dastani
• John-Jules Ch. Meyer

This paper presents the syntax and semantics of a simplified version of a logic-based agent-oriented programming language to implement agents with emotions. Four types of emotions are distinguished: happiness, sadness, anger and fear. These emotions are defined relative to agent's goals and plans. The emotions result from the agent's deliberation process and influence the deliberation process. The semantics of each emotion type is incorporated in the transition semantics of the presented agent-oriented programming language.

AAMAS Conference 2006 Conference Paper

Prototyping 3APL in the Maude Term Rewriting Language

• M. Birna van Riemsdijk
• Frank de Boer
• Mehdi Dastani
• John-Jules Meyer

AAMAS Conference 2006 Conference Paper

Structural Evaluation of Agent Organizations

• Davide Grossi
• Frank Dignum
• Virginia Dignum
• Mehdi Dastani
• Lamber Royakkers

AAMAS Conference 2005 Conference Paper

Coordination and Composition in Multi-Agent Systems

• Mehdi Dastani
• Farhad Arbab
• Frank de Boer

AAMAS Conference 2005 Conference Paper

Foundations of Organizational Structures in Multi-Agent Systems

• Davide Grossi
• Frank Dignum
• Lamber Royakkers
• Mehdi Dastani

LPAR Conference 2005 Conference Paper

Programming Cognitive Agents in Defeasible Logic

• Mehdi Dastani
• Guido Governatori
• Antonino Rotolo
• Leendert W. N. van der Torre

Abstract Defeasible Logic is extended to programming languages for cognitive agents with preferences and actions for planning. We define rule-based agent theories that contain preferences and actions, together with inference procedures. We discuss patterns of agent types in this setting. Finally, we illustrate the language by an example of an agent reasoning about web-services.

KER Journal 2005 Journal Article

Programming multi-agent systems

• Mehdi Dastani
• JORGE J. GOMEZ-SANZ

PROMAS (Programming Multi-Agent Systems) is an AgentLink technical forum that aims to bring together the researchers and practitioners from both academia and industry to discuss the problems related to the development of multi-agent systems and to evaluate the existing proposals and results. The specific focus of this technical forum is on agent-oriented programming languages and tools that can effectively and efficiently support the implementation of multi-agent systems. This paper provides an overview of the main contributions and conclusions derived from the first two editions of PROMAS technical forum.

JELIA Conference 2004 Conference Paper

Games for Cognitive Agents

• Mehdi Dastani
• Leendert W. N. van der Torre

Abstract Strategic games model the interaction among simultaneous decisions of agents. The starting point of strategic games is a set of players (agents) having strategies (decisions) and preferences on the game’s outcomes. In this paper we do not assume the decisions and preferences of agents to be given in advance, but we derive them from the agents’ mental attitudes. We specify such agents, define a mapping from their specification to the specification of the strategic game they play. We discuss a reverse mapping from the specification of strategic games that agents play to a specification of those agents. This mapping can be used to specify a group of agents that can play a strategic game, which shows that the notion of agent system specification is expressive enough to play any kind of game.

AAMAS Conference 2004 Conference Paper

Issues in Multiagent System Development

• Mehdi Dastani
• Joris Hulstijn
• Frank Dignum
• John-Jules Meyer

AAMAS Conference 2004 Conference Paper

Programming BOID Agents: a deliberation language for conflicts between mental attitudes and plans

• Mehdi Dastani
• Leendert van der Torre

IJCAI Conference 2003 Conference Paper

BDIO; Obligations and the Specification of Agent Behavior

• Jan Broersen
• Mehdi Dastani
• Leendert van der Torre

LPAR Conference 2002 Conference Paper

An Extension of BDI CTL with Functional Dependencies and Components

• Mehdi Dastani
• Leendert W. N. van der Torre

Abstract This paper discusses the formal specification of properties that determine the behavior of component based BDI agents, i. e. classical BDI agents in which the mental attitudes are conditional and represented by interconnected components. Some properties, such as realism and commitment strategies, have already been discussed in the BDI literature and can be formally specified by for example Rao and Georgeff’s BDI ctl formalism. Other properties are specific to component based cognitive agents and cannot be specified by existing BDI ctl formalisms. We focus here on the so-called functional dependencies between mental attitudes where a mental attitude is considered to be a function of one or more other mental attitudes. To formally specify the properties of functional dependencies we extend Rao and Georgeff’s BDI ctl formalism. In particular, for functional dependencies we introduce ‘only belief’, ‘only desire’and ‘only intend’ operators in the tradition of Levesque’s ‘all I know’ operator, and for components we distinguish between ‘belief in’and ‘belief out’, ‘desire in’and ‘desire out’, and ‘intention in’ and ‘intention out’ operators. We show how our extended formalism can be used to specify functionality properties such as conservativity, monotonic ity, and self-boundedness, as well as properties related to the connections between and control of the components.

NMR Workshop 2002 Conference Paper

What is a joint goal? Games with beliefs and defeasible desires

• Mehdi Dastani
• Leendert W. N. van der Torre

In this paper we introduce a qualitative decision and game theory based on belief (B) and desire (D) rules. We show that a group of agents acts as if it is maximizing achieved joint goals.