Sherief Abdallah

JAAMAS Journal 2026 Journal Article

Efficient Management of Multi-Linked Negotiation Based on a Formalized Model

• Xiaoqin Zhang
• Victor Lesser
• Sherief Abdallah

abstract A Multi-linked negotiation problem occurs when an agent needs to negotiate with multiple other agents about different subjects (tasks, conflicts, or resource requirements), and the negotiation over one subject has influence on negotiations over other subjects. The solution of the multi-linked negotiations problem will become increasingly important for the next generation of advanced multi-agent systems. However, most current negotiation research looks only at a single negotiation and thus does not present techniques to manage and reason about multi-linked negotiations. In this paper, we first present a technique based on the use of a partial-order schedule and a measure of the schedule, called flexibility, which enables an agent to reason explicitly about the interactions among multiple negotiations. Next, we introduce a formalized model of the multi-linked negotiation problem. Based on this model, a heuristic search algorithm is developed for finding a near-optimal ordering of negotiation issues and their parameters. Using this algorithm, an agent can evaluate and compare different negotiation approaches and choose the best one. We show how an agent uses this technology to effectively manage interacting negotiation issues. Experimental work is presented which shows the efficiency of this approach.

JMLR Journal 2016 Journal Article

Addressing Environment Non-Stationarity by Repeating Q-learning Updates

• Sherief Abdallah
• Michael Kaisers

Q-learning (QL) is a popular reinforcement learning algorithm that is guaranteed to converge to optimal policies in Markov decision processes. However, QL exhibits an artifact: in expectation, the effective rate of updating the value of an action depends on the probability of choosing that action. In other words, there is a tight coupling between the learning dynamics and underlying execution policy. This coupling can cause performance degradation in noisy non-stationary environments. Here, we introduce Repeated Update Q-learning (RUQL), a learning algorithm that resolves the undesirable artifact of Q-learning while maintaining simplicity. We analyze the similarities and differences between RUQL, QL, and the closest state-of-the-art algorithms theoretically. Our analysis shows that RUQL maintains the convergence guarantee of QL in stationary environments, while relaxing the coupling between the execution policy and the learning dynamics. Experimental results confirm the theoretical insights and show how RUQL outperforms both QL and the closest state-of-the-art algorithms in noisy non-stationary environments. [abs] [ pdf ][ bib ] &copy JMLR 2016. ( edit, beta )

AAMAS Conference 2012 Conference Paper

Using a hierarchy of coordinators to overcome the frontier effect in social learning

• Sherief Abdallah

KER Journal 2011 Journal Article

Representing and classifying arguments on the Semantic Web

• Iyad Rahwan
• Bita Banihashemi
• Chris Reed
• Douglas Walton
• Sherief Abdallah

Abstract Until recently, little work has been dedicated to the representation and interchange of informal, semi-structured arguments of the type found in natural language prose and dialogue. To redress this, the research community recently initiated work towards an Argument Interchange Format (AIF). The AIF aims to facilitate the exchange of semi-structured arguments among different argument analysis and argumentation-support tools. In this paper, we present a Description Logic ontology for annotating arguments, based on a new reification of the AIF and founded in Walton's theory of argumentation schemes. We demonstrate how this ontology enables a new kind of automated reasoning over argument structures, which complements classical reasoning about argument acceptability. In particular, Web Ontology Language reasoning enables significantly enhanced querying of arguments through automatic scheme classifications, instance classification, inference of indirect support in chained argument structures, and inference of critical questions. We present the implementation of a pilot Web-based system for authoring and querying argument structures using the proposed ontology.

AAMAS Conference 2010 Conference Paper

Self-Organization for Coordinating Decentralized Reinforcement Learning

• Chongjie Zhang
• Victor Lesser
• Sherief Abdallah

Decentralized reinforcement learning (DRL) has been applied to a number of distributed applications. However, oneof the main challenges faced by DRL is its convergence. Previous work has shown that hierarchically organizational control is an effective way of coordinating DRL to improve itsspeed, quality, and likelihood of convergence. In this paper, we develop a distributed, negotiation-based approach todynamically forming such hierarchical organizations. To reduce the complexity of coordinating DRL, our self-organizationapproach groups strongly-interacting learning agents together, whose exploration strategies are coordinated by one supervisor. We formalize this idea by characterizing interactionsamong agents in a decentralized Markov Decision Processmodel and defining and analyzing a measure that explicitlycaptures the strength of such interactions. Experimental results show that our dynamically evolving organizations outperform predefined organizations for coordinating DRL.

AAMAS Conference 2010 Conference Paper

Using Graph Analysis to Study Networks of Adaptive Agent

• Sherief Abdallah

Experimental analysis of networks of cooperative learningagents (to verify certain properties such as the system's stability) has been commonly used due to the complexity oftheoretical analysis in such cases. Due to the large numberof parameters to analyze, researchers used metrics that summarize the system in few parameters. Since in cooperativesystem the ultimate goal is to optimize some global metric, researchers typically analyzed the evolution of the globalperformance metric over time to verify system properties. For example, if the global metric improves and eventuallystabilizes, it is considered a reasonable verification of thesystem's stability. The global performance metric, however, overlooks an important aspect of the system: the network structure. Weshow an experimental case study where the convergence ofthe global performance metric is deceiving, hiding an underlying instability in the system that later leads to a significantdrop in performance. To expose such instability, we propose the use of the graph analysis methodology, where thenetwork structure is summarized using some network measures. We develop a new network measure that summarizesan agent's interaction with its neighbors and takes the disparity of these interactions into account. The new measureis applied to our case study, clearly exposing the instability that was previously hidden by the global performancemetric.

AAMAS Conference 2009 Conference Paper

Integrating Organizational Control into Multi-Agent Learning

• Chongjie Zhang
• Sherief Abdallah
• Victor Lesser

Multi-Agent Reinforcement Learning (MARL) algorithms suffer from slow convergence and even divergence, especially in largescale systems. In this work, we develop an organization-based control framework to speed up the convergence of MARL algorithms in a network of agents. Our framework defines a multi-level organizational structure for automated supervision and a communication protocol for exchanging information between lower-level agents and higher-level supervising agents. The abstracted states of lower-level agents travel upwards so that higher-level supervising agents generate a broader view of the state of the network. This broader view is used in creating supervisory information which is passed down the hierarchy. The supervisory policy adaptation then integrates supervisory information into existing MARL algorithms, guiding agents’ exploration of their state-action space. The generality of our framework is verified by its applications on different domains (distributed task allocation and network routing) with different MARL algorithms. Experimental results show that our framework improves both the speed and likelihood of MARL convergence.

AAMAS Conference 2008 Conference Paper

Efficient Multi-Agent Reinforcement Learning through Automated Supervision

• Chongjie Zhang
• Sherief Abdallah
• Victor Lesser

Multi-Agent Reinforcement Learning (MARL) algorithms suffer from slow convergence and even divergence, especially in large-scale systems. In this work, we develop a supervision framework to speed up the convergence of MARL algorithms in a network of agents. The framework defines an organizational structure for automated supervision and a communication protocol for exchanging information between lower-level agents and higher-level supervising agents. The abstracted states of lower-level agents travel upwards so that higher-level supervising agents generate a broader view of the state of the network. This broader view is used in creating supervisory information which is passed down the hierarchy. We present a generic extension to MARL algorithms that integrates supervisory information into the learning process, guiding agents’ exploration of their stateaction space.

AAMAS Conference 2008 Conference Paper

Non-linear Dynamics in Multiagent Reinforcement Learning Algorithms

• Sherief Abdallah
• Victor Lesser

Several multiagent reinforcement learning (MARL) algorithms have been proposed to optimize agents’ decisions. Only a subset of these MARL algorithms both do not require agents to know the underlying environment and can learn a stochastic policy (a policy that chooses actions according to a probability distribution). Weighted Policy Learner (WPL) is a MARL algorithm that belongs to this subset and was shown, experimentally in previous work, to converge and outperform previous MARL algorithms belonging to the same subset. The main contribution of this paper is analyzing the dynamics of WPL and showing the effect of its non-linear nature, as opposed to previous MARL algorithms that had linear dynamics. First, we represent the WPL algorithm as a set of differential equations. We then solve the equations and show that it is consistent with experimental results reported in previous work. We finally compare the dynamics of WPL with earlier MARL algorithms and discuss the interesting differences and similarities we have discovered.

AAMAS Conference 2007 Conference Paper

Multiagent Reinforcement Learning and Self-Organization in a Network of Agents

• Sherief Abdallah
• Victor Lesser

To cope with large scale, agents are usually organized in a network such that an agent interacts only with its immediate neighbors in the network. Reinforcement learning techniques have been commonly used to optimize agents local policies in such a network because they require little domain knowledge and can be fully distributed. However, all of the previous work assumed the underlying network was fixed throughout the learning process. This assumption was important because the underlying network defines the learning context of each agent. In particular, the set of actions and the state space for each agent is defined in terms of the agent's neighbors. If agents dynamically change the underlying network structure (also called self-organize) during learning, then one needs a mechanism for transferring what agents have learned so far before (in the old network structure) to their new learning context (in the new network structure).

AAMAS Conference 2006 Conference Paper

Learning the Task Allocation Game

• Sherief Abdallah
• Victor Lesser