Ofra Amir

AIJ Journal 2024 Journal Article

ASQ-IT: Interactive explanations for reinforcement-learning agents

• Yotam Amitai
• Ofra Amir
• Guy Avni

AAAI Conference 2024 Conference Paper

Explaining Reinforcement Learning Agents through Counterfactual Action Outcomes

• Yotam Amitai
• Yael Septon
• Ofra Amir

Explainable reinforcement learning (XRL) methods aim to help elucidate agent policies and decision-making processes. The majority of XRL approaches focus on local explanations, seeking to shed light on the reasons an agent acts the way it does at a specific world state. While such explanations are both useful and necessary, they typically do not portray the outcomes of the agent's selected choice of action. In this work, we propose ``COViz'', a new local explanation method that visually compares the outcome of an agent's chosen action to a counterfactual one. In contrast to most local explanations that provide state-limited observations of the agent's motivation, our method depicts alternative trajectories the agent could have taken from the given state and their outcomes. We evaluated the usefulness of COViz in supporting people's understanding of agents' preferences and compare it with reward decomposition, a local explanation method that describes an agent's expected utility for different actions by decomposing it into meaningful reward types. Furthermore, we examine the complementary benefits of integrating both methods. Our results show that such integration significantly improved participants' performance.

AAMAS Conference 2023 Conference Paper

Explaining Agent Preferences and Behavior: Integrating Reward Decomposition and Contrastive Highlights

• Yael Septon
• Yotam Amitai
• Ofra Amir

Explainable reinforcement learning methods aim to help elucidate agent policies and their underlying decision-making processes. One such method is reward decomposition, which aims to reveal an agent’s preferences in a specific world-state by presenting its expected utility decomposed to different components of the reward function. While this approach quantifies the expected decomposed rewards for alternative actions, it does not demonstrate the outcomes of these alternative actions in terms of the behavior of the agent. This work introduces “Contrastive Highlights”, a novel local explanation method that visually compares the agent’s chosen behavior to an alternative choice of action in a contrastive manner. We conducted user studies comparing participants’ understanding of agents’ preferences based on either reward decomposition, contrastive highlights, or a combination of both approaches. Our results show that integrating reward decomposition with contrastive highlights significantly improved participants’ performance compared to using each of the approaches separately.

AAAI Conference 2023 Conference Paper

Frustratingly Easy Truth Discovery

• Reshef Meir
• Ofra Amir
• Omer Ben-Porat
• Tsviel Ben Shabat
• Gal Cohensius
• Lirong Xia

Truth discovery is a general name for a broad range of statistical methods aimed to extract the correct answers to questions, based on multiple answers coming from noisy sources. For example, workers in a crowdsourcing platform. In this paper, we consider an extremely simple heuristic for estimating workers' competence using average proximity to other workers. We prove that this estimates well the actual competence level and enables separating high and low quality workers in a wide spectrum of domains and statistical models. Under Gaussian noise, this simple estimate is the unique solution to the MLE with a constant regularization factor. Finally, weighing workers according to their average proximity in a crowdsourcing setting, results in substantial improvement over unweighted aggregation and other truth discovery algorithms in practice.

AAMAS Conference 2023 Conference Paper

Mitigating Skewed Bidding for Conference Paper Assignment

• Inbal Rozenzweig
• Reshef Meir
• Nicholas Mattei
• Ofra Amir

The explosion of conference paper submissions in AI and related fields has underscored the need to improve many aspects of the peer review process, especially the matching of papers and reviewers. Recent work argues that the key to improve this matching is to modify aspects of the bidding phase itself, to ensure that the set of bids over papers is balanced, and in particular to avoid orphan papers, i. e. , those papers that receive no bids. In an attempt to understand and mitigate this problem, we have developed a flexible bidding platform to test adaptations to the bidding process. Using this platform, we performed a field experiment during the bidding phase of a medium-size international workshop that compared two bidding methods. We further examined via controlled experiments on Amazon Mechanical Turk various factors that affect bidding, in particular the order in which papers are presented [11, 17]; and information on paper demand [33]. Our results suggest that several simple adaptations, that can be added to any existing platform, may significantly reduce the skew in bids, thereby improving the allocation for both reviewers and conference organizers.

IJCAI Conference 2022 Conference Paper

Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps (Extended Abstract)

• Tobias Huber
• Katharina Weitz
• Elisabeth André
• Ofra Amir

With advances in reinforcement learning (RL), agents are now being developed in high-stakes application domains such as healthcare and transportation. Explaining the behavior of these agents is challenging, as they act in large state spaces, and their decision-making can be affected by delayed rewards. In this paper, we explore a combination of explanations that attempt to convey the global behavior of the agent and local explanations which provide information regarding the agent's decision-making in a particular state. Specifically, we augment strategy summaries that demonstrate the agent's actions in a range of states with saliency maps highlighting the information it attends to. Our user study shows that intelligently choosing what states to include in the summary (global information) results in an improved analysis of the agents. We find mixed results with respect to augmenting summaries with saliency maps (local information).

AAAI Conference 2022 Conference Paper

“I Don’t Think So”: Summarizing Policy Disagreements for Agent Comparison

• Yotam Amitai
• Ofra Amir

With Artificial Intelligence on the rise, human interaction with autonomous agents becomes more frequent. Effective human-agent collaboration requires users to understand the agent’s behavior, as failing to do so may cause reduced productivity, misuse or frustration. Agent strategy summarization methods are used to describe the strategy of an agent to users through demonstrations. A summary’s objective is to maximize the user’s understanding of the agent’s aptitude by showcasing its behaviour in a selected set of world states. While shown to be useful, we show that current methods are limited when tasked with comparing between agents, as each summary is independently generated for a specific agent. In this paper, we propose a novel method for generating dependent and contrastive summaries that emphasize the differences between agent policies by identifying states in which the agents disagree on the best course of action. We conducted user studies to assess the usefulness of disagreementbased summaries for identifying superior agents and conveying agent differences. Results show disagreement-based summaries lead to improved user performance compared to summaries generated using HIGHLIGHTS, a strategy summarization algorithm which generates summaries for each agent independently.

AIJ Journal 2021 Journal Article

Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps

• Tobias Huber
• Katharina Weitz
• Elisabeth André
• Ofra Amir

IJCAI Conference 2019 Conference Paper

Exploring Computational User Models for Agent Policy Summarization

• Isaac Lage
• Daphna Lifschitz
• Finale Doshi-Velez
• Ofra Amir

AI agents support high stakes decision-making processes from driving cars to prescribing drugs, making it increasingly important for human users to understand their behavior. Policy summarization methods aim to convey strengths and weaknesses of such agents by demonstrating their behavior in a subset of informative states. Some policy summarization methods extract a summary that optimizes the ability to reconstruct the agent's policy under the assumption that users will deploy inverse reinforcement learning. In this paper, we explore the use of different models for extracting summaries. We introduce an imitation learning-based approach to policy summarization; we demonstrate through computational simulations that a mismatch between the model used to extract a summary and the model used to reconstruct the policy results in worse reconstruction quality; and we demonstrate through a human-subject study that people use different models to reconstruct policies in different contexts, and that matching the summary extraction model to these can improve performance. Together, our results suggest that it is important to carefully consider user models in policy summarization.

AIJ Journal 2019 Journal Article

Personalized change awareness: Reducing information overload in loosely-coupled teamwork

• Ofra Amir
• Barbara J. Grosz
• Krzysztof Z. Gajos
• Limor Gultchin

JAAMAS Journal 2019 Journal Article

Summarizing agent strategies

• Ofra Amir
• Finale Doshi-Velez
• David Sarne

Abstract Intelligent agents and AI-based systems are becoming increasingly prevalent. They support people in different ways, such as providing users with advice, working with them to achieve goals or acting on users’ behalf. One key capability missing in such systems is the ability to present their users with an effective summary of their strategy and expected behaviors under different conditions and scenarios. This capability, which we see as complementary to those currently under development in the context of “interpretable machine learning” and “explainable AI”, is critical in various settings. In particular, it is likely to play a key role when a user needs to collaborate with an agent, when having to choose between different available agents to act on her behalf, or when requested to determine the level of autonomy to be granted to an agent or approve its strategy. In this paper, we pose the challenge of developing capabilities for strategy summarization, which is not addressed by current theories and methods in the field. We propose a conceptual framework for strategy summarization, which we envision as a collaborative process that involves both agents and people. Last, we suggest possible testbeds that could be used to evaluate progress in research on strategy summarization.

AAMAS Conference 2019 Conference Paper

Toward Robust Policy Summarization

• Isaac Lage
• Daphna Lifschitz
• Finale Doshi-Velez
• Ofra Amir

AI agents are being developed to help people with high stakes decision-making processes from driving cars to prescribing drugs. It is therefore becoming increasingly important to develop “explainable AI” methods that help people understand the behavior of such agents. Summaries of agent policies can help human users anticipate agent behavior and facilitate more effective collaboration. Prior work has framed agent summarization as a machine teaching problem where examples of agent behavior are chosen to maximize reconstruction quality under the assumption that people do inverse reinforcement learning to infer an agent’s policy from demonstrations. We compare summaries generated under this assumption to summaries generated under the assumption that people use imitation learning. We show through simulations that in some domains, there exist summaries that produce high-quality reconstructions under different models, but in other domains, only matching the summary extraction model to the reconstruction model produces high-quality reconstructions. These results highlight the importance of assuming correct computational models for how humans extrapolate from a summary, suggesting human-in-the-loop approaches to summary extraction.

AAMAS Conference 2018 Conference Paper

Agent Strategy Summarization

• Ofra Amir
• Finale Doshi-Velez
• David Sarne

Intelligent agents and AI-based systems are becoming increasingly prevalent. They support people in different ways, such as providing users with advice, working with them to achieve goals or acting on users’ behalf. One key capability missing in such systems is the ability to present their users with an effective summary of their strategy and expected behaviors under different conditions and scenarios. This capability, which we see as complimentary to those currently under development in the context of “interpretable machine learning” and “explainable AI”, is critical in various settings. In particular, it is likely to play a key role whenever a user needs to understand the strategy of an agent she is working along with, when having to choose between different available agents to act on her behalf, or when requested to determine the level of autonomy to be granted to the agent or approve its strategy. In this paper, we pose the challenge of developing capabilities for strategy summarization, which is not addressed by current theories and methods in the field. We propose a conceptual framework for strategy summarization, which we envision as a collaborative process that involves both agents and people. Last, we suggest possible testbeds that could be used to evaluate progress in research on strategy summarization.

AAMAS Conference 2018 Conference Paper

HIGHLIGHTS: Summarizing Agent Behavior to People

• Dan Amir
• Ofra Amir

People increasingly interact with autonomous agents. This paper introduces and formalizes the problem of automatically generating a summary of an agent’s behavior with the goal of increasing people’s familiarity with the agent’s capabilities and limitations. In contrast with prior approaches which developed methods for explaining a single decision made by an agent, our approach aims to provide users with a summary that describes the agent’s behavior in different situations. We hypothesize that reviewing such summaries could help people in tasks such as choosing between agents or determining the level of autonomy to grant to an agent. We develop “HIGHLIGHTS”, an algorithm that produces a summary of an agent’s behavior by extracting important trajectories from simulations of the agent. We conducted a human-subject experiment to evaluate whether HIGHLIGHTS summaries help people assess the capabilities of agents. Our results show that participants were more successful at evaluating the capabilities of agents when presented with HIGHLIGHTS summaries compared to baseline summaries, and rated them as more helpful. We also explore a variant of the HIGHLIGHTS algorithm which aims to increase the diversity of states included in the summary, and show that this modification further improves people’s ability to assess agents’ capabilities.

IJCAI Conference 2016 Conference Paper

Interactive Teaching Strategies for Agent Training

• Ofra Amir
• Ece Kamar
• Andrey Kolobov
• Barbara J. Grosz

Agents learning how to act in new environments can benefit from input from more experienced agents or humans. This paper studies interactive teaching strategies for identifying when a student can benefit from teacher-advice in a reinforcement learning framework. In student-teacher learning, a teacher agent can advise the student on which action to take. Prior work has considered heuristics for the teacher to choose advising opportunities. While these approaches effectively accelerate agent training, they assume that the teacher constantly monitors the student. This assumption may not be satisfied with human teachers, as people incur cognitive costs of monitoring and might not always pay attention. We propose strategies for a teacher and a student to jointly identify advising opportunities so that the teacher is not required to constantly monitor the student. Experimental results show that these approaches reduce the amount of attention required from the teacher compared to teacher-initiated strategies, while maintaining similar learning gains. The empirical evaluation also investigates the effect of the information communicated to the teacher and the quality of the student's initial policy on teaching outcomes.

AAAI Conference 2016 Conference Paper

MIP-Nets: Enabling Information Sharing in Loosely-Coupled Teamwork

• Ofra Amir
• Barbara Grosz
• Krzysztof Gajos

People collaborate in carrying out such complex activities as treating patients, co-authoring documents and developing software. While technologies such as Dropbox and Github enable groups to work in a distributed manner, coordinating team members’ individual activities poses significant challenges. In this paper, we formalize the problem of “information sharing in loosely-coupled extended-duration teamwork”. We develop a new representation, Mutual Influence Potential Networks (MIP-Nets), to model collaboration patterns and dependencies among activities, and an algorithm, MIP-DOI, that uses this representation to reason about information sharing.

IJCAI Conference 2016 Conference Paper

Mutual Influence Potential Networks: Enabling Information Sharing in Loosely-Coupled Extended-Duration Teamwork

• Ofra Amir
• Barbara J. Grosz
• Krzysztof Z. Gajos

Complex collaborative activities such as treating patients, co-authoring documents and developing software are often characterized by teamwork that is loosely coupled and extends in time. To remain coordinated and avoid conflicts, team members need to identify dependencies between their activities - which though loosely coupled may interact - and share information appropriately. The loose-coupling of tasks increases the difficulty of identifying dependencies, with the result that team members often lack important information or are overwhelmed by irrelevant information. This paper formalizes a new multi-agent systems problem, Information Sharing in Loosely-Coupled Extended-Duration Teamwork (ISLET). It defines a new representation, Mutual Influence Potential Networks (MIP-Nets) and an algorithm, MIP-DOI, that uses this representation to determine the information that is most relevant to each team member. Importantly, because the extended duration of the teamwork precludes team-members developing complete plans in advance, the MIP-Nets approach, unlike prior work on information sharing, does not rely on a priori knowledge of a team's possible plans. Instead, it models collaboration patterns and dependencies among people and their activities based on team-members' interactions. Empirical evaluations show that this approach is able to learn collaboration patterns and identify relevant information to share with team members.

AAAI Conference 2015 Conference Paper

Multi-Agent Pathfinding as a Combinatorial Auction

• Ofra Amir
• Guni Sharon
• Roni Stern

This paper proposes a mapping between multi-agent pathfinding (MAPF) and combinatorial auctions (CAs). In MAPF, agents need to reach their goal destinations without colliding. Algorithms for solving MAPF aim at assigning agents non-conflicting paths that minimize agents’ travel costs. In CA problems, agents bid over bundles of items they desire. Auction mechanisms aim at finding an allocation of bundles that maximizes social welfare. In the proposed mapping of MAPF to CAs, agents bid on paths to their goals and the auction allocates non-colliding paths to the agents. Using this formulation, auction mechanisms can be naturally used to solve a range of MAPF problem variants. In particular, auction mechanisms can be applied to non-cooperative settings with self-interested agents while providing optimality guarantees and robustness to manipulations by agents. The paper further shows how to efficiently implement an auction mechanism for MAPF, utilizing methods and representations from both the MAPF and CA literatures.

AAAI Conference 2014 Conference Paper

Information Sharing for Care Coordination

• Ofra Amir

Teamwork and care coordination are of increasing importance to health care delivery and patient safety and health. My research aims at developing agents that are able to make intelligent information sharing decisions to support a diverse, evolving team of care providers in constructing and maintaining a shared plan that operates in uncertain environments.

AAAI Conference 2014 Conference Paper

To Share or Not to Share? The Single Agent in a Team Decision Problem

• Ofra Amir
• Barbara Grosz
• Roni Stern

This paper defines the “Single Agent in a Team Decision” (SATD) problem. SATD differs from prior multi-agent communication problems in the assumptions it makes about teammates’ knowledge of each other’s plans and possible observations. The paper proposes a novel integrated logical-decisiontheoretic approach to solving SATD problems, called MDP- PRT. Evaluation of MDP-PRT shows that it outperforms a previously proposed communication mechanism that did not consider the timing of communication and compares favorably with a coordinated Dec-POMDP solution that uses knowledge about all possible observations.

AAMAS Conference 2013 Conference Paper

Information Sharing for Care Coordination

• Ofra Amir

Teamwork and care coordination are of increasing importance to health care delivery and patient safety and health. This thesis aims at developing agents that are able to make intelligent information sharing decisions to support a diverse, evolving team of care providers in constructing and maintaining a shared plan that operates in uncertain environments and over a long time horizon.

IJCAI Conference 2011 Conference Paper

Plan Recognition in Virtual Laboratories

• Ofra Amir
• Ya'akov (Kobi) Gal

This paper presents a plan recognition algorithm for inferring student behavior using virtual science laboratories. The algorithm extends existing plan recognition technology and was integrated with an existing educational application for chemistry. Automatic recognition of students' activities in virtual laboratories can provide important information to teachers as well as serve as the basis for intelligent tutoring. Student use of virtual laboratories presents several challenges: Students may repeat activities indefinitely, interleave between activities, and engage in exploratory behavior using trial-anderror. The plan recognition algorithm uses a recursive grammar that heuristically generates plans on the fly, taking into account chemical reactions and effects to determine students' intended high-level actions. The algorithm was evaluated empirically on data obtained from college students using virtual laboratory software for teaching chemistry. Results show that the algorithm was able to (1) infer the plans used by students to construct their models; (2) recognize such key processes as titration and dilution when they occurred in students' work; (3) identify partial solutions; (4) isolate sequences of actions that were part of a single error.