Matthias Scheutz

AAAI Conference 2026 System Paper

IntelliProof: An Argumentation Network-based Conversational Helper for Organized Reflection

• Kaveh Eskandari Miandoab
• Katharine Kowalyshyn
• Kabir Pamnani
• Anesu Gavhera
• Vasanth Sarathy
• Matthias Scheutz

We present IntelliProof, an interactive system for analyzing argumentative essays through LLMs. IntelliProof structures an essay as an argumentation graph, where claims are represented as nodes, supporting evidence is attached as node properties, and edges encode supporting or attacking relations. Unlike existing automated essay scoring systems, IntelliProof emphasizes the user experience: each relation is initially classified and scored by an LLM, then visualized for enhanced understanding. The system provides justifications for classifications and produces quantitative measures for essay coherence. It enables rapid exploration of argumentative quality while retaining human oversight. In addition, IntelliProof provides a set of tools for a better understanding of an argumentative essay and its corresponding graph in natural language, bridging the gap between the structural semantics of argumentative essays and the user's understanding of a given text.

TIST Journal 2026 Journal Article

On Evaluating LLM Integration into Robotic Architectures

• Vasanth Sarathy
• Marlow Fawn
• Matthew McWilliams
• Matthias Scheutz
• Bradley Oosterveld

LLMs are being increasingly integrated into embodied robotic systems. A useful capability that the LLMs bring to robots is translating noisy spoken human natural language instructions into executable robot actions. However, these integrations are somewhat ad hoc and understudied as they tend to not consider the gamut of syntactic, semantic, as well as pragmatic aspects of embodied human communication. What is missing is a characterization of the different paradigms for integrating LLMs into robotic architectures as well as a set of evaluation metrics that capture whether an LLM-equipped robot can correctly understand these different aspects of human instruction. In this article, we present a suite of evaluation metrics together with data augmentation techniques for evaluating these architectures, using concepts from the cognitive science and human communication literature. To illustrate an application of these metrics and augmentation techniques, we conduct experiments to compare two integration methods: LLMs as pre-processing components that map human instructions into more constrained versions to be processed by the architecture’s natural language understanding (NLU) subsystem, or LLMs as a wholesale replacement for the NLU’s parser. We provide experimental evaluations and a robotic implementation to show the inherent tradeoffs between the methods. Our results suggest that while they offer increased explainability, traditional parsing tools coupled with LLMs do not perform as well as an LLM that replaces a parser entirely. The proposed evaluation metrics together with the characterization of different LLM integration approaches offer the promise of systematically evaluating LLMs as natural language interfaces to robotic systems as well as tackle the important tradeoff between explainability/verifiability/interpretability and robustness to noisy input and broad language understanding in an open-world embodied setting.

AAAI Conference 2026 Conference Paper

Where Norms and References Collide: Evaluating LLMs on Normative Reasoning

• Mitchell Abrams
• Kaveh Eskandari Miandoab
• Felix Gervits
• Vasanth Sarathy
• Matthias Scheutz

Embodied agents, such as robots, will need to interact in situated environments where successful communication often depends on reasoning over social norms: shared expectations that constrain what actions are appropriate in context. A key capability in such settings is norm-based reference resolution (NBRR), where interpreting referential expressions requires inferring implicit normative expectations grounded in physical and social context. Yet it remains unclear whether Large Language Models (LLMs) can support this kind of reasoning. In this work, we introduce SNIC (Situated Norms in Context), a human-validated diagnostic testbed designed to probe how well state-of-the-art LLMs can extract and utilize normative principles relevant to NBRR. SNIC emphasizes physically grounded norms that arise in everyday tasks such as cleaning, tidying, and serving. Across a range of controlled evaluations, we find that even the strongest LLMs struggle to consistently identify and apply social norms—particularly when norms are implicit, underspecified, or in conflict. These findings reveal a blind spot in current LLMs and highlight a key challenge for deploying language-based systems in socially situated, embodied settings.

ICRA Conference 2025 Conference Paper

Curiosity-Driven Imagination: Discovering Plan Operators and Learning Associated Policies for Open-World Adaptation

• Pierrick Lorang
• Hong Lu
• Matthias Scheutz

Adapting quickly to dynamic, uncertain environments—often called “open worlds” —remains a major challenge in robotics. Traditional Task and Motion Planning (TAMP) approaches struggle to cope with unforeseen changes, are data-inefficient when adapting, and do not leverage world models during learning. We address this issue with a hybrid planning and learning system that integrates two models: a low-level neural network-based model that learns stochastic transitions and drives exploration via an Intrinsic Curiosity Module (ICM), and a high-level symbolic planning model that captures abstract transitions using operators, enabling the agent to plan in an “imaginary” space and generate reward machines. Our evaluation in a robotic manipulation domain with sequential novelty injections demonstrates that our approach converges faster and outperforms state-of-the-art hybrid methods.

ICRA Conference 2025 Conference Paper

FLEX: A Framework for Learning Robot-Agnostic Force-Based Skills Involving Sustained Contact Object Manipulation

• Shijie Fang
• Wenchang Gao
• Shivam Goel
• Christopher Thierauf
• Matthias Scheutz
• Jivko Sinapov

Learning to manipulate objects efficiently, particularly those involving sustained contact (e. g. , pushing, sliding) and articulated parts (e. g. , drawers, doors), presents significant challenges. Traditional methods, such as robot-centric reinforce-ment learning (RL), imitation learning, and hybrid techniques, require massive training and often struggle to generalize across different objects and robot platforms. We propose a novel framework for learning object-centric manipulation policies in force space, decoupling the robot from the object. By directly applying forces to selected regions of the object, our method simplifies the action space, reduces unnecessary exploration, and decreases simulation overhead. This approach, trained in simulation on a small set of representative objects, captures ob-ject dynamics—such as joint configurations—allowing policies to generalize effectively to new, unseen objects. Decoupling these policies from robot-specific dynamics enables direct transfer to different robotic platforms (e. g. , Kinova, Panda, URS) with-out retraining. Our evaluations demonstrate that the method significantly outperforms baselines, achieving over an order of magnitude improvement in training efficiency compared to other state-of-the-art methods. Additionally, operating in force space enhances policy transferability across diverse robot plat-forms and object types. We further showcase the applicability of our method in a real-world robotic setting. Link: https://tufts-ai-robotics-group.github.io/FLEX/

IROS Conference 2025 Conference Paper

Incremental Language Understanding for Online Motion Planning of Robot Manipulators

• Mitchell Abrams
• Thies Oelerich
• Christian Hartl-Nesic
• Andreas Kugi
• Matthias Scheutz

Human-robot interaction requires robots to process language incrementally, adapting their actions in real-time based on evolving speech input. Existing approaches to language-guided robot motion planning typically assume fully specified instructions, resulting in inefficient stop-and-replan behavior when corrections or clarifications occur. In this paper, we introduce a novel reasoning-based incremental parser which integrates an online motion planning algorithm within the cognitive architecture. Our approach enables continuous adaptation to dynamic linguistic input, allowing robots to update motion plans without restarting execution. The incremental parser maintains multiple candidate parses, leveraging reasoning mechanisms to resolve ambiguities and revise interpretations when needed. By combining symbolic reasoning with online motion planning, our system achieves greater flexibility in handling speech corrections and dynamically changing constraints. We evaluate our framework in real-world human-robot interaction scenarios, demonstrating online adaptions of goal poses, constraints, or task objectives. Our results highlight the advantages of integrating incremental language understanding with real-time motion planning for natural and fluid human-robot collaboration. The experiments are demonstrated in the accompanying video at www. acin. tuwien. ac. at/42d5.

IROS Conference 2024 Conference Paper

A Framework for Neurosymbolic Goal-Conditioned Continual Learning in Open World Environments

• Pierrick Lorang
• Shivam Goel
• Yash Shukla
• Patrik Zips
• Matthias Scheutz

In dynamic open-world environments, agents continually face new challenges due to sudden and unpredictable novelties, hindering Task and Motion Planning (TAMP) in autonomous systems. We introduce a novel TAMP architecture that integrates symbolic planning with reinforcement learning to enable autonomous adaptation in such environments, operating without human guidance. Our approach employs symbolic goal representation within a goal-oriented learning framework, coupled with planner-guided goal identification, effectively managing abrupt changes where traditional reinforcement learning, re-planning, and hybrid methods fall short. Through sequential novelty injections in our experiments, we assess our method’s adaptability to continual learning scenarios. Extensive simulations conducted in a robotics domain corroborate the superiority of our approach, demonstrating faster convergence to higher performance compared to traditional methods. The success of our framework in navigating diverse novelty scenarios within a continuous domain underscores its potential for critical real-world applications.

AIJ Journal 2024 Journal Article

A neurosymbolic cognitive architecture framework for handling novelties in open worlds

• Shivam Goel
• Panagiotis Lymperopoulos
• Ravenna Thielstrom
• Evan Krause
• Patrick Feeney
• Pierrick Lorang
• Sarah Schneider
• Yichen Wei

“Open world” environments are those in which novel objects, agents, events, and more can appear and contradict previous understandings of the environment. This runs counter to the “closed world” assumption used in most AI research, where the environment is assumed to be fully understood and unchanging. The types of environments AI agents can be deployed in are limited by the inability to handle the novelties that occur in open world environments. This paper presents a novel cognitive architecture framework to handle open-world novelties. This framework combines symbolic planning, counterfactual reasoning, reinforcement learning, and deep computer vision to detect and accommodate novelties. We introduce general algorithms for exploring open worlds using inference and machine learning methodologies to facilitate novelty accommodation. The ability to detect and accommodate novelties allows agents built on this framework to successfully complete tasks despite a variety of novel changes to the world. Both the framework components and the entire system are evaluated in Minecraft-like simulated environments. Our results indicate that agents are able to efficiently complete tasks while accommodating “concealed novelties” not shared with the architecture development team.

KR Conference 2024 Conference Paper

Action Language mA* with Higher-Order Action Observability

• David Buckingham
• Matthias Scheutz
• Tran Cao Son
• Francesco Fabiano

This paper presents a novel semantics for the mA* epistemic action language that takes into consideration dynamic per-agent observability of events. Different from the original mA* semantics, the observability of events is defined locally at the level of possible worlds, giving a new method for compiling event models. Locally defined observability represents agents' uncertainty and false-beliefs about each others' ability to observe events. This allows for modeling second-order false-belief tasks where one agent does not know the truth about another agent's observations and resultant beliefs. The paper presents detailed constructions of event models for ontic, sensing, and truthful announcement action occurrences and proves various properties relating to agents' beliefs after the execution of an action. It also shows that the proposed approach can model second order false-belief tasks and satisfies the robustness and faithfulness criteria discussed by Bolander (2018, https: //doi. org/10. 1007/978-3-319-62864-6_8).

ICRA Conference 2024 Conference Paper

Adapting to the "Open World": The Utility of Hybrid Hierarchical Reinforcement Learning and Symbolic Planning

• Pierrick Lorang
• Helmut Horvath
• Tobias Kietreiber
• Patrik Zips
• Clemens Heitzinger
• Matthias Scheutz

Open-world robotic tasks such as autonomous driving pose significant challenges to robot control due to unknown and unpredictable events that disrupt task performance. Neural network-based reinforcement learning (RL) techniques (like DQN, PPO, SAC, etc.) struggle to adapt in large domains and suffer from catastrophic forgetting. Hybrid planning and RL approaches have shown some promise in handling environmental changes but lack efficiency in accommodation speed. To address this limitation, we propose an enhanced hybrid system with a nested hierarchical action abstraction that can utilize previously acquired skills to effectively tackle unexpected novelties. We show that it can adapt faster and generalize better compared to state-of-the-art RL and hybrid approaches, significantly improving robustness when multiple environmental changes occur at the same time.

IROS Conference 2024 Conference Paper

Fixing symbolic plans with reinforcement learning in object-based action spaces

• Christopher Thierauf
• Matthias Scheutz

Reinforcement learning techniques are widely used when robots have to learn new tasks but they typically operate on action spaces defined by the joints of the robot. We present a contrasting approach where actions spaces are the trajectories of objects in the environment, requiring robots to discover events such as object changes and behaviors that must occur to accomplish the task. We show that this allows robots to learn faster, to learn semantic representations that can be communicated to humans, and to learn in a manner that does not depend on the robot itself, enabling low-cost policy transfer between different types of robots. Our demonstrations can be replicated using provided source code 1.

AAMAS Conference 2024 Conference Paper

NovelGym: A Flexible Ecosystem for Hybrid Planning and Learning Agents Designed for Open Worlds

• Shivam Goel
• Yichen Wei
• Panagiotis Lymperopoulos
• Klára Churá
• Matthias Scheutz
• Jivko Sinapov

As AI agents leave the lab and venture into the real world as autonomous vehicles, delivery robots, and cooking robots, it is increasingly necessary to design and comprehensively evaluate algorithms that tackle the “open-world”. To this end, we introduce NovelGym1, a flexible and adaptable ecosystem designed to simulate gridworld environments, serving as a robust platform for benchmarking reinforcement learning (RL) and hybrid planning and learning agents in open-world contexts. The modular architecture of NovelGym facilitates rapid creation and modification of task environments, including multi-agent scenarios, with multiple environment transformations, thus providing a dynamic testbed for researchers to develop open-world AI agents.

AAMAS Conference 2024 Conference Paper

Oh, Now I See What You Want: Learning Agent Models with Internal States from Observations

• Panagiotis Lymperopoulos
• Matthias Scheutz

Learning behavior models of other agents from observations is challenging because agents typically do not act based on observable states alone, but usually take their internal, for external agents unobservable, states such as desires, motivations, preferences, and others into account. Consequently, methods that only use observational states for modeling other agents’ behaviors are insufficient for capturing and predicting agent behavior, especially for agents with rich internal processes. We propose a novel approach to online agent model learning that works incrementally with limited data, provides fine-grained and interpretable descriptions of the agent’s behavior, and, most importantly, is able to hypothesize agent-internal states to better explain observed behavioral trajectories. We show in various proof-of-concept experiments that our method avoids the pitfalls of common agent-modeling strategies when agent-internal states govern behavior and is able to build accurate and interpretable behavior models. We also discuss how the method can work in conjunction with existing approaches (e. g. , for goal recognition) to facilitate better modeling of open-world agents.

AAMAS Conference 2023 Conference Paper

Improving Human-Robot Team Performance with Proactivity and Shared Mental Models

• Gwendolyn Edgar
• Matthew McWilliams
• Matthias Scheutz

Recent work in human-robot teaming has demonstrated that when robots build and maintain “shared mental models”, the effectiveness of the whole human-robot team is overall better compared to a baseline with no shared mental models. In this work, we expand on this insight by introducing proactive behaviors to investigate potential further improvements of team performance and task efficiency. We hypothesize that, combined with shared mental models, robots with these more proactive behaviors become even more effective teammates. To this end, we developed a set of robot behaviors aligned with reactive, active and proactive team behaviors in human-human teams. We ran a human behavioral study to evaluate our system. The results show that proactive robot behavior improves task efficiency and performance over mere reactive behavior in high cognitive load environments.

UAI Conference 2023 Conference Paper

Investigating a Generalization of Probabilistic Material Implication and Bayesian Conditionals

• Michael Jahn
• Matthias Scheutz

Probabilistic "if A then B" rules are typically formalized as Bayesian conditionals P(B|A), as many (e. g. , Pearl) have argued that Bayesian conditionals are the correct way to think about such rules. However, there are challenges with standard inferences such as modus ponens and modus tollens that might make probabilistic material implication a better candidate at times for rule-based systems employing forward-chaining; and arguably material implication is still suitable when information about prior or conditional probabilities is not available at all. We investigate a generalization of probabilistic material implication and Bayesian conditionals that combines the advantages of both formalisms in a systematic way and prove basic properties of the generalized rule, in particular, for inference chains in graphs.

AAMAS Conference 2023 Conference Paper

Methods and Mechanisms for Interactive Novelty Handling in Adversarial Environments

• Tung Thai
• Mudit Verma
• Utkarsh Soni
• Sriram Gopalakrishnan
• Ming Shen
• Mayank Garg
• Ayush Kalani
• Nakul Vaidya

Learning to detect, characterize and accommodate novelties is a challenge that agents operating in open-world domains need to address to achieve satisfactory task performance. We sketch general methods for detecting and characterizing different types of novelties, and for building an appropriate adaptive model to accommodate them utilizing logical representations and reasoning methods in stochastic partially observable multi-agent environments. We also briefly report results from evaluations of our algorithms in the game domain of Monopoly. The results show high novelty detection and accommodation rates.

TMLR Journal 2023 Journal Article

NovelCraft: A Dataset for Novelty Detection and Discovery in Open Worlds

• Cynthia Feeney
• Sarah Schneider
• Panagiotis Lymperopoulos
• Liping Liu
• Matthias Scheutz
• Michael C Hughes

In order for artificial agents to successfully perform tasks in changing environments, they must be able to both detect and adapt to novelty. However, visual novelty detection research often only evaluates on repurposed datasets such as CIFAR-10 originally intended for object classification, where images focus on one distinct, well-centered object. New benchmarks are needed to represent the challenges of navigating the complex scenes of an open world. Our new NovelCraft dataset contains multimodal episodic data of the images and symbolic world-states seen by an agent completing a pogo stick assembly task within a modified Minecraft environment. In some episodes, we insert novel objects of varying size within the complex 3D scene that may impact gameplay. Our visual novelty detection benchmark finds that methods that rank best on popular area-under-the-curve metrics may be outperformed by simpler alternatives when controlling false positives matters most. Further multimodal novelty detection experiments suggest that methods that fuse both visual and symbolic information can improve time until detection as well as overall discrimination. Finally, our evaluation of recent generalized category discovery methods suggests that adapting to new imbalanced categories in complex scenes remains an exciting open problem.

AAMAS Conference 2023 Conference Paper

The Resilience Game: A New Formalization of Resilience for Groups of Goal-Oriented Autonomous Agents

• Michael A. Goodrich
• Jennifer Leaf
• Julie A. Adams
• Matthias Scheutz

Groups of autonomous robots should be resilient. They should have the ability to cope with unknown events, long-lasting alterations to the environment, degradation of capacities, robot losses, and changes to communication networks. This paper presents a multiagent resilience formulation for goal-based agents. The formulation applies to mixed motive groups where agent goals have commonalities but are not perfectly aligned. Resilient groups must not only be resilient to chance exogenous perturbations but also intentional endogenous perturbations among the agents. Defining resilience using expected utilities leads to a new way of looking at multiagent resilience, namely the resilience game. The resilience game makes it possible to use the notion of equilibrium from game theory to evaluate how the intentional stances of agents determine when multiagent algorithms are resilient. A guided diffusion of innovations problem is used to demonstrate how the resilience game provides insight into the effectiveness of various joint algorithms.

PRL Workshop 2022 Workshop Paper

Speeding-up Continual Learning through Information Gaines in Novel Experiences

• Pierrick Lorang
• Shivam Goel
• Patrik Zips
• Jivko Sinapov
• Matthias Scheutz

Adapting to novelties in open-world environments is an important and difficult challenge, and it has been recently shown that hybrid planning and reinforcement learning approaches can lead to better adaptations. However, these approaches still face intriguing difficulties induced by their heavy dependence on training samples to overcome changes in the environment quickly. In this work, we propose an integrated planning and learning approach that utilizes learning from failures and transferring knowledge over time to overcome novelty scenarios. Our proposed approach is much more sample efficient in adapting to sudden and unknown changes (i. e. , novelties) than the existing hybrid approaches. We showcase our results on a Minecraft-inspired gridworld environment called NovelGridworlds by injecting three novelties in the agent’s environment at test time. We show that our approach can speed up continual learning through information gained in each novel experience and, thus, more sample-efficient. Motivation: Fast Life-Long Learning* Figure 1: The impact of new task on performance is significantly reduced in the agents proposed in this paper compared to typical life-long learning agents and there are no negative effects when previously learned tasks need to be performed.

AAMAS Conference 2021 Conference Paper

A Novelty-Centric Agent Architecture for Changing Worlds

• Faizan Muhammad
• Vasanth Sarathy
• Gyan Tatiya
• Shivam Goel
• Saurav Gyawali
• Mateo Guaman
• Jivko Sinapov
• Matthias Scheutz

Open-world AI requires artificial agents to cope with novelties that arise during task performance, i. e. , they must (1) detect novelties, (2) characterize them, in order to (3) accommodate them, especially in cases where sudden changes to the environment make task accomplishment impossible without utilizing the novelty. We present a formal framework and implementation thereof in a cognitive agent for novelty handling and demonstrate the efficacy of the proposed methods for detecting and handling a large set of novelties in a crafting task in a simulated environment. We discuss the success of the proposed knowledge-based methods and propose heuristic extensions that will further improve novelty handling in open-worlds tasks.

AAAI Conference 2021 Conference Paper

Enabling Fast Instruction-Based Modification of Learned Robot Skills

• Tyler Frasca
• Bradley Oosterveld
• Meia Chita-Tegmark
• Matthias Scheutz

Much research effort in HRI has focused on how to enable robots to learn new skills from observations, demonstrations, and instructions. Less work, however, has focused on how skills can be corrected if they were learned incorrectly, adapted to changing circumstances, or generalized/specialized to different contexts. In this paper, a skill modification framework is introduced that allows users to modify a robot’s stored skills quickly through instructions to (1) reduce inefficiencies, (2) fix errors, and (3) enable generalizations, all in a way for modified skills to be immediately available for task performance. A thorough evaluation of the implemented framework shows the operation of the algorithms integrated in a cognitive robotic architecture on different fully autonomous robots in various HRI case studies. An additional online HRI user study verifies that subjects prefer to quickly modify robot knowledge in the way we proposed in the framework.

ICRA Conference 2021 Conference Paper

Robot Development and Path Planning for Indoor Ultraviolet Light Disinfection

• Jonathan Conroy
• Christopher Thierauf
• Parker Rule
• Evan A. Krause
• Hugo A. Akitaya
• Andrei Gonczi
• Matias Korman
• Matthias Scheutz

Regular irradiation of indoor environments with ultraviolet C (UVC) light has become a regular task for many in-door settings as a result of COVID-19, but current robotic systems attempting to automate it suffer from high costs and inefficient irradiation. In this paper, we propose a purpose-made inexpensive robotic platform with off-the-shelf components and standard navigation software that, with a novel algorithm for finding optimal irradiation locations, addresses both shortcomings to offer affordable and efficient solutions for UVC irradiation. We demonstrate in simulations the efficacy of the algorithm and show a prototypical run of the autonomous integrated robotic system in an indoor environment. In our sample instances, our proposed algorithm reduces the time needed by roughly 30% while it increases the coverage by a factor of 35% (when compared to the best possible placement of a static light).

AAMAS Conference 2021 Conference Paper

SPOTTER: Extending Symbolic Planning Operators through Targeted Reinforcement Learning

• Vasanth Sarathy
• Daniel Kasenberg
• Shivam Goel
• Jivko Sinapov
• Matthias Scheutz

Symbolic planning models allow decision-making agents to sequence actions in arbitrary ways to achieve a variety of goals in dynamic domains. However, they are typically handcrafted and tend to require precise formulations that are not robust to human error. Reinforcement learning (RL) approaches do not require such models, and instead learn domain dynamics by exploring the environment and collecting rewards. However, RL approaches tend to require millions of episodes of experience and often learn policies that are not easily transferable to other tasks. In this paper, we address one aspect of the open problem of integrating these approaches: how can decision-making agents resolve discrepancies in their symbolic planning models while attempting to accomplish goals? We propose an integrated framework named SPOTTER that uses RL to augment and support (“spot”) a planning agent by discovering new operators needed by the agent to accomplish goals that are initially unreachable for the agent. SPOTTER outperforms pure-RL approaches while also discovering transferable symbolic knowledge and does not require supervision, successful plan traces or any a priori knowledge about the missing planning operator.

ICAPS Conference 2020 Conference Paper

Generating Explanations for Temporal Logic Planner Decisions

• Daniel Kasenberg
• Ravenna Thielstrom
• Matthias Scheutz

Although temporal logic has been touted as a fruitful language for specifying interpretable agent objectives, there has been little emphasis on generating explanations for agents with temporal logic objectives. In this paper, we develop an approach to generating explanations for the behavior of agents planning with several temporal logic objectives. We focus on agents operating in deterministic Markov decision processes (MDPs), and specify objectives using linear temporal logic (LTL). Given an agent planning to maximally satisfy some set of LTL objectives (with an associated preference structure) in a deterministic MDP, we introduce an algorithm for constructing explanations answering both factual and “why” queries, which queries are also specified in LTL.

IROS Conference 2020 Conference Paper

Going Cognitive: A Demonstration of the Utility of Task-General Cognitive Architectures for Adaptive Robotic Task Performance

• Tyler M. Frasca
• Zhao Han
• Jordan Allspaw
• Holly A. Yanco
• Matthias Scheutz

It has been claimed that a main advantage of cognitive architectures (compared to other types of specialized robotic architectures) is that they are task-general and can thus learn to perform any task as long as they have the right perceptual and action primitives. In this paper, we provide empirical evidence for this claim by directly comparing a high-performing custom robotic architecture developed for the standardized robotic "FetchIt! " challenge task to a hybrid cognitive robotic architecture that allows for online one-shot task learning and task modifications through natural language instructions. The results show that there is no disadvantage of running the hybrid architecture (i. e. , no significant difference in overall performance or computational overhead compared to the custom architecture) while adding the flexibility of online one-shot task instruction and modification not available in the custom architecture.

KR Conference 2020 Conference Paper

Simultaneous Representation of Knowledge and Belief for Epistemic Planning with Belief Revision

• David Buckingham
• Daniel Kasenberg
• Matthias Scheutz

We propose a novel approach to the problem of false belief revision in epistemic planning. Our state representations are pointed Kripke models with two binary relations over possible worlds: one representing agents' necessarily true knowledge, and one representing agents' possibly false beliefs. State transition functions maintain S5n properties in the knowledge relation and KD45n properties in the belief relation. When new information contradicts an agent's beliefs, belief revision draws new possible worlds from the agent's knowledge relation. Our method also improves upon prior work by accommodating false announcements. We develop our system as an extension to the mA* action language, presenting transition functions for ontic, sensing, and announcement actions.

ICRA Conference 2019 Conference Paper

Acquisition of Word-Object Associations from Human-Robot and Human-Human Dialogues

• Sepideh Sadeghi
• Bradley Oosterveld
• Evan A. Krause
• Matthias Scheutz

Past work on acquisition of word-object associations in robots has focused on either fast instruction-based methods which accept highly constrained input or gradual cross-situational learning methods, but not a mixture of both. In this paper, we present an integrated robotic system which allows for a combination of these methods to contribute to the task of learning the labels of objects in AI agents. We demonstrate the expanded word learning capabilities in the outcome system and how learning from both human-human and human-robot dialogues can be achieved in one integrated system.

AAAI Conference 2019 Conference Paper

On Resolving Ambiguous Anaphoric Expressions in Imperative Discourse

• Vasanth Sarathy
• Matthias Scheutz

Anaphora resolution is a central problem in natural language understanding. We study a subclass of this problem involving object pronouns when they are used in simple imperative sentences (e. g. , “pick it up. ”). Specifically, we address cases where situational and contextual information is required to interpret these pronouns. Current state-of-the art statisticallydriven coreference systems and knowledge-based reasoning systems are insufficient to address these cases. In this paper, we introduce, with examples, a general class of situated anaphora resolution problems, propose a proof-of-concept system for disambiguating situated pronouns, and discuss some general types of reasoning that might be needed.

AAAI Conference 2018 Conference Paper

Early Syntactic Bootstrapping in an Incremental Memory-Limited Word Learner

• Sepideh Sadeghi
• Matthias Scheutz

It has been suggested that early human word learning occurs across learning situations and is bootstrapped by syntactic regularities such as word order. Simulation results from ideal learners and models assuming prior access to structured syntactic and semantic representations suggest that it is possible to jointly acquire word order and meanings and that learning is improved as each language capability bootstraps the other. We first present a probabilistic framework for early syntactic bootstrapping in the absence of advanced structured representations, then we use our framework to study the utility of joint acquisition of word order and word referent and its onset, in a memory-limited incremental model. Comparing learning results in the presence and absence of joint acquisition of word order in different ambiguous contexts, improvement in word order results showed an immediate onset, starting in early trials while being affected by context ambiguity. Improvement in word learning results on the other hand, was hindered in early trials where the acquired word order was imperfect, while being facilitated by word order learning in future trials as the acquired word order improved. Furthermore, our results showed that joint acquisition of word order and word referent facilitates one-shot learning of new words as well as inferring intentions of the speaker in ambiguous contexts.

AAAI Conference 2018 Conference Paper

Norm Conflict Resolution in Stochastic Domains

• Daniel Kasenberg
• Matthias Scheutz

Artificial agents will need to be aware of human moral and social norms, and able to use them in decision-making. In particular, artificial agents will need a principled approach to managing conflicting norms, which are common in human social interactions. Existing logic-based approaches suffer from normative explosion and are typically designed for deterministic environments; reward-based approaches lack principled ways of determining which normative alternatives exist in a given environment. We propose a hybrid approach, using Linear Temporal Logic (LTL) representations in Markov Decision Processes (MDPs), that manages norm conflicts in a systematic manner while accommodating domain stochasticity. We provide a proof-of-concept implementation in a simulated vacuum cleaning domain.

IJCAI Conference 2018 Conference Paper

Recursive Spoken Instruction-Based One-Shot Object and Action Learning

• Matthias Scheutz
• Evan Krause
• Bradley Oosterveld
• Tyler Frasca
• Robert Platt

Learning new knowledge from single instructions and being able to apply it immediately is highly desirable for artificial agents. We provide the first demonstration of spoken instruction-based one-shot object and action learning in a cognitive robotic architecture and briefly discuss the architectural modifications required to enable such fast learning, demonstrating the new capabilities on a fully autonomous robot.

AAMAS Conference 2017 Conference Paper

A Tale of Two Architectures: A Dual-Citizenship Integration of Natural Language and the Cognitive Map

• Tom Williams
• Collin Johnson
• Matthias Scheutz
• Benjamin Kuipers

Vulcan and DIARC are two robot architectures with very different capabilities: Vulcan uses rich spatial representations to facilitate navigation capabilities in real-world, campus-like environments, while DIARC uses high-level cognitive representations to facilitate human-like tasking through natural language. In this work, we show how the integration of Vulcan and DIARC enables not only the capabilities of the two individual architectures, but new synergistic capabilities as well, as each architecture leverages the strengths of the other. This integration presents interesting challenges, as DIARC and Vulcan are implemented in distinct multi-agent system middlewares. Accordingly, a second major contribution of this paper is the Vulcan-ADE Development Environment (VADE): a novel multi-agent system framework comprised of both (1) software agents belonging to a single robot architecture and implemented in a single multi-agent system middleware, and (2) “Dual-Citizen” agents that belong to both robot architectures and that use elements of both multi-agent system middlewares. As one example application, we demonstrate the implementation of the new joint architecture and novel multi-agent system framework on a robotic wheelchair, and show how this integration advances the state-of-the-art for NL-enabled wheelchairs.

AAMAS Conference 2017 Conference Paper

Cognition-enabled Task Interpretation for Human-Robot Teams in a Simulation-based Search and Rescue Mission

• Fereshta Yazdani
• Matthias Scheutz
• Michael Beetz

IROS Conference 2017 Conference Paper

Differences in interaction patterns and perception for teleoperated and autonomous humanoid robots

• Maxwell Bennett
• Tom Williams 0001
• Daria Thames
• Matthias Scheutz

As the linguistic capabilities of interactive robots advance, it becomes increasingly important to understand how humans will instruct robots through natural language. What is more, with the increased use of teleoperated humanoid robots, it is important to recognize whether any differences between instructions given to humans and to robots are due to the physical embodiment or to the perceived autonomy of the instructee. In this paper, we present the results of a human-subject experiment in which participants interacted in a collaborative, task-based setting with both a human and a suit-based, teleoperated humanoid robot said to be either autonomous or teleoperated. Our results suggest that humans will use politeness strategies equally with human, autonomous robotic, and teleoperated robotic teammates, reinforcing recent findings that autonomous robots must comprehend and appropriately respond to human utterances that follow such strategies. Our results also suggest variations in how different teammates were perceived. Specifically, our results suggest that human-teleoperated robots were perceived as less intelligent than human teammates; a finding with serious implications for human-robot team dynamics.

IS Journal 2017 Journal Article

Interactive Task Learning

• John E. Laird
• Kevin Gluck
• John Anderson
• Kenneth D. Forbus
• Odest Chadwicke Jenkins
• Christian Lebiere
• Dario Salvucci
• Matthias Scheutz

This article presents a new research area called interactive task learning (ITL), in which an agent actively tries to learn not just how to perform a task better but the actual definition of a task through natural interaction with a human instructor while attempting to perform the task. The authors provide an analysis of desiderata for ITL systems, a review of related work, and a discussion of possible application areas for ITL systems.

AAMAS Conference 2017 Conference Paper

Spoken Instruction-Based One-Shot Object and Action Learning in a Cognitive Robotic Architecture

• Matthias Scheutz
• Evan Krause
• Brad Oosterveld
• Tyler Frasca
• Robert Platt

Learning new knowledge from single instructions and being able to apply it immediately is a highly desirable capability for artificial agents. We provide the first demonstration of spoken instructionbased one-shot object and action learning in a cognitive robotic architecture and discuss the modifications to several architectural components required to enable such fast learning, demonstrating the new capabilities on two different fully autonomous robots. CCS Concepts •Human-centered computing → Natural language interfaces; •Computing methodologies → Online learning settings;

AAAI Conference 2016 Conference Paper

A Framework for Resolving Open-World Referential Expressions in Distributed Heterogeneous Knowledge Bases

• Tom Williams
• Matthias Scheutz

We present a domain-independent approach to reference resolution that allows a robotic or virtual agent to resolve references to entities (e. g. , objects and locations) found in open worlds when the information needed to resolve such references is distributed among multiple heterogeneous knowledge bases in its architecture. An agent using this approach can combine information from multiple sources without the computational bottleneck associated with centralized knowledge bases. The proposed approach also facilitates “lazy constraint evaluation”, i. e. , verifying properties of the referent through different modalities only when the information is needed. After specifying the interfaces by which a reference resolution algorithm can request information from distributed knowledge bases, we present an algorithm for performing open-world reference resolution within that framework, analyze the algorithm’s performance, and demonstrate its behavior on a simulated robot.

AAMAS Conference 2016 Conference Paper

Analogical Generalization of Actions from Single Exemplars in a Robotic Architecture

• Jason R. Wilson
• Evan Krause
• Matthias Scheutz
• Morgan Rivers

Humans are often able to generalize knowledge learned from a single exemplar. In this paper, we present a novel integration of mental simulation and analogical generalization algorithms into a cognitive robotic architecture that enables a similarly rudimentary generalization capability in robots. Specifically, we show how a robot can generate variations of a given scenario and then use the results of those new scenarios run in a physics simulator to generate generalized action scripts using analogical mappings. The generalized action scripts then allow the robot to perform the originally learned activity in a wider range of scenarios with different types of objects without the need for additional exploration or practice. In a proof-of-concept demonstration we show how the robot can generalize from a previously learned pickand-place action performed with a single arm on an object with a handle to a pick-and-place action of a cylindrical object with no handle with two arms.

KR Conference 2016 Short Paper

Cognitive Affordance Representations in Uncertain Logic

• Vasanth Sarathy
• Matthias Scheutz

The concept of “affordance” represents the relationship between human perceivers and their environment. Affordance perception, representation, and inference are central to commonsense reasoning, tool-use and creative problem-solving in artificial agents. Existing approaches fail to provide flexibility with which to reason about affordances in the open world, where they are influenced by changing context, social norms, historical precedence, and uncertainty. We develop a formal rulesbased logical representational format coupled with an uncertainty-processing framework to reason about cognitive affordances in a more general manner than shown in the existing literature. Our framework allows agents to make deductive and abductive inferences about functional and social affordances, collectively and dynamically, thereby allowing the agent to adapt to changing conditions. We demonstrate our approach with an example, and show that an agent can successfully reason through situations that involve a tight interplay between various social and functional norms. Background James Gibson (1979) introduced the concept of “affordance” to represent the relationship between the agent and its environment. Past work in formalizing this relationship has largely focused on modeling affordance using either statistical formalisms or ontology-based approaches. For example, Montesano et al. have developed statistically inspired causal models of affordance using Bayesian Networks to formalize the relationship between object features, actions and effects (Montesano et al. 2007). Varadarajan et al. (Varadarajan 2015) developed a detailed knowledge-ontology based on conceptual, functional and part properties of objects, and then used a combination of detection and query matching algorithms to pinpoint the affordances for objects. Despite these efforts, affordance representation faces many challenges that have not been overcome in the previous work. These approaches fail to provide flexibility with which to reason about affordances in the open world, where they are influenced by changing context, social norms, historical precedence, and uncertainty. For example, these current approaches cannot reason that coffee mugs afford grasping and drinking, while also affording serving as a paperweight or cupholder, or depending on the context, as family heirloom not meant to be used at all.

AAAI Conference 2015 Conference Paper

Going Beyond Literal Command-Based Instructions: Extending Robotic Natural Language Interaction Capabilities

• Tom Williams
• Gordon Briggs
• Bradley Oosterveld
• Matthias Scheutz

The ultimate goal of human natural language interaction is to communicate intentions. However, these intentions are often not directly derivable from the semantics of an utterance (e. g. , when linguistic modulations are employed to convey politeness, respect, and social standing). Robotic architectures with simple command-based natural language capabilities are thus not equipped to handle more liberal, yet natural uses of linguistic communicative exchanges. In this paper, we propose novel mechanisms for inferring intentions from utterances and generating clarification requests that will allow robots to cope with a much wider range of task-based natural language interactions. We demonstrate the potential of these inference algorithms for natural humanrobot interactions by running them as part of an integrated cognitive robotic architecture on a mobile robot in a dialoguebased instruction task.

IROS Conference 2015 Conference Paper

Planning for serendipity

• Tathagata Chakraborti
• Gordon Briggs
• Kartik Talamadupula
• Yu Zhang 0055
• Matthias Scheutz
• David E. Smith 0001
• Subbarao Kambhampati

Recently there has been a lot of focus on human robot co-habitation issues that are often orthogonal to many aspects of human-robot teaming; e. g. on producing socially acceptable behaviors of robots and de-conflicting plans of robots and humans in shared environments. However, an interesting offshoot of these settings that has largely been overlooked is the problem of planning for serendipity - i. e. planning for stigmergic collaboration without explicit commitments on agents in co-habitation. In this paper we formalize this notion of planning for serendipity for the first time, and provide an Integer Programming based solution for this problem. Further, we illustrate the different modes of this planning technique on a typical Urban Search and Rescue scenario and show a real-life implementation of the ideas on the Nao Robot interacting with a human colleague.

IROS Conference 2015 Conference Paper

POWER: A domain-independent algorithm for Probabilistic, Open-World Entity Resolution

• Tom Williams 0001
• Matthias Scheutz

The problem of uniquely identifying an entity described in natural language, known as reference resolution, has become recognized as a critical problem for the field of robotics, as it is necessary in order for robots to be able to discuss, reason about, or perform actions involving any people, locations, or objects in their environments. However, most existing algorithms for reference resolution are domain-specific and limited to environments assumed to be known a priori. In this paper we present an algorithm for reference resolution which is both domain independent and designed to operate in an open world. We call this algorithm POWER: Probabilistic Open-World Entity Resolution. We then present the results of an empirical study demonstrating the success of POWER both in properly identifying the referents of referential expressions and in properly modifying the world model based on such expressions.

IROS Conference 2014 Conference Paper

Coordination in human-robot teams using mental modeling and plan recognition

• Kartik Talamadupula
• Gordon Briggs
• Tathagata Chakraborti
• Matthias Scheutz
• Subbarao Kambhampati

Beliefs play an important role in human-robot teaming scenarios, where the robots must reason about other agents' intentions and beliefs in order to inform their own plan generation process, and to successfully coordinate plans with the other agents. In this paper, we cast the evolving and complex structure of beliefs, and inference over them, as a planning and plan recognition problem. We use agent beliefs and intentions modeled in terms of predicates in order to create an automated planning problem instance, which is then used along with a known and complete domain model in order to predict the plan of the agent whose beliefs are being modeled. Information extracted from this predicted plan is used to inform the planning process of the modeling agent, to enable coordination. We also look at an extension of this problem to a plan recognition problem. We conclude by presenting an evaluation of our technique through a case study implemented on a real robot.

IROS Conference 2014 Conference Paper

Investigating human perceptions of robot capabilities in remote human-robot team tasks based on first-person robot video feeds

• Cody Canning
• Thomas J. Donahue
• Matthias Scheutz

It is well-known that a robot's appearance and its observable behavior can affect a human interactant's perceptions of the robot's capabilities and propensities in settings where humans and robots are co-located; for remote interactions the specific effects are less clear. Here, we use a remote interaction setting to investigate possible effects of simulated versus real first-person robot video feeds. The first experiment uses subject-level comparisons of the two video conditions in a multi-robot setting while the second and third experiments focus on a single robot, single video condition using a larger population (via Amazon Mechanical Turk) to study between-subjects effects. The latter experiments also probe the effects of robot appearance, video feed type, and stake humans have in the task. We observe a complex interplay between interaction, robot appearance, and video feed type as they affect perceived collaboration, utility, competence, and warmth of the robot.

AAAI Conference 2014 Conference Paper

Learning to Recognize Novel Objects in One Shot through Human-Robot Interactions in Natural Language Dialogues

• Evan Krause
• Michael Zillich
• Thomas Williams
• Matthias Scheutz

Being able to quickly and naturally teach robots new knowledge is critical for many future open-world human-robot interaction scenarios. In this paper we present a novel approach to using natural language context for one-shot learning of visual objects, where the robot is immediately able to recognize the described object. We describe the architectural components and demonstrate the proposed approach on a robotic platform in a proof-of-concept evaluation.

AAAI Conference 2013 Conference Paper

A Hybrid Architectural Approach to Understanding and Appropriately Generating Indirect Speech Acts

• Gordon Briggs
• Matthias Scheutz

Current approaches to handling indirect speech acts (ISAs) do not account for their sociolinguistic underpinnings (i. e. , politeness strategies). Deeper understanding and appropriate generation of indirect acts will require mechanisms that integrate natural language (NL) understanding and generation with social information about agent roles and obligations, which we introduce in this paper. Additionally, we tackle the problem of understanding and handling indirect answers that take the form of either speech acts or physical actions, which requires an inferential, plan-reasoning approach. In order to enable artificial agents to handle an even wider-variety of ISAs, we present a hybrid approach, utilizing both the idiomatic and inferential strategies. We then demonstrate our system successfully generating indirect requests and handling indirect answers, and discuss avenues of future research.

AAAI Conference 2013 Conference Paper

Grounding Natural Language References to Unvisited and Hypothetical Locations

• Thomas Williams
• Rehj Cantrell
• Gordon Briggs
• Paul Schermerhorn
• Matthias Scheutz

While much research exists on resolving spatial natural language references to known locations, little work deals with handling references to unknown locations. In this paper we introduce and evaluate algorithms integrated into a cognitive architecture which allow an agent to learn about its environment while resolving references to both known and unknown locations. We also describe how multiple components in the architecture jointly facilitate these capabilities.

ICRA Conference 2012 Conference Paper

Abstract planning for reactive robots

• Saket Joshi
• Paul W. Schermerhorn
• Roni Khardon
• Matthias Scheutz

Hybrid reactive-deliberative architectures in robotics combine reactive sub-policies for fast action execution with goal sequencing and deliberation. The need for replanning, however, presents a challenge for reactivity and hinders the potential for guarantees about the plan quality. In this paper, we argue that one can integrate abstract planning provided by symbolic dynamic programming in first order logic into a reactive robotic architecture, and that such an integration is in fact natural and has advantages over traditional approaches. In particular, it allows the integrated system to spend off-line time planning for a policy, and then use the policy reactively in open worlds, in situations with unexpected outcomes, and even in new environments, all by simply reacting to a state change executing a new action proposed by the policy. We demonstrate the viability of the approach by integrating the FODD-Planner with the robotic DIARC architecture showing how an appropriate interface can be defined and that this integration can yield robust goal-based action execution on robots in open worlds.

AAAI Conference 2012 Conference Paper

Crossing Boundaries: Multi-Level Introspection in a Complex Robotic Architecture for Automatic Performance Improvements

• Evan Krause
• Paul Schermerhorn
• Matthias Scheutz

Introspection mechanisms are employed in agent architectures to improve agent performance. However, there is currently no approach to introspection that makes automatic adjustments at multiple levels in the implemented agent system. We introduce our novel multi-level introspection framework that can be used to automatically adjust architectural configurations based on the introspection results at the agent, infrastructure and component level. We demonstrate the utility of such adjustments in a concrete implementation on a robot where the high-level goal of the robot is used to automatically configure the vision system in a way that minimizes resource consumption while improving overall task performance.

AAMAS Conference 2012 Conference Paper

What am I doing? Automatic Construction of an Agent's State-Transition Diagram through Introspection

• Constantin Berzan
• Matthias Scheutz

Infrastructures for implementing agent architectures are currently unaware of what tasks the implemented agent is performing. Such knowledge would allow the infrastructure to improve the agent's autonomy and reliability. For example, the infrastructure could detect abnormal system states, predict likely faults and take preventive measures ahead of time, or balance system load based on predicted computational needs. In this paper we introduce a learning algorithm to automatically discover a state-transition model of the agent's behavior. The algorithm monitors the communication between architectural components, in the form of function calls, and finds the frequencies at which various functions are polled. It then determines the states according to what polling frequencies are active at any time. The two main novel features of the algorithm are that it is completely \emph{unsupervised} (it requires no human input) and \emph{task-agnostic} (it can be applied to any new task or architecture with minimal effort).

AAAI Conference 2010 Conference Paper

Integrating a Closed World Planner with an Open World Robot: A Case Study

• Kartik Talamadupula
• J. Benton
• Paul Schermerhorn
• Subbarao Kambhampati
• Matthias Scheutz

In this paper, we present an integrated planning and robotic architecture that actively directs an agent engaged in an urban search and rescue (USAR) scenario. We describe three salient features that comprise the planning component of this system, namely (1) the ability to plan in a world open with respect to objects, (2) execution monitoring and replanning abilities, and (3) handling soft goals, and detail the interaction of these parts in representing and solving the USAR scenario at hand. We show that though insufficient in an individual capacity, the integration of this trio of features is sufficient to solve the scenario that we present. We test our system with an example problem that involves soft and hard goals, as well as goal deadlines and action costs, and show that the planner is capable of incorporating sensing actions and execution monitoring in order to produce goal-fulfilling plans that maximize the net benefit accrued.

TIST Journal 2010 Journal Article

Planning for human-robot teaming in open worlds

• Kartik Talamadupula
• J. Benton
• Subbarao Kambhampati
• Paul Schermerhorn
• Matthias Scheutz

As the number of applications for human-robot teaming continue to rise, there is an increasing need for planning technologies that can guide robots in such teaming scenarios. In this article, we focus on adapting planning technology to Urban Search And Rescue (USAR) with a human-robot team. We start by showing that several aspects of state-of-the-art planning technology, including temporal planning, partial satisfaction planning, and replanning, can be gainfully adapted to this scenario. We then note that human-robot teaming also throws up an additional critical challenge, namely, enabling existing planners, which work under closed-world assumptions, to cope with the open worlds that are characteristic of teaming problems such as USAR. In response, we discuss the notion of conditional goals, and describe how we represent and handle a specific class of them called open world quantified goals. Finally, we describe how the planner, and its open world extensions, are integrated into a robot control architecture, and provide an empirical evaluation over USAR experimental runs to establish the effectiveness of the planning components.

ICRA Conference 2010 Conference Paper

Using logic to handle conflicts between system, component, and infrastructure goals in complex robotic architectures

• Paul W. Schermerhorn
• Matthias Scheutz

Complex robots with many interacting components in their control architectures are subject to component failures from which neither the control architecture nor the implementing infrastructure can recover. Moreover, the operating conditions for these components might be at odds with goals the robot might have adopted (e. g. , through external commands or in the course of the execution of the current task). We argue that the best (if not the only) way to resolve any difficulties that arise from the different requirements at the agent, component and infrastructure levels is to use a common formal logical goal representation for all three layers. We discuss how these representations can be integrated into a complex robotic architecture and demonstrate in an experimental evaluation on a robot how the architecture can recover from a failure situation that it would not have been able to handle without explicit multi-level unified goal representations and their associated monitoring and reasoning processes.

IROS Conference 2009 Conference Paper

Finding and exploiting goal opportunities in real-time during plan execution

• Paul W. Schermerhorn
• J. Benton 0001
• Matthias Scheutz
• Kartik Talamadupula
• Subbarao Kambhampati

Autonomous robots that operate in real-world domains face multiple challenges that make planning and goal selection difficult. Not only must planning and execution occur in real time, newly acquired knowledge can invalidate previous plans, and goals and their utilities can change during plan execution. However, these events can also provide opportunities, if the architecture is designed to react appropriately. We present here an architecture that integrates the SapaReplan planner with the DIARC robot architecture, allowing the architecture to react dynamically to changes in the robot's goal structures.

IROS Conference 2009 Conference Paper

Gendered voice and robot entities: Perceptions and reactions of male and female subjects

• Charles R. Crowell
• Michael Villano
• Matthias Scheutz
• Paul W. Schermerhorn

There is recent evidence that males and females view robots differently, from the way robots are conceptualized, to the way humans respond when they interact with them. In this paper, we further explore gender-based differences in human-robot interaction. Moreover, we provide the first available evidence for sex-related differences in reactions to gendered synthetic voices that are either disembodied or physically embodied within a robot. Results indicate that physical embodiment and perceived entity gender may interact with human sex-related characteristics and pre-experimental attitudes in determining how people respond to artificial entities.

ICRA Conference 2009 Conference Paper

What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution

• Juraj Dzifcak
• Matthias Scheutz
• Chitta Baral
• Paul W. Schermerhorn

Robots that can be given instructions in spoken language need to be able to parse a natural language utterance quickly, determine its meaning, generate a goal representation from it, check whether the new goal conflicts with existing goals, and if acceptable, produce an action sequence to achieve the new goal (ideally being sensitive to the existing goals). In this paper, we describe an integrated robotic architecture that can achieve the above steps by translating natural language instructions incrementally and simultaneously into formal logical goal description and action languages, which can be used both to reason about the achievability of a goal as well as to generate new action scripts to pursue the goal. We demonstrate the implementation of our approach on a robot taking spoken natural language instructions in an office environment.

AAMAS Conference 2008 Conference Paper

Physical Parameter Optimization in Swarms of Ultra-Low Complexity Agents

• Ryan Connaughton
• Paul Schermerhorn
• Matthias Scheutz

Physical agents (such as wheeled vehicles, UAVs, hovercraft, etc.) with simple control systems are often sensitive to changes in their physical design and control parameters. As such, it is crucial to evaluate the agent’s control systems together with the agent’s physical implementation. This can consequently lead to an explosion in the parameter space to be considered. In this paper we investigate the use of swarms of ultra-low complexity agents, and address the issue of finding workable physical agent parameters. We describe a technique for reducing the dimensionality of the search space by performing evaluation tasks that can be used to predict near-optimal parameter values for agents in related multi-agent tasks. We validate our approach on an example task, and demonstrate that this technique can greatly reduce the computational resources required to design a multi-agent system.

IROS Conference 2007 Conference Paper

"Talk to me! ": enabling communication between robotic architectures and their implementing infrastructures

• James F. Kramer
• Matthias Scheutz
• Paul W. Schermerhorn

Complex, autonomous robots integrate a large set of sometimes very diverse algorithms across at least three levels of system organization: the agent architecture, the implementation environment, and the hardware devices. Insofar as a distinction is maintained between them, the levels serve different purposes and thus exhibit different characteristic strengths and weaknesses. Exchanging information among organizational levels can be used to mitigate the shortcomings of one level by making use of the strengths of another. In this paper, we highlight the roles, characteristics, and relations between the infrastructure and the architecture of complex robots, describing a novel form of integration that results from enabling the exchange of information between these two levels, which otherwise is maintained internally. The information from the infrastructure is especially amenable for use by the architecture to achieve a higher level of robustness and system awareness. We demonstrate the functionality and utility of the proposed mechanisms in a set of experiments in which failures of architectural components are induced on an actual robot engaged in a joint human-robot team task.

ICRA Conference 2007 Conference Paper

Reflection and Reasoning Mechanisms for Failure Detection and Recovery in a Distributed Robotic Architecture for Complex Robots

• Matthias Scheutz
• James F. Kramer

Complex robots that interact naturally with humans require the integration, coordination and maintenance of many diverse software components and algorithms. An architecture that incorporates explicit knowledge about the relationships among these components and the overall system state can be used for introspection and consequently to reason about the best configurations of the computing environment under changing conditions; potential uses include maintaining the system's integrity, promoting its health, and providing the ability to dynamically reconfigure system components (e. g. , after component failure). In this paper, we describe a rudimentary reasoning system, part of our distributed integrated affect reflection cognition (DIARC) architecture for human-robot interaction, that can autonomously perform failure detection, failure recovery, and system reconfiguration of distributed architectural components to ensure sustained operation and interactions. We demonstrate the functionality and utility of the proposed mechanisms on a robot, where architectural components are forcefully removed by hand and automatically recovered by the system while the robot is continuing its interactions with humans as part of a joint human-robot task.

IROS Conference 2007 Conference Paper

Speech and action: integration of action and language for mobile robots

• Timothy R. Brick
• Paul W. Schermerhorn
• Matthias Scheutz

We describe the tight integration of incremental natural language understanding, goal management, and action processing in a complex robotic architecture, which is required for natural interactions between robots and humans. Specifically, the natural language components need to process utterances while they are still spoken to be able to initiate feedback actions in a timely fashion, while the action manager might need information at various points during action execution that must be obtained from humans. We argue that a finer- grained integration provides much more natural human-robot interactions and much more reasonable multitasking.

IROS Conference 2006 Conference Paper

ADE: A Framework for Robust Complex Robotic Architectures

• James F. Kramer
• Matthias Scheutz

Robots that can interact naturally with humans require the integration and coordination of many different components with heavy computational demands. We argue that an architecture framework with facilities for dynamic, reliable, fault-recovering, remotely accessible, distributed computing is needed for the development and operation of applications that support and enhance human activities and capabilities. We describe a robotic architecture development system, called ADE, that is built on top of a multi-agent system in order to provide all of the above features. Specifically, we discuss support for autonomic computing in ADE, briefly comparing it to related features of other commonly used robotic systems. We also report our experiences with ADE in the development of an architecture for an intelligent robot assistant and provide experimental results demonstrating the system's utility

AAMAS Conference 2006 Conference Paper

RADIC - A Generic Component for the Integration of Existing Reactive and Deliberative Layers

• Matthias Scheutz
• James Kramer

AAMAS Conference 2006 Conference Paper

Social Coordination without Communication in Multi-Agent Territory Exploration Tasks

• Paul Schermerhorn
• Matthias Scheutz

AAAI Conference 2005 Conference Paper

Toward Affective Cognitive Robots for Human-Robot Interaction

• Matthias Scheutz
• Chris Middendorff
• Michael Heilman

We present a brief overview of an architecture for a complex affective robot for human-robot interaction.

AAAI Conference 2004 System Paper

A Robotic Model of Human Reference Resolution

• Matthias Scheutz

Evidence from psychology suggests that humans process definite descriptions that refer to objects present in a visual scene incrementally upon hearing them, rather than constructing explicit parse trees after the whole sentence was said, which are then used to determine the referents. In this paper, we describe a real-time distributed robotic architectures for human reference resolution that demonstrates various interactions of auditory, visual, and semantic processing components hypothesized to underlie human processes.

IROS Conference 2004 Conference Paper

Fast, reliable, adaptive, bimodal people tracking for indoor environments

• Matthias Scheutz
• John McRaven
• György Cserey

We present a real-time system for a mobile robot that can reliably detect and track people in uncontrolled indoor environments. The system uses a combination of leg detection based on distance information from a laser range sensor and visual face detection based on an analogical algorithm implemented on specialized hardware (the CNN universal machine). Results from tests in a variety of environments with different lighting conditions, a different number of appearing and disappearing people, and different obstacles are reported to demonstrate that the system can find and subsequently track several, possibly people simultaneously in indoor environments. Applications of the system include in particular service robots for social events.

AAMAS Conference 2004 Conference Paper

Integrating Theory and Practice: The APOC Architecture Framework and its Development Environment ADE

• Andronache Virgil
• Matthias Scheutz

AAAI Conference 2004 Conference Paper

Useful Roles of Emotions in Artificial Agents: A Case Study from Artificial Life

• Matthias Scheutz

In this paper, we discuss the role of emotions in AI and possible ways to determine their utility for the design of artificial agents. We propose a research methodology for determining the utility of emotional control and apply it to the study of autonomous agents that compete for resources in an artificial life environment. The results show that the emotional control can improve performance in some circumstances.