Andrea Thomaz

ICRA Conference 2023 Conference Paper

Using Learning Curve Predictions to Learn from Incorrect Feedback

• Taylor Kessler Faulkner
• Andrea Thomaz

Robots can incorporate data from human teachers when learning new tasks. However, this data can often be noisy, which can cause robots to learn slowly or not at all. One method for learning from human teachers is Human-in-the-loop Reinforcement Learning (HRL), which can combine information from both an environmental reward and external feedback from human teachers. However, many HRL methods assume near-perfect information from teachers or must know the skill level of each teacher before starting the learning process. Our algorithm, Classification for Learning Erroneous Assessments using Rewards (CLEAR), is a feedback filter for Reinforcement Learning (RL) algorithms, enabling learning agents to learn from imperfect teachers without prior modeling. CLEAR is able to determine whether human feedback is correct based on observations of the RL learning curve. Our results suggest that CLEAR improves the quality of human feedback - from 57. 5% to 65% correct in a human study - and performs more reliably than baselines by matching or outperforming RL without human teachers in all tested cases.

IJCAI Conference 2022 Conference Paper

Abstraction in Data-Sparse Task Transfer (Extended Abstract)

• Tesca Fitzgerald
• Ashok Goel
• Andrea Thomaz

When a robot adapts a learned task for a novel environment, any changes to objects in the novel environment have an unknown effect on its task execution. For example, replacing an object in a pick-and-place task affects where the robot should target its actions, but does not necessarily affect the underlying action model. In contrast, replacing a tool that the robot will use to complete a task will effectively alter its end-effector pose with respect to the robot's base coordinate system, and thus the robot's motion must be replanned accordingly. These examples highlight the relationship among (i) differences between the source and target environments, (ii) the level of abstraction at which a robot's task model should be represented to enable transfer to the target environment, and (iii) the information needed to ground the abstracted task representation in the target environment. In this abstract, summarizing our full article [Fitzgerald et al. , 2021], we present our taxonomy of transfer problems based on this relationship. We also describe a knowledge representation called the Tiered Task Abstraction (TTA) and demonstrate its applicability to a variety of transfer problems in the taxonomy. Our experimental results indicate a trade-off between the generality and data requirements of a task representation, and reinforce the need for multiple transfer methods that operate at different levels of abstraction.

IROS Conference 2022 Conference Paper

Understanding Acoustic Patterns of Human Teachers Demonstrating Manipulation Tasks to Robots

• Akanksha Saran
• Kush Desai
• Mai Lee Chang
• Rudolf Lioutikov
• Andrea Thomaz
• Scott Niekum

Humans use audio signals in the form of spoken language or verbal reactions effectively when teaching new skills or tasks to other humans. While demonstrations allow humans to teach robots in a natural way, learning from trajectories alone does not leverage other available modalities including audio from human teachers. To effectively utilize audio cues accompanying human demonstrations, first it is important to understand what kind of information is present and conveyed by such cues. This work characterizes audio from human teachers demonstrating multi-step manipulation tasks to a situated Sawyer robot along three dimensions: (1) duration of speech used, (2) expressiveness in speech or prosody, and (3) semantic content of speech. We analyze these features for four different independent variables and find that teachers convey similar semantic content via spoken words for different conditions of (1) demonstration types, (2) audio usage instructions, (3) subtasks, and (4) errors during demonstrations. However, differentiating properties of speech in terms of duration and expressiveness are present for the four independent variables, highlighting that human audio carries rich information, potentially beneficial for technological advancement of robot learning from demonstration methods.

AIJ Journal 2021 Journal Article

Abstraction in data-sparse task transfer

• Tesca Fitzgerald
• Ashok Goel
• Andrea Thomaz

When a robot adapts a learned task for a novel environment, any changes to objects in the novel environment have an unknown effect on its task execution. For example, replacing an object in a pick-and-place task affects where the robot should target its actions, but does not necessarily affect the underlying action model. In contrast, replacing a tool that the robot will use to complete a task will effectively alter its end-effector pose with respect to the robot's base coordinate system, and thus the robot's motion must be replanned accordingly. These examples highlight the relationship among (i) differences between the source and target environments, (ii) the level of abstraction at which a robot's task model should be represented to enable transfer to the target environment, and (iii) the information needed to ground the abstracted task representation in the target environment. In this article, we present a taxonomy of transfer problems based on this relationship. We also describe a knowledge representation called the Tiered Task Abstraction (TTA) and demonstrate its applicability to a variety of transfer problems in the taxonomy. Our experimental results indicate a trade-off between the generality and data requirements of a task representation, and reinforce the need for multiple transfer methods that operate at different levels of abstraction.

ICRA Conference 2021 Conference Paper

Communication Strategy for Efficient Guidance Providing: Domain-structure Awareness, Performance Trade-offs, and Value of Future Observations

• Shih-Yun Lo
• Andrea Thomaz

Service robots are gaining capabilities to be deployed in public environments for human assistance. While robot actively providing guidance has shown great success in field study, the communication strategy (the strategy to decide whom to initiate the service for and when), and hence the performance evaluation, has been based on behavioral-based qualitative analysis. We attribute this to the challenge of accessing large-scale field data with condition control, and approach the problem with simulation from the agent-based modeling literature, to simulate pedestrian behavior in unfamiliar environments and estimate travel cost. We contribute a planning approach that uses the pedestrian behavior prediction from the model, to decide whom to initiate guidance and when for performance maximization. The results suggest that our approach is more efficient based on the measure of saved pedestrian travel time, compared to the behavioral-based strategy and a baseline that maximizes service counts.

ICRA Conference 2021 Conference Paper

Robust Planning with Emergent Human-like Behavior for Agents Traveling in Groups

• Shih-Yun Lo
• Elaine Schaertl Short
• Andrea Thomaz

To enable robots to smoothly interact with humans during their travels together as a group, robots need the ability to adapt their motions under environmental changes and ensure all group members’ routes are feasible. To achieve this ability, robots require knowledge of the final destination and the subgoals in between. In practice, such information is seldom shared explicitly among group members, and may be frequently updated. Under this uncertain setting, maintaining travel efficiency and behavior appropriateness becomes a challenge. Previous literature approached the problem by generating compliant coordinating motions inspired by human groups, with subgoal uncertainty remaining isolated from the plan evaluation process. We show that such coordination can lead the robot to "bad" transient states where inefficient planning and lost tracking may incur. We propose to resolve the problem by formulating the coordinating motion as a Bayesian stochastic game, to plan for the robot as a group member, in the meanwhile considering the long-term effect of uncertainty during path coordination. We show that the approach improves travel efficiency and partner tracking robustness, by preventing assertive decisions during the inference update process. Moreover, the approach presents "agency", in the sense that it can generate human-like motions, which can be applied and contribute to the pedestrian simulation literature; the approach also affords variants from the human-like motions to generate robot behaviors based on sensing capabilities, contributing to the methodology of robot behavior design.

ICRA Conference 2021 Conference Paper

Towards Safe Motion Planning in Human Workspaces: A Robust Multi-agent Approach

• Shih-Yun Lo
• Benito Fernandez
• Peter Stone 0001
• Andrea Thomaz

It is becoming increasingly feasible for robots to share a workspace with humans. However, for them to do so safely while maintaining agile performance, they need the ability to smoothly handle the dynamics and uncertainty caused by human motions. Markov Decision Processes (MDPs) serve as a common framework to formulate robot planning problems. However, because of its single-agent formulation, such planner cannot account for human reaction when evaluating robot actions. The robot can thus suffer from unsafe motions and move in ways that are hard for nearby humans to understand. To resolve this, we instead model robot planning in human workspaces as a Stochastic Game, and contribute a robust planning algorithm, which enables the robot to account for its prediction errors in human responses to prevent collision, while not losing agility, opposed to traditional maximin optimization techniques, by applying maximin operation only at "critical states". We validate the approach under partial knowledge of pedestrian behaviors, and show that our approach encounters zero collision despite imperfect prediction, while improving path efficiency, compared to baselines.

ICRA Conference 2020 Conference Paper

Interactive Reinforcement Learning with Inaccurate Feedback

• Taylor Kessler Faulkner
• Elaine Schaertl Short
• Andrea Thomaz

Interactive Reinforcement Learning (RL) enables agents to learn from two sources: rewards taken from observations of the environment, and feedback or advice from a secondary critic source, such as human teachers or sensor feedback. The addition of information from a critic during the learning process allows the agents to learn more quickly than non-interactive RL. There are many methods that allow policy feedback or advice to be combined with RL. However, critics can often give imperfect information. In this work, we introduce a framework for characterizing Interactive RL methods with imperfect teachers and propose an algorithm, Revision Estimation from Partially Incorrect Resources (REPaIR), which can estimate corrections to imperfect feedback over time. We run experiments both in simulations and demonstrate performance on a physical robot, and find that when baseline algorithms do not have prior information on the exact quality of a feedback source, using REPaIR matches or improves the expected performance of these algorithms.

IROS Conference 2020 Conference Paper

TASC: Teammate Algorithm for Shared Cooperation

• Mai Lee Chang
• Taylor Kessler Faulkner
• Thomas Benjamin Wei
• Elaine Schaertl Short
• Gokul Anandaraman
• Andrea Thomaz

For robots to be perceived as full-fledged team members, they must display intelligent behavior along multiple dimensions. One challenge is that even when the robot and human are on the same team, the interaction may not feel like teamwork to the human. We present a novel algorithm, Teammate Algorithm for Shared Cooperation (TASC). TASC is motivated by the concept of shared cooperative activity (SCA) for human-human teamwork, developed in prior work by Bratman. We focus on enabling the robot to prioritize certain SCA facets in its action selection depending on the task. We evaluated TASC in three experiments using different tasks with human users on Amazon Mechanical Turk. Our results show that TASC enabled participants to predict the robot’s goal earlier by one robot move and with greater confidence. The robot also helped reduce participants’ energy usage in a simulated block-moving task. Altogether, these results show that considering the SCA facets in the robot’s action selection improves teamwork.

AAMAS Conference 2019 Conference Paper

Human-guided Trajectory Adaptation for Tool Transfer

• Tesca Fitzgerald
• Elaine Short
• Ashok Goel
• Andrea Thomaz

We introduce “transfer by correction": a method for transferring a robot’s tool-based task models to use unfamiliar tools. By having the robot receive corrections from a human teacher when repeating a known task with a new tool, it can learn the relationship between the two tools, allowing it to transfer additional tasks learned with the original tool to the new tool. The goal is to enable the robot to generalize its task knowledge to accommodate tool replacements and thus be more robust to changes in its environment. We demonstrate how the tool transform models learned from one episode of task corrections can be used to perform that task with ≥ 85% of maximum performance in 83% of tool/task combinations. Furthermore, these transformations generalize to unseen tool/task combinations in 27. 8% of our transfer evaluations, and up to 41% of transfer problems when the source and replacement tool share tooltip similarities. Overall, these results indicate that successful task adaptation for a new tool is dependent on the the tool’s usage within that task, and that the transform model learned from interactive corrections can be generalized to other tasks providing a similar context for the new tool.

ICRA Conference 2019 Conference Paper

Real-time Multisensory Affordance-based Control for Adaptive Object Manipulation

• Vivian Chu
• Reymundo A. Gutierrez
• Sonia Chernova
• Andrea Thomaz

We address the challenge of how a robot can adapt its actions to successfully manipulate objects it has not previously encountered. We introduce Real-time Multisensory Affordance-based Control (RMAC), which enables a robot to adapt existing affordance models using multisensory inputs. We show that using the combination of haptic, audio, and visual information with RMAC allows the robot to learn afforance models and adaptively manipulate two very different objects (drawer, lamp), in multiple novel configurations. Offline evaluations and real-time online evaluations show that RMAC allows the robot to accurately open different drawer configurations and turn-on novel lamps with an average accuracy of 75%.

AAMAS Conference 2018 Conference Paper

Affordance Discovery using Simulated Exploration

• Adam Allevato
• Andrea Thomaz
• Mitch Pryor

IROS Conference 2018 Conference Paper

Effects of Integrated Intent Recognition and Communication on Human-Robot Collaboration

• Mai Lee Chang
• Reymundo A. Gutierrez
• Priyanka Khante
• Elaine Schaertl Short
• Andrea Thomaz

Human-robot interaction research to date has investigated intent recognition and communication separately. In this paper, we explore the effects of integrating both the robot's ability to generate intentional motion and predict the human's motion in a collaborative physical task. We implemented an intent recognition system to recognize the human partner's hand motion intent and a motion planner system to enable the robot to communicate its intent by using legible and predictable motion. We tested this bi-directional intent system in a 2-way within-subjects user study. Results suggest that an integrated intent recognition and communication system may facilitate more collaborative behavior among team members.

IROS Conference 2018 Conference Paper

Human Gaze Following for Human-Robot Interaction

• Akanksha Saran
• Srinjoy Majumdar
• Elaine Schaertl Short
• Andrea Thomaz
• Scott Niekum

Gaze provides subtle informative cues to aid fluent interactions among people. Incorporating human gaze predictions can signify how engaged a person is while interacting with a robot and allow the robot to predict a human's intentions or goals. We propose a novel approach to predict human gaze fixations relevant for human-robot interaction tasks-both referential and mutual gaze-in real time on a robot. We use a deep learning approach which tracks a human's gaze from a robot's perspective in real time. The approach builds on prior work which uses a deep network to predict the referential gaze of a person from a single 2D image. Our work uses an interpretable part of the network, a gaze heat map, and incorporates contextual task knowledge such as location of relevant objects, to predict referential gaze. We find that the gaze heat map statistics also capture differences between mutual and referential gaze conditions, which we use to predict whether a person is facing the robot's camera or not. We highlight the challenges of following a person's gaze on a robot in real time and show improved performance for referential gaze and mutual gaze prediction.

ICRA Conference 2018 Conference Paper

Human-Driven Feature Selection for a Robotic Agent Learning Classification Tasks from Demonstration

• Kalesha Bullard
• Sonia Chernova
• Andrea Thomaz

The state features available to a robot define the variables on which the learning computation depends. However, little prior work considers feature selection in the context of deploying a general-purpose robot able to learn new tasks. In this work, we explore human-driven feature selection in which a robotic agent can identify useful features with the aid of a human user, by extracting information from users about which features are most informative for discriminating between classes of objects needed for a given task (e. g. sorting groceries). The research questions examine (a) whether a domain expert is able to identify a subset of informative task features, (b) whether human selected features will enable the agent to classify unseen examples as accurately as using computational feature selection, and (c) if the interaction strategy used to elicit the information from the user impacts the quality of resulting feature selection. Toward that end, we conducted a user study with 30 participants on campus, given a multi-class classification task and one of five different approaches for conveying information about informative features to a robot learner. Our findings show that when features are semantically interpretable, human feature selection is effective in LfD scenarios because it is able to outperform computational methods when there is limited training data, yet still remains on-par with computational methods as the training sample size increases.

ICRA Conference 2018 Conference Paper

Incremental Task Modification via Corrective Demonstrations

• Reymundo A. Gutierrez
• Vivian Chu
• Andrea Thomaz
• Scott Niekum

In realistic environments, fully specifying a task model such that a robot can perform a task in all situations is impractical. In this work, we present Incremental Task Modification via Corrective Demonstrations (ITMCD), a novel algorithm that allows a robot to update a learned model by making use of corrective demonstrations from an end-user in its environment. We propose three different types of model updates that make structural changes to a finite state automaton (FSA) representation of the task by first converting the FSA into a state transition auto-regressive hidden Markov model (STARHMM). The STARHMM's probabilistic properties are then used to perform approximate Bayesian model selection to choose the best model update, if any. We evaluate ITMCD Model Selection in a simulated block sorting domain and the full algorithm on a real-world pouring task. The simulation results show our approach can choose new task models that sufficiently incorporate new demonstrations while remaining as simple as possible. The results from the pouring task show that ITMCD performs well when the modeled segments of the corrective demonstrations closely comply with the original task model.

IROS Conference 2018 Conference Paper

Policy Shaping with Supervisory Attention Driven Exploration

• Taylor Kessler Faulkner
• Elaine Schaertl Short
• Andrea Thomaz

Robots deployed for long periods of time need to be able to explore and learn from their environment. One approach to this problem has been reinforcement learning (RL), in which robots receive rewards from the environment that allow them to choose optimal actions. To speed learning when human supervision is available, interactive reinforcement learning solicits feedback from a human teacher. However, this approach typically assumes that learning takes place under continuous supervision, which is unlikely to hold in long-term scenarios. We propose an extension to a method of interactive reinforcement learning, policy shaping, that takes into account human attention. Our approach enables better performance while unattended by favoring information-gathering actions when attended and actions that have received positive feedback when unattended. We test our approach in both simulation and on a robot, finding that our method learns faster than policy shaping and performs more safely than policy shaping while no one is paying attention to the robot.

IROS Conference 2018 Conference Paper

Towards Intelligent Arbitration of Diverse Active Learning Queries

• Kalesha Bullard
• Andrea Thomaz
• Sonia Chernova

Active learning literature has explored the selection of optimal queries by a learning agent with respect to given criteria, but prior work in classification has focused only on obtaining labels for queried samples. In contrast, proficient learners, like humans, integrate multiple forms of information during learning. This work seeks to enable an active learner to reason about multiple query types concurrently, aimed at soliciting both instance and feature information from the teacher, and to autonomously arbitrate between queries of different types. We contribute the design of rule-based and decision-theoretic arbitration strategies and evaluate all against baselines of more traditional passive and active learning. Our findings show that all arbitration strategies lead to more efficient learning, compared to the baselines. Moreover, given a dynamically changing environment and constrained questioning budget (typical in human settings), the decision-theoretic strategy statistically outperforms all other methods since it reasons about both what query to make and when to make a query, in order to most effectively utilize its questioning budget.

IS Journal 2017 Journal Article

Interactive Task Learning

• John E. Laird
• Kevin Gluck
• John Anderson
• Kenneth D. Forbus
• Odest Chadwicke Jenkins
• Christian Lebiere
• Dario Salvucci
• Matthias Scheutz

This article presents a new research area called interactive task learning (ITL), in which an agent actively tries to learn not just how to perform a task better but the actual definition of a task through natural interaction with a human instructor while attempting to perform the task. The authors provide an analysis of desiderata for ITL systems, a review of related work, and a discussion of possible application areas for ITL systems.

AAMAS Conference 2016 Conference Paper

Directing Policy Search with Interactively Taught Via-Points

• Yannick Schroecker
• Heni Ben Amor
• Andrea Thomaz

Policy search has been successfully applied to robot motor learning problems. However, for moderately complex tasks the necessity of good heuristics or initialization still arises. One method that has been used to alleviate this problem is to utilize demonstrations obtained by a human teacher as a starting point for policy search in the space of trajectories. In this paper we describe an alternative way of giving demonstrations as soft via-points and show how they can be used for initialization as well as for active corrections during the learning process. With this approach, we restrict the search space to trajectories that will be close to the taught via-points at the taught time and thereby significantly reduce the number of samples necessary to learn a good policy. We show with a simulated robot arm that our method can efficiently learn to insert an object in a hole with just a minimal demonstration and evaluate our method further on a synthetic letter reproduction task.

ICRA Conference 2016 Conference Paper

Hierarchical rejection sampling for informed kinodynamic planning in high-dimensional spaces

• Tobias Kunz
• Andrea Thomaz
• Henrik I. Christensen

We present hierarchical rejection sampling (HRS) to improve the efficiency of asymptotically optimal sampling-based planners for high-dimensional problems with differential constraints. Pruning nodes and rejecting samples that cannot improve the currently best solution have been shown to improve performance for certain problems. We show that in high-dimensional domains this improvement can be so large that rejecting samples becomes the bottleneck of the algorithm because almost all samples are rejected. This contradicts general wisdom that collision checking is always the bottleneck of sampling-based planners. Only samples in the informed subset of the state space can potentially improve the current solution. For systems without differential constraints the informed subset forms an ellipsoid, which can be parameterized and sampled directly. For systems with differential constraints the informed subset is more complicated and no such direct sampling methods exist. HRS improves the efficiency of finding samples within the informed subset without parameterizing it explicitly. Thus, it can also be applied to systems with differential constraints for which a steering method is available. In our experiments we demonstrate efficiency improvements of an RRT* planner of up to two orders of magnitude.

IROS Conference 2016 Conference Paper

Humanoid manipulation planning using backward-forward search

• Michael X. Grey
• Caelan Reed Garrett
• C. Karen Liu
• Aaron D. Ames
• Andrea Thomaz

This paper explores combining task and manipulation planning for humanoid robots. Existing methods tend to either take prohibitively long to compute for humanoids or artificially limit the physical capabilities of the humanoid platform by restricting the robot's actions to predetermined trajectories. We present a hybrid planning system which is able to scale well for complex tasks without relying on predetermined robot actions. Our system utilizes the hybrid backward-forward planning algorithm for high-level task planning combined with humanoid primitives for standing and walking motion planning. These primitives are designed to be efficiently computable during planning, despite the large amount of complexity present in humanoid robots, while still informing the task planner of the geometric constraints present in the problem. Our experiments apply our method to simulated pick-and-place problems with additional gate constraints impacting navigation using the DRC-HUBO1 robot. Our system is able to solve puzzle-like problems on a humanoid within a matter of minutes.

AAMAS Conference 2016 Conference Paper

Object-Focused Advice in Reinforcement Learning (Extended Abstract)

• Samantha Krening
• Brent Harrison
• Karen M. Feigh
• Charles Isbell
• Andrea Thomaz

In order for robots and intelligent agents to interact with and learn from people with no machine-learning expertise, robots should be able to learn from natural human instruction. Many human explanations consist of simple sentences without state information, yet most machine learning techniques that incorporate human guidance cannot use nonspecific explanations. This work aims to learn policies from a few sentences that aren’t state specific. The proposed Object-focused advice links an object to an action, and allows a person to generalize over an object’s state space. To evaluate this technique, agents were trained using Objectfocused advice collected from participants in an experiment in the Mario Bros. domain. The results show that Objectfocused advice performs better than when no advice is given, the agent can learn where to apply the advice in the state space, and the agent can recover from adversarial advice. Also, including warnings of what not do to in addition to advice of what actions to take improves performance. CCS Concepts •Human-centered computing → Text input; •Computing methodologies → Reinforcement learning;

AAMAS Conference 2016 Conference Paper

Policy Shaping in Domains with Multiple Optimal Policies (Extended Abstract)

• Himanshu Sahni
• Brent Harrison
• Kaushik Subramanian
• Thomas Cederborg
• Charles Isbell
• Andrea Thomaz

In many domains, there exist multiple ways for an agent to achieve optimal performance. Feedback may be provided along one or more of them to aid learning. In this work, we investigate whether humans have a preference towards providing feedback along one optimal policy over the other in two gridworld domains. We find that for the domain with significant risk to exploration, 60% of our participants prefer to discourage the agent’s exploration along the risky portion of the state space, while 40% state that they have no preference. We also use the interactive reinforcement learning algorithm Policy Shaping to evaluate the performance of simulated oracles with a number of feedback strategies. We find that certain domain traits, such as risk during exploration and number of optimal policies play an important role in determining the best performing feedback strategy.

ICRA Conference 2016 Conference Paper

Work those arms: Toward dynamic and stable humanoid walking that optimizes full-body motion

• Christian Hubicki
• Ayonga Hereid
• Michael X. Grey
• Andrea Thomaz
• Aaron D. Ames

Humanoid robots are designed with dozens of actuated joints to suit a variety of tasks, but walking controllers rarely make the best use of all of this freedom. We present a framework for maximizing the use of the full humanoid body for the purpose of stable dynamic locomotion, which requires no restriction to a planning template (e. g. LIPM). Using a hybrid zero dynamics (HZD) framework, this approach optimizes a set of outputs which provides requirements for the motion for all actuated links, including arms. These output equations are then rapidly solved by a whole-body inverse-kinematic (IK) solver, providing a set of joint trajectories to the robot. We apply this procedure to a simulation of the humanoid robot, DRC-HUBO, which has over 27 actuators. As a consequence, the resulting gaits swing their arms, not by a user defining swinging motions a priori or superimposing them on gaits post hoc, but as an emergent behavior from optimizing the dynamic gait. We also present preliminary dynamic walking experiments with DRC-HUBO in hardware, thereby building a case that hybrid zero dynamics as augmented by inverse kinematics (HZD+IK) is becoming a viable approach for controlling the full complexity of humanoid locomotion.

IROS Conference 2015 Conference Paper

An evaluation of GUI and kinesthetic teaching methods for constrained-keyframe skills

• Andrey Kurenkov
• Baris Akgün
• Andrea Thomaz

Keyframe-based Learning from Demonstration has been shown to be an effective method for allowing end-users to teach robots skills. We propose a method for using multiple keyframe demonstrations to learn skills as sequences of positional constraints (c-keyframes) which can be planned between for skill execution. We also introduce an interactive GUI which can be used for displaying the learned c-keyframes to the teacher, for altering aspects of the skill after it has been taught, or for specifying a skill directly without providing kinesthetic demonstrations. We compare 3 methods of teaching c-keyframe skills: kinesthetic teaching, GUI teaching, and kinesthetic teaching followed by GUI editing of the learned skill (K-GUI teaching). Based on user evaluation, the K-GUI method of teaching is found to be the most preferred, and the GUI to be the least preferred. Kinesthetic teaching is also shown to result in more robust constraints than GUI teaching, and several use cases of K-GUI teaching are discussed to show how the GUI can be used to improve the results of kinesthetic teaching.

IROS Conference 2015 Conference Paper

Real-time changes to social dynamics in human-robot turn-taking

• Justin S. Smith
• Crystal Chao
• Andrea Thomaz

In order for robots to work alongside humans in a range of domains, they will need to operate with a variety of social dynamics that each context will require. This paper builds on previous work with a parameterized turn-taking model, CADENCE, in which different parameter settings resulted in different social dynamics. In contrast to the static parameter settings of previous work, we now investigate the problem of changing these turn-taking parameter sets dynamically within a single interaction session. This ability is necessary for successful peer-to-peer collaborations, in which balance of control between leading and following must be maintained. We present our dynamic switching approach and an experiment with 15 participants. Our results confirm that it is possible to achieve the same changes in social dynamics within a single interaction session that were previously seen only between independent sessions of different parameter settings. Moreover, we show that such a change in social dynamics is contingent upon changing parameters at socially appropriate turn boundaries.

RLDM Conference 2015 Conference Abstract

Robots Learning from Human Teachers

• Andrea Thomaz

In this talk I present recent work from the Socially Intelligent Machines Lab at Georgia Tech. The vision of our research is to enable robots to function in real human environments; such as, service robots helping at home, co-worker robots to revolutionize manufacturing, and assistive robots empowering healthcare workers and enabling aging adults to live longer in their homes. To do this, we need to build intelligent robots that can be embedded into human environments to interact with everyday people. Many of the successes of robotics to date rely on structured environments and repeatable tasks, but what all of these visions have in common is deploying robots into dynamic human environments where pre-programmed controllers won’t be an option. These robots will need to interact with end users in order to learn what they need to do on- the-job. Our research aims to computationally model mechanisms of human social learning in order to build robots and other machines that are intuitive for people to teach. We take Machine Learning interactions and redesign interfaces and algorithms to support the collection of learning input from end users instead of ML experts. This talk covers results on building models of reciprocal interactions for high-level task goal learning, low-level skill learning, and active learning interactions using humanoid robot platforms. Wednesday, June 10, 2015

IROS Conference 2015 Conference Paper

Self-improvement of learned action models with learned goal models

• Baris Akgün
• Andrea Thomaz

We introduce a new method for robots to further improve upon skills acquired through Learning from Demonstration. Previously, we have introduced a method to learn both an action model to execute the skill and a goal model to monitor the execution of the skill. In this paper we show how to use the learned goal models to improve the learned action models autonomously, without further user interaction. Trajectories are sampled from the action model and executed on the robot. The goal model then labels them as success or failure and the successful ones are used to update the action model. We introduce an adaptive sampling method to speed up convergence. We show through both simulation and real robot experiments that our method can fix a failed action model.

IROS Conference 2014 Conference Paper

Multimodal real-time contingency detection for HRI

• Vivian Chu
• Kalesha Bullard
• Andrea Thomaz

Our goal is to develop robots that naturally engage people in social exchanges. In this paper, we focus on the problem of recognizing that a person is responsive to a robot's request for interaction. Inspired by human cognition, our approach is to treat this as a contingency detection problem. We present a simple discriminative Support Vector Machine (SVM) classifier to compare against previous generative methods introduced in prior work by Lee et al. [1]. We evaluate these methods in two ways. First, by training three separate SVMs with multi-modal sensory input on a set of batch data collected in a controlled setting, where we obtain an average F 1 score of 0. 82. Second, in an open-ended experiment setting with seven participants, we show that our model is able to perform contingency detection in real-time and generalize to new people with a best F 1 score of 0. 72.

NeurIPS Conference 2013 Conference Paper

Policy Shaping: Integrating Human Feedback with Reinforcement Learning

• Shane Griffith
• Kaushik Subramanian
• Jonathan Scholz
• Charles Isbell
• Andrea Thomaz

A long term goal of Interactive Reinforcement Learning is to incorporate non-expert human feedback to solve complex tasks. State-of-the-art methods have approached this problem by mapping human information to reward and value signals to indicate preferences and then iterating over them to compute the necessary control policy. In this paper we argue for an alternate, more effective characterization of human feedback: Policy Shaping. We introduce Advise, a Bayesian approach that attempts to maximize the information gained from human feedback by utilizing it as direct labels on the policy. We compare Advise to state-of-the-art approaches and highlight scenarios where it outperforms them and importantly is robust to infrequent and inconsistent human feedback.

RLDM Conference 2013 Conference Abstract

Policy Shaping: Integrating Human Feedback with Reinforcement Learning

• Shane Griffith
• Kaushik Subramanian
• Jonathan Scholz
• Andrea Thomaz

A long term goal of Interactive Reinforcement Learning is to incorporate non-expert human feedback to solve complex tasks. Some state-of-the-art methods have approached this problem by mapping human information to rewards and values and iterating over them to compute better control policies. In this paper we argue for an alternate and more effective characterization of human feedback: Policy Shaping. We introduce Advise, a Bayesian approach that attempts to maximize the information gained from human feedback by utilizing it as direct policy labels. We compare Advise to state-of-the-art approaches using a series of experiments. These experiments use two classic arcade games, together with feedback from a simulated human teacher, which allows us to systematically test performance under a variety of cases of infrequent and inconsistent feedback. We show that Advise has similar performance to the state of the art, but is more robust to a noisy signal from the human and fairs well with an inaccurate estimate of its single input parameter. With these advancements this paper may help to make learning from human feedback an increasingly viable option for intelligent systems.

RLDM Conference 2013 Conference Abstract

Policy shaping: Integrating human feedback with reinforcement learning

• Shane Griffith
• Kaushik Subramanian
• Jonathan Scholz
• Charles Isbell
• Andrea Thomaz

A long term goal of Interactive Reinforcement Learning is to incorporate non-expert human feedback to solve complex tasks. Some state-of-the-art methods have approached this problem by mapping human in- formation to rewards and values and iterating over them to compute better control policies. In this paper we argue for an alternate and more effective characterization of human feedback: Policy Shaping. We introduce Advise, a Bayesian approach that attempts to maximize the information gained from human feedback by utilizing it as direct policy labels. We compare Advise to state-of-the-art approaches using a series of exper- iments. These experiments use two classic arcade games, together with feedback from a simulated human teacher, which allows us to systematically test performance under a variety of cases of infrequent and in- consistent feedback. We show that Advise has similar performance to the state of the art, but is more robust to a noisy signal from the human and fairs well with an inaccurate estimate of its single input parameter. With these advancements this paper may help to make learning from human feedback an increasingly viable option for intelligent systems. Sunday, October 27, 2013

AAMAS Conference 2012 Conference Paper

Automatic Task Decomposition and State Abstraction from Demonstration

• Luis C. Cobo
• Charles L. Isbell Jr.
• Andrea Thomaz

Both Learning from Demonstration (LfD) and Reinforcement Learning (RL) are popular approaches for building decision-making agents. LfD applies supervised learning to a set of human demonstrations to infer and imitate the human policy, while RL uses only a reward signal and exploration to find an optimal policy. For complex tasks both of these techniques may be ineffective. LfD may require many more demonstrations than it is feasible to obtain, and RL can take an inadmissible amount of time to converge. We present Automatic Decomposition and Abstraction from demonstration (ADA), an algorithm that uses mutual information measures over a set of human demonstrations to decompose a sequential decision process into several subtasks, finding state abstractions for each one of these subtasks. ADA then projects the human demonstrations into the abstracted state space to build a policy. This policy can later be improved using RL algorithms to surpass the performance of the human teacher. We find empirically that ADA can find satisficing policies for problems that are too complex to be solved with traditional LfD and RL algorithms. In particular, we show that we can use mutual information across state features to leverage human demonstrations to reduce the effects of the curse of dimensionality by finding subtasks and abstractions in sequential decision processes.

AAAI Conference 2011 Conference Paper

Learning Tasks and Skills Together From a Human Teacher

• Baris Akgun
• Kaushik Subramanian
• Jaeeun Shim
• Andrea Thomaz

ICRA Conference 2011 Conference Paper

Task-aware variations in robot motion

• Michael J. Gielniak
• C. Karen Liu
• Andrea Thomaz

Social robots can benefit from motion variance because non-repetitive gestures will be more natural and intuitive for human partners. We introduce a new approach for synthesizing variance, both with and without constraints, using a stochastic process. Based on optimal control theory and operational space control, our method can generate an infinite number of variations in real-time that resemble the kinematic and dynamic characteristics from the single input motion sequence. We also introduce a stochastic method to generate smooth but nondeterministic transitions between arbitrary motion variants. Furthermore, we quantitatively evaluate task-aware variance against random white torque noise, operational space control, style-based inverse kinematics, and retargeted human motion to prove that task-aware variance generates human-like motion. Finally, we demonstrate the ability of task-aware variance to maintain velocity and time-dependent features that exist in the input motion.

AAMAS Conference 2010 Conference Paper

Using Training Regimens to Teach Expanding Function Approximators

• Peng Zang
• Arya Irani
• Peng Zhou
• Andrea Thomaz
• Charles Isbell

In complex real-world environments, traditional (tabular)techniques for solving Reinforcement Learning (RL) do notscale. Function approximation is needed, but unfortunately, existing approaches generally have poor convergence and optimality guarantees. Additionally, for the case of humanenvironments, it is valuable to be able to leverage humaninput. In this paper we introduce Expanding Value Function Approximation (EVFA), a function approximation algorithm that returns the optimal value function given sufficient rounds. To leverage human input, we introduce a newhuman-agent interaction scheme, training regimens, whichallow humans to interact with and improve agent learning inthe setting of a machine learning game. In experiments, weshow EVFA compares favorably to standard value approximation approaches. We also show that training regimensenable humans to further improve EVFA performance. Inour user study, we find that non-experts are able to provideeffective regimens and that they found the game fun.

IROS Conference 2009 Conference Paper

Effective Robot Task Learning by focusing on Task-relevant objects

• Kyuhwa Lee
• Jinhan Lee
• Andrea Thomaz
• Aaron F. Bobick

In a Robot Learning from Demonstration framework involving environments with many objects, one of the key problems is to decide which objects are relevant to a given task. In this paper, we analyze this problem and propose a biologically-inspired computational model that enables the robot to focus on the task-relevant objects. To filter out incompatible task models, we compute a Task Relevance Value (TRV) for each object, which shows a human demonstrator's implicit indication of the relevance to the task. By combining an intentional action representation with ‘motionese’ [2], our model exhibits recognition capabilities compatible with the way that humans demonstrate. We evaluate the system on demonstrations from five different human subjects, showing its ability to correctly focus on the appropriate objects in these demonstrations.

ICRA Conference 2008 Conference Paper

Learning from human teachers with Socially Guided Exploration

• Cynthia Breazeal
• Andrea Thomaz

We present a learning mechanism, Socially Guided Exploration, in which a robot learns new tasks through a combination of self-exploration and social interaction. The system’s motivational drives (novelty, mastery), along with social scaffolding from a human partner, bias behavior to create learning opportunities for a Reinforcement Learning mechanism. The system is able to learn on its own, but can flexibly use the guidance of a human partner to improve performance. An experiment with non-expert human subjects shows a human is able to shape the learning process through suggesting actions and drawing attention to goal states. Human guidance results in a task set that is significantly more focused and efficient, while self exploration results in a broader set.

IROS Conference 2005 Conference Paper

Effects of nonverbal communication on efficiency and robustness in human-robot teamwork

• Cynthia Breazeal
• Cory D. Kidd
• Andrea Thomaz
• Guy Hoffman
• Matt Berlin

Nonverbal communication plays an important role in coordinating teammates' actions for collaborative activities. In this paper, we explore the impact of non-verbal social cues and behavior on task performance by a human-robot team. We report our results from an experiment where naive human subjects guide a robot to perform a physical task using speech and gesture. Both self-report via questionnaire and behavioral analysis of video offer evidence to support our hypothesis that implicit non-verbal communication positively impacts human-robot task performance with respect to understandability of the robot, efficiency of task performance, and robustness to errors that arise from miscommunication.

IROS Conference 2004 Conference Paper

Tutelage and socially guided robot learning

• Andrea Thomaz
• Cynthia Breazeal

We view the problem of machine learning as a collaboration between the human and the machine. Inspired by human-style tutelage, we situate the learning problem within a dialog in which social interaction structures the learning experience, providing instruction, directing attention, and controlling the complexity of the task. We present a learning mechanism, implemented on a humanoid robot, to demonstrate that a collaborative dialog framework allows a robot to efficiently learn a task from a human, generalize this ability to a new task configuration, and show commitment to the overall goal of the learned task. We also compare this approach to traditional machine learning approaches.