Jivko Sinapov

AAAI Conference 2026 Conference Paper

Towards Reinforcement Learning from Neural Feedback: Mapping fNIRS Signals to Agent Performance

• Julia Santaniello
• Matthew Russell
• Benson Jiang
• Donatello Sassaroli
• Robert Jacob
• Jivko Sinapov

Reinforcement Learning from Human Feedback (RLHF) is a methodology that aligns agent behavior with human preferences by integrating human feedback into the agent's training process. We introduce a possible framework that employs passive Brain-Computer Interfaces (BCI) to guide agent training from implicit neural signals. We present and release a novel dataset of functional near-infrared spectroscopy (fNIRS) recordings collected from 25 human participants across three domains: a Pick-and-Place Robot, Lunar Lander, and Flappy Bird. We train classifiers to predict levels of agent performance (optimal, sub-optimal, or worst-case) from windows of preprocessed fNIRS feature vectors, achieving an average F1 score of 67% for binary classification and 46% for multi-class models averaged across conditions and domains. We also train regressors to predict the degree of deviation between an agent's chosen action and a set of near-optimal policies, providing a continuous measure of performance. We evaluate cross-subject generalization and demonstrate that fine-tuning pre-trained models with a small sample of subject-specific data increases average F1 scores by 17% and 41% for binary and multi-class models, respectively. Our work demonstrates that mapping implicit fNIRS signals to agent performance is feasible and can be improved, laying the foundation for future brain-driven RLHF systems.

ICRA Conference 2025 Conference Paper

FLEX: A Framework for Learning Robot-Agnostic Force-Based Skills Involving Sustained Contact Object Manipulation

• Shijie Fang
• Wenchang Gao
• Shivam Goel
• Christopher Thierauf
• Matthias Scheutz
• Jivko Sinapov

Learning to manipulate objects efficiently, particularly those involving sustained contact (e. g. , pushing, sliding) and articulated parts (e. g. , drawers, doors), presents significant challenges. Traditional methods, such as robot-centric reinforce-ment learning (RL), imitation learning, and hybrid techniques, require massive training and often struggle to generalize across different objects and robot platforms. We propose a novel framework for learning object-centric manipulation policies in force space, decoupling the robot from the object. By directly applying forces to selected regions of the object, our method simplifies the action space, reduces unnecessary exploration, and decreases simulation overhead. This approach, trained in simulation on a small set of representative objects, captures ob-ject dynamics—such as joint configurations—allowing policies to generalize effectively to new, unseen objects. Decoupling these policies from robot-specific dynamics enables direct transfer to different robotic platforms (e. g. , Kinova, Panda, URS) with-out retraining. Our evaluations demonstrate that the method significantly outperforms baselines, achieving over an order of magnitude improvement in training efficiency compared to other state-of-the-art methods. Additionally, operating in force space enhances policy transferability across diverse robot plat-forms and object types. We further showcase the applicability of our method in a real-world robotic setting. Link: https://tufts-ai-robotics-group.github.io/FLEX/

ICRA Conference 2025 Conference Paper

Tool-Mediated Robot Perception of Granular Substances Using Multiple Sensory Modalities

• Si Liu
• Jivko Sinapov

People use tools to interact with and perceive the world, with multimodal sensory inputs forming the basis of how we understand our environment. For example, a blind person uses a walking cane to tap the road and detect obstacles, and a builder uses a hammer to strike a wall to assess its structural integrity. Using tools extends our sensory capabilities during exploratory behaviors, enabling us to perceive object properties that are otherwise inaccessible. Inspired by this cognitive process, we propose a framework in which a multisensory robot employs exploratory behaviors using various tools to recognize granular substances. Our framework effectively integrates multiple non-visual sensory inputs (e. g. , audio, haptic, and tactile) gathered through multiple tools (e. g. , spoon, fork) and behaviors (e. g. , stirring, poking) to perceive object properties. The framework segments interactions into time windows and aligns different modalities, enhancing data efficiency and interactive perception. Additionally, we conducted tool-transfer experiments to evaluate similarities between tools. Our experiments demonstrate that combining multiple tools and behaviors outperforms single-tool and singlebehavior approaches. While the audio modality dominates the non-visual multimodal system, other modalities contribute. We further demonstrate that tool similarities vary depending on the behavior, and notably, the robot does not need to complete entire interactions to achieve optimal recognition accuracy.

AIJ Journal 2024 Journal Article

A neurosymbolic cognitive architecture framework for handling novelties in open worlds

• Shivam Goel
• Panagiotis Lymperopoulos
• Ravenna Thielstrom
• Evan Krause
• Patrick Feeney
• Pierrick Lorang
• Sarah Schneider
• Yichen Wei

“Open world” environments are those in which novel objects, agents, events, and more can appear and contradict previous understandings of the environment. This runs counter to the “closed world” assumption used in most AI research, where the environment is assumed to be fully understood and unchanging. The types of environments AI agents can be deployed in are limited by the inability to handle the novelties that occur in open world environments. This paper presents a novel cognitive architecture framework to handle open-world novelties. This framework combines symbolic planning, counterfactual reasoning, reinforcement learning, and deep computer vision to detect and accommodate novelties. We introduce general algorithms for exploring open worlds using inference and machine learning methodologies to facilitate novelty accommodation. The ability to detect and accommodate novelties allows agents built on this framework to successfully complete tasks despite a variety of novel changes to the world. Both the framework components and the entire system are evaluated in Minecraft-like simulated environments. Our results indicate that agents are able to efficiently complete tasks while accommodating “concealed novelties” not shared with the architecture development team.

AAMAS Conference 2024 Conference Paper

LgTS: Dynamic Task Sampling using LLM-generated Sub-Goals for Reinforcement Learning Agents

• Yash Shukla
• Wenchang Gao
• Vasanth Sarathy
• Alvaro Velasquez
• Robert Wright
• Jivko Sinapov

Recent advancements in reasoning abilities of Large Language Models (LLM) has promoted their usage in problems that require highlevel planning for artificial agents. However, current techniques that utilize LLMs for such planning tasks make certain key assumptions such as, access to datasets that permit finetuning, meticulously engineered prompts that only provide relevant and essential information to the LLM, and most importantly, a deterministic approach to allow execution of the LLM responses either in the form of existing policies or plan operators. In this work, we propose LgTS (LLM-guided Teacher-Student learning), a novel approach that explores the planning abilities of LLMs to provide a graphical representation of the sub-goals to a reinforcement learning (RL) agent that does not have access to the transition dynamics of the environment. The RL agent uses Teacher-Student learning algorithm to learn a set of successful policies for reaching the goal state from the start state while simultaneously minimizing the number of environmental interactions. Unlike previous methods that utilize LLMs, our approach does not assume access to a fine-tuned LLM, nor does it require pre-trained policies that achieve the sub-goals proposed by the LLM. Through experiments on a gridworld based DoorKey domain and a search-and-rescue inspired domain, we show that a LLM-proposed graphical structure for sub-goals combined with a Teacher-Student RL algorithm achieves sample-efficient policies. More details at https: //llm-guided-task-sampling. github. io/

ICAPS Conference 2024 Conference Paper

Logical Specifications-guided Dynamic Task Sampling for Reinforcement Learning Agents

• Yash Shukla
• Tanushree Burman
• Abhishek Kulkarni
• Robert Wright
• Alvaro Velasquez
• Jivko Sinapov

Reinforcement Learning (RL) has made significant strides in enabling artificial agents to learn diverse behaviors. However, learning an effective policy often requires a large number of environment interactions. To mitigate sample complexity issues, recent approaches have used high-level task specifications, such as Linear Temporal Logic (LTLf) formulas or Reward Machines (RM), to guide the learning progress of the agent. In this work, we propose a novel approach, called Logical Specifications-guided Dynamic Task Sampling (LSTS), that learns a set of RL policies to guide an agent from an initial state to a goal state based on a high-level task specification, while minimizing the number of environmental interactions. Unlike previous work, LSTS does not assume information about the environment dynamics or the Reward Machine, and dynamically samples promising tasks that lead to successful goal policies. We evaluate LSTS on a gridworld and show that it achieves improved time-to-threshold performance on complex sequential decision-making problems compared to state-of-the-art RM and Automaton-guided RL baselines, such as Q-Learning for Reward Machines and Compositional RL from logical Specifications (DIRL). Moreover, we demonstrate that our method outperforms RM and Automaton-guided RL baselines in terms of sample-efficiency, both in a partially observable robotic task and in a continuous control robotic manipulation task.

ICRA Conference 2024 Conference Paper

MOSAIC: Learning Unified Multi-Sensory Object Property Representations for Robot Learning via Interactive Perception

• Gyan Tatiya
• Jonathan Francis
• Ho-Hsiang Wu
• Yonatan Bisk
• Jivko Sinapov

A holistic understanding of object properties across diverse sensory modalities (e. g. , visual, audio, and haptic) is essential for tasks ranging from object categorization to complex manipulation. Drawing inspiration from cognitive science studies that emphasize the significance of multi-sensory integration in human perception, we introduce MOSAIC (Multimodal Object property learning with Self-Attention and Interactive Comprehension), a novel framework designed to facilitate the learning of unified multi-sensory object property representations. While it is undeniable that visual information plays a prominent role, we acknowledge that many fundamental object properties extend beyond the visual domain to encompass attributes like texture, mass distribution, or sounds, which significantly influence how we interact with objects. In MOSAIC, we leverage this profound insight by distilling knowledge from multimodal foundation models and aligning these representations not only across vision but also haptic and auditory sensory modalities. Through extensive experiments on a dataset where a humanoid robot interacts with 100 objects across 10 exploratory behaviors, we demonstrate the versatility of MOSAIC in two task families: object categorization and object-fetching tasks. Our results underscore the efficacy of MOSAIC's unified representations, showing competitive performance in category recognition through a simple linear probe setup and excelling in the fetch object task under zero-shot transfer conditions. This work pioneers the application of sensory grounding in foundation models for robotics, promising a significant leap in multi-sensory perception capabilities for autonomous systems. We have released the code, datasets, and additional results: https://github.com/gtatiya/MOSAIC.

AAMAS Conference 2024 Conference Paper

NovelGym: A Flexible Ecosystem for Hybrid Planning and Learning Agents Designed for Open Worlds

• Shivam Goel
• Yichen Wei
• Panagiotis Lymperopoulos
• Klára Churá
• Matthias Scheutz
• Jivko Sinapov

As AI agents leave the lab and venture into the real world as autonomous vehicles, delivery robots, and cooking robots, it is increasingly necessary to design and comprehensively evaluate algorithms that tackle the “open-world”. To this end, we introduce NovelGym1, a flexible and adaptable ecosystem designed to simulate gridworld environments, serving as a robust platform for benchmarking reinforcement learning (RL) and hybrid planning and learning agents in open-world contexts. The modular architecture of NovelGym facilitates rapid creation and modification of task environments, including multi-agent scenarios, with multiple environment transformations, thus providing a dynamic testbed for researchers to develop open-world AI agents.

IROS Conference 2023 Conference Paper

A Framework for Few-Shot Policy Transfer Through Observation Mapping and Behavior Cloning

• Yash Shukla
• Bharat Kesari
• Shivam Goel
• Robert Wright
• Jivko Sinapov

Despite recent progress in Reinforcement Learning for robotics applications, many tasks remain prohibitively difficult to solve because of the expensive interaction cost. Transfer learning helps reduce the training time in the target domain by transferring knowledge learned in a source domain. Sim2Real transfer helps transfer knowledge from a simulated robotic domain to a physical target domain. Knowledge transfer reduces the time required to train a task in the physical world, where the cost of interactions is high. However, most existing approaches assume exact correspondence in the task structure and the physical properties of the two domains. This work proposes a framework for Few-Shot Policy Transfer between two domains through Observation Mapping and Behavior Cloning. We use Generative Adversarial Networks (GANs) along with a cycle-consistency loss to map the observations between the source and target domains and later use this learned mapping to clone the successful source task behavior policy to the target domain. We observe successful behavior policy transfer with limited target task interactions and in cases where the source and target task are semantically dissimilar.

ICAPS Conference 2023 Conference Paper

Automaton-Guided Curriculum Generation for Reinforcement Learning Agents

• Yash Shukla
• Abhishek Kulkarni
• Robert Wright
• Alvaro Velasquez
• Jivko Sinapov

Despite advances in Reinforcement Learning, many sequential decision making tasks remain prohibitively expensive and impractical to learn. Recently, approaches that automatically generate reward functions from logical task specifications have been proposed to mitigate this issue; however, they scale poorly on long-horizon tasks (i. e. , tasks where the agent needs to perform a series of correct actions to reach the goal state, considering future transitions while choosing an action). Employing a curriculum (a sequence of increasingly complex tasks) further improves the learning speed of the agent by sequencing intermediate tasks suited to the learning capacity of the agent. However, generating curricula from the logical specification still remains an unsolved problem. To this end, we propose AGCL, Automaton-guided Curriculum Learning, a novel method for automatically generating curricula for the target task in the form of Directed Acyclic Graphs (DAGs). AGCL encodes the specification in the form of a deterministic finite automaton (DFA), and then uses the DFA along with the Object-Oriented MDP (OOMDP) representation to generate a curriculum as a DAG, where the vertices correspond to tasks, and edges correspond to the direction of knowledge transfer. Experiments in gridworld and physics-based simulated robotics domains show that the curricula produced by AGCL achieve improved time-to-threshold performance on a complex sequential decision-making problem relative to state-of-the-art curriculum learning (e. g, teacher-student, self-play) and automaton-guided reinforcement learning baselines (e. g, Q-Learning for Reward Machines). Further, we demonstrate that AGCL performs well even in the presence of noise in the task

IJCAI Conference 2023 Conference Paper

Creative Problem Solving in Artificially Intelligent Agents: A Survey and Framework (Extended Abstract)

• Evana Gizzi
• Lakshmi Nair
• Sonia Chernova
• Jivko Sinapov

Creative Problem Solving (CPS) is a sub-area within artificial intelligence that focuses on methods for solving off-nominal, or anomalous problems in autonomous systems. Despite many advancements in planning and learning in AI, resolving novel problems or adapting existing knowledge to a new context, especially in cases where the environment may change in unpredictable ways, remains a challenge. To stimulate further research in CPS, we contribute a definition and a framework of CPS, which we use to categorize existing AI methods in this field. We conclude our survey with open research questions, and suggested future directions.

AAMAS Conference 2023 Conference Paper

Methods and Mechanisms for Interactive Novelty Handling in Adversarial Environments

• Tung Thai
• Mudit Verma
• Utkarsh Soni
• Sriram Gopalakrishnan
• Ming Shen
• Mayank Garg
• Ayush Kalani
• Nakul Vaidya

Learning to detect, characterize and accommodate novelties is a challenge that agents operating in open-world domains need to address to achieve satisfactory task performance. We sketch general methods for detecting and characterizing different types of novelties, and for building an appropriate adaptive model to accommodate them utilizing logical representations and reasoning methods in stochastic partially observable multi-agent environments. We also briefly report results from evaluations of our algorithms in the game domain of Monopoly. The results show high novelty detection and accommodation rates.

ICRA Conference 2023 Conference Paper

Transferring Implicit Knowledge of Non-Visual Object Properties Across Heterogeneous Robot Morphologies

• Gyan Tatiya
• Jonathan Francis
• Jivko Sinapov

Humans leverage multiple sensor modalities when interacting with objects and discovering their intrinsic properties. Using the visual modality alone is insufficient for deriving intuition behind object properties (e. g. , which of two boxes is heavier), making it essential to consider non-visual modalities as well, such as the tactile and auditory. Whereas robots may leverage various modalities to obtain object property understanding via learned exploratory interactions with objects (e. g. , grasping, lifting, and shaking behaviors), challenges remain: the implicit knowledge acquired by one robot via object exploration cannot be directly leveraged by another robot with different morphology, because the sensor models, observed data distributions, and interaction capabilities are different across these different robot configurations. To avoid the costly process of learning interactive object perception tasks from scratch, we propose a multi-stage projection framework for each new robot for transferring implicit knowledge of object properties across heterogeneous robot morphologies. We evaluate our approach on the object-property recognition and object-identity recognition tasks, using a dataset containing two heterogeneous robots that perform 7, 600 object interactions. Results indicate that knowledge can be transferred across robots, such that a newly-deployed robot can bootstrap its recognition models without exhaustively exploring all objects. We also propose a data augmentation technique and show that this technique improves the generalization of models. We release code, datasets, and additional results, here: https://github.com/gtatiya/Implicit-Knowledge-Transfer.

AAMAS Conference 2022 Conference Paper

ACuTE: Automatic Curriculum Transfer from Simple to Complex Environments

• Yash Shukla
• Christopher Thierauf
• Ramtin Hosseini
• Gyan Tatiya
• Jivko Sinapov

Despite recent advances in Reinforcement Learning (RL), many problems, especially real-world tasks, remain prohibitively expensive to learn. To address this issue, several lines of research have explored how tasks, or data samples themselves, can be sequenced into a curriculum to learn a problem that may otherwise be too difficult to learn from scratch. However, generating and optimizing a curriculum in a realistic scenario still requires extensive interactions with the environment. To address this challenge, we formulate the curriculum transfer problem, in which the schema of a curriculum optimized in a simpler, easy-to-solve environment (e. g. , a grid world) is transferred to a complex, realistic scenario (e. g. , a physicsbased robotics simulation or the real world). We present “ACuTE”, Automatic Curriculum Transfer from Simple to Complex Environments, a novel framework to solve this problem, and evaluate our proposed method by comparing it to other baseline approaches (e. g. , domain adaptation) designed to speed up learning. We observe that our approach produces improved jumpstart and time-to-threshold performance even when adding task elements that further increase the difficulty of the realistic scenario. Finally, we demonstrate that our approach is independent of the learning algorithm used for curriculum generation, and is Sim2Real transferable to a real world scenario using a physical robot.

JAIR Journal 2022 Journal Article

Creative Problem Solving in Artificially Intelligent Agents: A Survey and Framework

• Evana Gizzi
• Lakshmi Nair
• Sonia Chernova
• Jivko Sinapov

Creative Problem Solving (CPS) is a sub-area within Artificial Intelligence (AI) that focuses on methods for solving off-nominal, or anomalous problems in autonomous systems. Despite many advancements in planning and learning, resolving novel problems or adapting existing knowledge to a new context, especially in cases where the environment may change in unpredictable ways post deployment, remains a limiting factor in the safe and useful integration of intelligent systems. The emergence of increasingly autonomous systems dictates the necessity for AI agents to deal with environmental uncertainty through creativity. To stimulate further research in CPS, we present a definition and a framework of CPS, which we adopt to categorize existing AI methods in this field. Our framework consists of four main components of a CPS problem, namely, 1) problem formulation, 2) knowledge representation, 3) method of knowledge manipulation, and 4) method of evaluation. We conclude our survey with open research questions, and suggested directions for the future.

PRL Workshop 2022 Workshop Paper

Speeding-up Continual Learning through Information Gaines in Novel Experiences

• Pierrick Lorang
• Shivam Goel
• Patrik Zips
• Jivko Sinapov
• Matthias Scheutz

Adapting to novelties in open-world environments is an important and difficult challenge, and it has been recently shown that hybrid planning and reinforcement learning approaches can lead to better adaptations. However, these approaches still face intriguing difficulties induced by their heavy dependence on training samples to overcome changes in the environment quickly. In this work, we propose an integrated planning and learning approach that utilizes learning from failures and transferring knowledge over time to overcome novelty scenarios. Our proposed approach is much more sample efficient in adapting to sudden and unknown changes (i. e. , novelties) than the existing hybrid approaches. We showcase our results on a Minecraft-inspired gridworld environment called NovelGridworlds by injecting three novelties in the agent’s environment at test time. We show that our approach can speed up continual learning through information gained in each novel experience and, thus, more sample-efficient. Motivation: Fast Life-Long Learning* Figure 1: The impact of new task on performance is significantly reduced in the agents proposed in this paper compared to typical life-long learning agents and there are no negative effects when previously learned tasks need to be performed.

ICRA Conference 2021 Conference Paper

A Framework for Multisensory Foresight for Embodied Agents

• Xiaohui Chen
• Ramtin Hosseini
• Karen Panetta
• Jivko Sinapov

Predicting future sensory states is crucial for learning agents such as robots, drones, and autonomous vehicles. In this paper, we couple multiple sensory modalities with exploratory actions and propose a predictive neural network architecture to address this problem. Most existing approaches rely on large, manually annotated datasets, or only use visual data as a single modality. In contrast, the unsupervised method presented here uses multi-modal perceptions for predicting future visual frames. As a result, the proposed model is more comprehensive and can better capture the spatio-temporal dynamics of the environment, leading to more accurate visual frame prediction. The other novelty of our framework is the use of sub-networks dedicated to anticipating future haptic, audio, and tactile signals. The framework was tested and validated with a dataset containing 4 sensory modalities (vision, haptic, audio, and tactile) on a humanoid robot performing 9 behaviors multiple times on a large set of objects. While the visual information is the dominant modality, utilizing the additional non-visual modalities improves the accuracy of predictions.

AAMAS Conference 2021 Conference Paper

A Novelty-Centric Agent Architecture for Changing Worlds

• Faizan Muhammad
• Vasanth Sarathy
• Gyan Tatiya
• Shivam Goel
• Saurav Gyawali
• Mateo Guaman
• Jivko Sinapov
• Matthias Scheutz

Open-world AI requires artificial agents to cope with novelties that arise during task performance, i. e. , they must (1) detect novelties, (2) characterize them, in order to (3) accommodate them, especially in cases where sudden changes to the environment make task accomplishment impossible without utilizing the novelty. We present a formal framework and implementation thereof in a cognitive agent for novelty handling and demonstrate the efficacy of the proposed methods for detecting and handling a large set of novelties in a crafting task in a simulated environment. We discuss the success of the proposed knowledge-based methods and propose heuristic extensions that will further improve novelty handling in open-worlds tasks.

AAMAS Conference 2021 Conference Paper

SPOTTER: Extending Symbolic Planning Operators through Targeted Reinforcement Learning

• Vasanth Sarathy
• Daniel Kasenberg
• Shivam Goel
• Jivko Sinapov
• Matthias Scheutz

Symbolic planning models allow decision-making agents to sequence actions in arbitrary ways to achieve a variety of goals in dynamic domains. However, they are typically handcrafted and tend to require precise formulations that are not robust to human error. Reinforcement learning (RL) approaches do not require such models, and instead learn domain dynamics by exploring the environment and collecting rewards. However, RL approaches tend to require millions of episodes of experience and often learn policies that are not easily transferable to other tasks. In this paper, we address one aspect of the open problem of integrating these approaches: how can decision-making agents resolve discrepancies in their symbolic planning models while attempting to accomplish goals? We propose an integrated framework named SPOTTER that uses RL to augment and support (“spot”) a planning agent by discovering new operators needed by the agent to accomplish goals that are initially unreachable for the agent. SPOTTER outperforms pure-RL approaches while also discovering transferable symbolic knowledge and does not require supervision, successful plan traces or any a priori knowledge about the missing planning operator.

JMLR Journal 2020 Journal Article

Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey

• Sanmit Narvekar
• Bei Peng
• Matteo Leonetti
• Jivko Sinapov
• Matthew E. Taylor
• Peter Stone

Reinforcement learning (RL) is a popular paradigm for addressing sequential decision tasks in which the agent has only limited environmental feedback. Despite many advances over the past three decades, learning in many domains still requires a large amount of interaction with the environment, which can be prohibitively expensive in realistic scenarios. To address this problem, transfer learning has been applied to reinforcement learning such that experience gained in one task can be leveraged when starting to learn the next, harder task. More recently, several lines of research have explored how tasks, or data samples themselves, can be sequenced into a curriculum for the purpose of learning a problem that may otherwise be too difficult to learn from scratch. In this article, we present a framework for curriculum learning (CL) in reinforcement learning, and use it to survey and classify existing CL methods in terms of their assumptions, capabilities, and goals. Finally, we use our framework to find open problems and suggest directions for future RL curriculum learning research. [abs] [ pdf ][ bib ] &copy JMLR 2020. ( edit, beta )

IROS Conference 2020 Conference Paper

Haptic Knowledge Transfer Between Heterogeneous Robots using Kernel Manifold Alignment

• Gyan Tatiya
• Yash Shukla
• Michael Edegware
• Jivko Sinapov

Humans learn about object properties using multiple modes of perception. Recent advances show that robots can use non-visual sensory modalities (i. e. , haptic and tactile sensory data) coupled with exploratory behaviors (i. e. , grasping, lifting, pushing, dropping, etc.) for learning objects' properties such as shape, weight, material and affordances. However, non-visual sensory representations cannot be easily transferred from one robot to another, as different robots have different bodies and sensors. Therefore, each robot needs to learn its task-specific sensory models from scratch. To address this challenge, we propose a framework for knowledge transfer using kernel manifold alignment (KEMA) that enables source robots to transfer haptic knowledge about objects to a target robot. The idea behind our approach is to learn a common latent space from multiple robots' feature spaces produced by respective sensory data while interacting with objects. To test the method, we used a dataset in which 3 simulated robots interacted with 25 objects and showed that our framework speeds up haptic object recognition and allows novel object recognition.

JAIR Journal 2020 Journal Article

Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog

• Jesse Thomason
• Aishwarya Padmakumar
• Jivko Sinapov
• Nick Walker
• Yuqian Jiang
• Harel Yedidsion
• Justin Hart
• Peter Stone

In this work, we present methods for using human-robot dialog to improve language understanding for a mobile robot agent. The agent parses natural language to underlying semantic meanings and uses robotic sensors to create multi-modal models of perceptual concepts like red and heavy. The agent can be used for showing navigation routes, delivering objects to people, and relocating objects from one location to another. We use dialog clari_cation questions both to understand commands and to generate additional parsing training data. The agent employs opportunistic active learning to select questions about how words relate to objects, improving its understanding of perceptual concepts. We evaluated this agent on Amazon Mechanical Turk. After training on data induced from conversations, the agent reduced the number of dialog questions it asked while receiving higher usability ratings. Additionally, we demonstrated the agent on a robotic platform, where it learned new perceptual concepts on the y while completing a real-world task.

ICRA Conference 2019 Conference Paper

Deep Multi-Sensory Object Category Recognition Using Interactive Behavioral Exploration

• Gyan Tatiya
• Jivko Sinapov

When identifying an object and its properties, humans use features from multiple sensory modalities produced when manipulating the object. Motivated by this cognitive process, we propose a deep learning methodology for object category recognition which uses visual, auditory, and haptic sensory data coupled with exploratory behaviors (e. g. , grasping, lifting, pushing, etc.). In our method, as the robot performs an action on an object, it uses a Tensor-Train Gated Recurrent Unit network to process its visual data, and Convolutional Neural Networks to process haptic and auditory data. We propose a novel strategy to train a single neural network that inputs video, audio and haptic data, and demonstrate that its performance is better than separate neural networks for each sensory modality. The proposed method was evaluated on a dataset in which the robot explored 100 different objects, each belonging to one of 20 categories. While the visual information was the dominant modality for most categories, adding the additional haptic and auditory networks further improves the robot's category recognition accuracy. For some of the behaviors, our approach outperforms the previous published baseline for the dataset which used handcrafted features for each modality. We also show that a robot does not need the sensory data from the entire interaction, but instead can make a good prediction early on during behavior execution.

ICRA Conference 2019 Conference Paper

Improving Grounded Natural Language Understanding through Human-Robot Dialog

• Jesse Thomason
• Aishwarya Padmakumar
• Jivko Sinapov
• Nick Walker 0001
• Yuqian Jiang
• Harel Yedidsion
• Justin W. Hart
• Peter Stone 0001

Natural language understanding for robotics can require substantial domain- and platform-specific engineering. For example, for mobile robots to pick-and-place objects in an environment to satisfy human commands, we can specify the language humans use to issue such commands, and connect concept words like red can to physical object properties. One way to alleviate this engineering for a new domain is to enable robots in human environments to adapt dynamically-continually learning new language constructions and perceptual concepts. In this work, we present an end-to-end pipeline for translating natural language commands to discrete robot actions, and use clarification dialogs to jointly improve language parsing and concept grounding. We train and evaluate this agent in a virtual setting on Amazon Mechanical Turk, and we transfer the learned agent to a physical robot platform to demonstrate it in the real world.

AAAI Conference 2018 Conference Paper

Guiding Exploratory Behaviors for Multi-Modal Grounding of Linguistic Descriptions

• Jesse Thomason
• Jivko Sinapov
• Raymond Mooney
• Peter Stone

A major goal of grounded language learning research is to enable robots to connect language predicates to a robot’s physical interactive perception of the world. Coupling object exploratory behaviors such as grasping, lifting, and looking with multiple sensory modalities (e. g. , audio, haptics, and vision) enables a robot to ground non-visual words like “heavy” as well as visual words like “red”. A major limitation of existing approaches to multi-modal language grounding is that a robot has to exhaustively explore training objects with a variety of actions when learning a new such language predicate. This paper proposes a method for guiding a robot’s behavioral exploration policy when learning a novel predicate based on known grounded predicates and the novel predicate’s linguistic relationship to them. We demonstrate our approach on two datasets in which a robot explored large sets of objects and was tasked with learning to recognize whether novel words applied to those objects.

IJCAI Conference 2018 Conference Paper

Multi-modal Predicate Identification using Dynamically Learned Robot Controllers

• Saeid Amiri
• Suhua Wei
• Shiqi Zhang
• Jivko Sinapov
• Jesse Thomason
• Peter Stone

Intelligent robots frequently need to explore the objects in their working environments. Modern sensors have enabled robots to learn object properties via perception of multiple modalities. However, object exploration in the real world poses a challenging trade-off between information gains and exploration action costs. Mixed observability Markov decision process (MOMDP) is a framework for planning under uncertainty, while accounting for both fully and partially observable components of the state. Robot perception frequently has to face such mixed observability. This work enables a robot equipped with an arm to dynamically construct query-oriented MOMDPs for multi-modal predicate identification (MPI) of objects. The robot's behavioral policy is learned from two datasets collected using real robots. Our approach enables a robot to explore object properties in a way that is significantly faster while improving accuracies in comparison to existing methods that rely on hand-coded exploration strategies.

AAAI Conference 2017 Conference Paper

Automatic Curriculum Graph Generation for Reinforcement Learning Agents

• Maxwell Svetlik
• Matteo Leonetti
• Jivko Sinapov
• Rishi Shah
• Nick Walker
• Peter Stone

In recent years, research has shown that transfer learning methods can be leveraged to construct curricula that sequence a series of simpler tasks such that performance on a final target task is improved. A major limitation of existing approaches is that such curricula are handcrafted by humans that are typically domain experts. To address this limitation, we introduce a method to generate a curriculum based on task descriptors and a novel metric of transfer potential. Our method automatically generates a curriculum as a directed acyclic graph (as opposed to a linear sequence as done in existing work). Experiments in both discrete and continuous domains show that our method produces curricula that improve the agent’s learning performance when compared to the baseline condition of learning on the target task from scratch.

RLDM Conference 2017 Conference Abstract

Autonomous Task Sequencing for Customized Curriculum Design in Reinforcement Learning

• Sanmit Narvekar
• Jivko Sinapov
• Peter Stone

Transfer learning is a method where an agent reuses knowledge learned in a source task to improve learning on a target task. Recent work has shown that transfer learning can be extended to the idea of curriculum learning, where the agent incrementally accumulates knowledge over a sequence of tasks (i. e. a curriculum). In most existing work, such curricula have been constructed manually. Furthermore, they are fixed ahead of time, and do not adapt to the progress or abilities of the agent. In this paper, we formulate the design of a curriculum as a Markov Decision Process, which directly models the accumulation of knowledge as an agent interacts with tasks, and propose a method that approximates an execution of an optimal policy in this MDP to produce an agent-specific curriculum. We use our approach to automatically sequence tasks for 3 agents with varying sensing and action capabilities in an experimental domain, and show that our method produces curricula customized for each agent that improve performance relative to learning from scratch or using a different agent’s curriculum. This paper was accepted to IJCAI 2017. Upon publication, the full version will be available at: http: //www. cs. utexas. edu/users/pstone/Papers/bib2html-links/IJCAI17- Narvekar. pdf

IJCAI Conference 2017 Conference Paper

Autonomous Task Sequencing for Customized Curriculum Design in Reinforcement Learning

• Sanmit Narvekar
• Jivko Sinapov
• Peter Stone

Transfer learning is a method where an agent reuses knowledge learned in a source task to improve learning on a target task. Recent work has shown that transfer learning can be extended to the idea of curriculum learning, where the agent incrementally accumulates knowledge over a sequence of tasks (i. e. a curriculum). In most existing work, such curricula have been constructed manually. Furthermore, they are fixed ahead of time, and do not adapt to the progress or abilities of the agent. In this paper, we formulate the design of a curriculum as a Markov Decision Process, which directly models the accumulation of knowledge as an agent interacts with tasks, and propose a method that approximates an execution of an optimal policy in this MDP to produce an agent-specific curriculum. We use our approach to automatically sequence tasks for 3 agents with varying sensing and action capabilities in an experimental domain, and show that our method produces curricula customized for each agent that improve performance relative to learning from scratch or using a different agent's curriculum.

RLDM Conference 2017 Conference Abstract

Robust Extensions to Policy Gradient Methods

• Rishi Shah
• Jivko Sinapov

Reinforcement learning is a computational approach to learning from interaction. Specifically, it describes the framework for modelling an agent interacting with an environment as a Markov Decision Process (MDP). A class of techniques called policy gradient methods are used to solve MDPs by optimizing a parametrized policy via the use of stochastic gradient descent. The major goal of this paper is to use the tools of convex optimization to improve these methods. First, we draw inspiration from LASSO and add l1 regularization to our policy gradient objective in order to induce sparse parameters. The motivation for doing so comes from the fact that many domains have noisy and irrelevant features that would benefit from sparsity. Next, we introduce Mirrored PG, which applies mirror descent to policy gradient methods. Many mirror maps, such as p-norms, have been shown to handle noise particularly well, and furthermore, there exists a rich collection of feasible mirror maps. For this reason, we integrate mirror descent into the policy gradient framework.

IJCAI Conference 2016 Conference Paper

Learning Multi-Modal Grounded Linguistic Semantics by Playing "I Spy"

• Jesse Thomason
• Jivko Sinapov
• Maxwell Svetlik
• Peter Stone
• Raymond J. Mooney

Grounded language learning bridges words like red and square with robot perception. The vast majority of existing work in this space limits robot perception to vision. In this paper, we build perceptual models that use haptic, auditory, and proprioceptive data acquired through robot exploratory behaviors to go beyond vision. Our system learns to ground natural language words describing objects using supervision from an interactive human-robot I Spy game. In this game, the human and robot take turns describing one object among several, then trying to guess which object the other has described. All supervision labels were gathered from human participants physically present to play this game with a robot. We demonstrate that our multi-modal system for grounding natural language outperforms a traditional, vision-only grounding framework by comparing the two on the "I Spy" task. We also provide a qualitative analysis of the groundings learned in the game, visualizing what words are understood better with multi-modal sensory information as well as identifying learned word meanings that correlate with physical object properties (e. g. "small" negatively correlates with object weight).

IJCAI Conference 2016 Conference Paper

Learning to Order Objects Using Haptic and Proprioceptive Exploratory Behaviors

• Jivko Sinapov
• Priyanka Khante
• Maxwell Svetlik
• Peter Stone

This paper proposes a novel framework that enables a robot to learn ordinal object relations. While most related work focuses on classifying objects into discrete categories, such approaches cannot learn object properties (e. g. , weight, height, size, etc. ) that are context-specific and relative to other objects. To address this problem, we propose that a robot should learn to order objects based on ordinal object relations. In our experiments, the robot explored a set of 32 objects that can be ordered by three properties: height, weight, and width. Next, the robot used unsupervised learning to discover multiple ways that the objects can be ordered based on the haptic and proprioceptive perceptions detected while exploring the objects. Following, the robot's model was presented with labeled object series, allowing it to ground the three ordinal relations in terms of how similar they are to the orders discovered during the unsupervised stage. Finally, the grounded models were used to recognize whether new object series were ordered by any of the three properties as well as to correctly insert additional objects into an existing series.

AAMAS Conference 2016 Conference Paper

Source Task Creation for Curriculum Learning

• Sanmit Narvekar
• Jivko Sinapov
• Matteo Leonetti
• Peter Stone

Transfer learning in reinforcement learning has been an active area of research over the past decade. In transfer learning, training on a source task is leveraged to speed up or otherwise improve learning on a target task. This paper presents the more ambitious problem of curriculum learning in reinforcement learning, in which the goal is to design a sequence of source tasks for an agent to train on, such that final performance or learning speed is improved. We take the position that each stage of such a curriculum should be tailored to the current ability of the agent in order to promote learning new behaviors. Thus, as a first step towards creating a curriculum, the trainer must be able to create novel, agent-specific source tasks. We explore how such a space of useful tasks can be created using a parameterized model of the domain and observed trajectories on the target task. We experimentally show that these methods can be used to form components of a curriculum and that such a curriculum can be used successfully for transfer learning in 2 challenging multiagent reinforcement learning domains.

ICRA Conference 2014 Conference Paper

Learning relational object categories using behavioral exploration and multimodal perception

• Jivko Sinapov
• Connor Schenck
• Alexander Stoytchev

This paper proposes a framework for learning human-provided category labels that describe individual objects, pairwise object relationships, as well as groups of objects. The framework was evaluated using an experiment in which the robot interactively explored 36 objects that varied by color, weight, and contents. The proposed method allowed the robot not only to learn categories describing individual objects, but also to learn categories describing pairs and groups of objects with high recognition accuracy. Furthermore, by grounding the category representations in its own sensorimotor repertoire, the robot was able to estimate how similar two categories are in terms of the behaviors and sensory modalities that are used to recognize them. Finally, this grounded measure of similarity enabled the robot to boost its recognition performance when learning a new category by relating it to a set of familiar categories.

ICRA Conference 2013 Conference Paper

Grounded object individuation by a humanoid robot

• Jivko Sinapov
• Alexander Stoytchev

This paper proposes a theoretical model that enables a robot to partition its unlabeled sensorimotor experience with different objects into discrete clusters, each corresponding to a specific object. To solve this object individuation problem, the robot was trained to detect whether two perceptual stimuli were produced by the same object or by two different objects. The model was tested using a large-scale experiment in which a humanoid robot explored 100 different objects by performing a variety of exploratory behaviors on them and detecting the resulting sensory feedback from several sensory modalities. The results show that with a small amount of prior training, the robot's model was able to successfully individuate the objects with a high degree of accuracy.

ICRA Conference 2011 Conference Paper

Object category recognition by a humanoid robot using behavior-grounded relational learning

• Jivko Sinapov
• Alexander Stoytchev

The ability to form and recognize object categories is fundamental to human intelligence. This paper proposes a behavior-grounded relational classification model that allows a robot to recognize the categories of household objects. In the proposed approach, the robot initially explores the objects by applying five exploratory behaviors (lift, shake, drop, crush and push) on them while recording the proprioceptive and auditory sensory feedback produced by each interaction. The sensorimotor data is used to estimate multiple measures of similarity between the objects, each corresponding to a specific coupling between an exploratory behavior and a sensory modality. A graph-based recognition model is trained by extracting features from the estimated similarity relations, allowing the robot to recognize the category memberships of a novel object based on the object's similarity to the set of familiar objects. The framework was evaluated on an upper-torso humanoid robot with two large sets of household objects. The results show that the robot's model is able to recognize complex object categories (e. g. , metal objects, empty bottles, etc.) significantly better than chance.

ICRA Conference 2010 Conference Paper

How to separate containers from non-containers? a behavior-grounded approach to acoustic object categorization

• Shane Griffith
• Jivko Sinapov
• Vladimir Sukhoy
• Alexander Stoytchev

This paper describes an approach to interactive object categorization that couples exploratory behaviors and their resulting acoustic signatures to form object categories. The framework was tested with an upper-torso humanoid robot on a container/non-container categorization task. The robot used six exploratory behaviors (drop block, grasp, move, shake, flip, and drop object) and applied them to twenty objects. The results from this large-scale experimental study show that the robot was able to learn meaningful object categories using only acoustic information. The results also show that the quality of the categorization depends on the exploratory behavior used to derive it as some behaviors elicit more salient acoustic signatures than others.

AAAI Conference 2010 Conference Paper

The Boosting Effect of Exploratory Behaviors

• Jivko Sinapov
• Alexander Stoytchev

Active object exploration is one of the hallmarks of human and animal intelligence. Research in psychology has shown that the use of multiple exploratory behaviors is crucial for learning about objects. Inspired by such research, recent work in robotics has demonstrated that by performing multiple exploratory behaviors a robot can dramatically improve its object recognition rate. But what is the cause of this improvement? To answer this question, this paper examines the conditions under which combining information from multiple behaviors and sensory modalities leads to better object recognition results. Two different problems are considered: interactive object recognition using auditory and proprioceptive feedback, and surface texture recognition using tactile and proprioceptive feedback. Analysis of the results shows that metrics designed to estimate classifier model diversity can explain the improvement in recognition accuracy. This finding establishes, for the first time, an important link between empirical studies of exploratory behaviors in robotics and theoretical results on boosting in machine learning.

ICRA Conference 2009 Conference Paper

Interactive learning of the acoustic properties of household objects

• Jivko Sinapov
• Mark Wiemer
• Alexander Stoytchev

Human beings can perceive object properties such as size, weight, and material type based solely on the sounds that the objects make when an action is performed on them. In order to be successful, the household robots of the near future must also be capable of learning and reasoning about the acoustic properties of everyday objects. Such an ability would allow a robot to detect and classify various interactions with objects that occur outside of the robot's field of view. This paper presents a framework that allows a robot to infer the object and the type of behavioral interaction performed with it from the sounds generated by the object during the interaction. The framework is evaluated on a 7-d. o. f. Barrett WAM robot which performs grasping, shaking, dropping, pushing and tapping behaviors on 36 different household objects. The results show that the robot can learn models that can be used to recognize objects (and behaviors performed on objects) from the sounds generated during the interaction. In addition, the robot can use the learned models to estimate the similarity between two objects in terms of their acoustic properties.

AAAI Conference 2008 Short Paper

Toward Autonomous Learning of an Ontology of Tool Affordances by a Robot

• Jivko Sinapov