Mohan Sridharan

ICRA Conference 2025 Conference Paper

AdaptBot: Combining LLM with Knowledge Graphs and Human Input for Generic-to-Specific Task Decomposition and Knowledge Refinement

• Shivam Singh
• Karthik Swaminathan
• Nabanita Dash
• Ramandeep Singh
• Snehasis Banerjee
• Mohan Sridharan
• K. Madhava Krishna

An embodied agent assisting humans is often asked to complete new tasks, and there may not be sufficient time or labeled examples to train the agent to perform these new tasks. Large Language Models (LLMs) trained on considerable knowledge across many domains can be used to predict a sequence of abstract actions for completing such tasks, although the agent may not be able to execute this sequence due to task-, agent-, or domain-specific constraints. Our framework addresses these challenges by leveraging the generic predictions provided by LLM and the prior domain knowledge encoded in a Knowledge Graph (KG), enabling an agent to quickly adapt to new tasks. The robot also solicits and uses human input as needed to refine its existing knowledge. Based on experimental evaluation in the context of cooking and cleaning tasks in simulation domains, we demonstrate that the interplay between LLM, KG, and human input leads to substantial performance gains compared with just using the LLM. Project website 1 § Project supported in part by TCS Research India: https://sssshivvvv.github.io/adaptbot/

AAAI Conference 2025 Conference Paper

Back to the Future of Integrated Robot Systems

• Mohan Sridharan

Robots are increasingly being used in different application domains due to rapid advancements in hardware and computational methods. However, state of the art methods for many problems in robotics are based on deep networks and similar data-driven models. These methods and models are resource-hungry and opaque, and they are known to provide arbitrary decisions in previously unseen situations, whereas practical robot application domains require transparent, multi-step, multi-level decision-making and ad hoc collaboration under resource constraints and open world uncertainty. In this talk, I argue that for widespread use of robots, we need to revisit principles such as refinement and adaptive satisficing, which can be traced back to the early pioneers of AI. We also need to make these principles the foundation of the architectures we develop for robots, with modern data-driven methods being just another tool in our toolbox. I then illustrate the potential benefits of this approach in the context of fundamental problems in robotics such as visual scene understanding, planning, changing-contact manipulation, and multiagent/human-agent collaboration.

ICRA Conference 2024 Conference Paper

Anticipate & Act: Integrating LLMs and Classical Planning for Efficient Task Execution in Household Environments †

• Raghav Arora
• Shivam Singh
• Karthik Swaminathan
• Ahana Datta
• Snehasis Banerjee
• Brojeshwar Bhowmick
• Krishna Murthy Jatavallabhula
• Mohan Sridharan

Assistive agents performing household tasks such as making the bed or cooking breakfast often compute and execute actions that accomplish one task at a time. However, efficiency can be improved by anticipating upcoming tasks and computing an action sequence that jointly achieves these tasks. State-of-the-art methods for task anticipation use data-driven deep networks and Large Language Models (LLMs), but they do so at the level of high-level tasks and/or require many training examples. Our framework leverages the generic knowledge of LLMs through a small number of prompts to perform high-level task anticipation, using the anticipated tasks as goals in a classical planning system to compute a sequence of finer-granularity actions that jointly achieve these goals. We ground and evaluate our framework’s abilities in realistic scenarios in the VirtualHome environment and demonstrate a 31% reduction in execution time compared with a system that does not consider upcoming tasks.

AAAI Conference 2023 Conference Paper

Back to the Future: Toward a Hybrid Architecture for Ad Hoc Teamwork

• Hasra Dodampegama
• Mohan Sridharan

State of the art methods for ad hoc teamwork, i.e., for collaboration without prior coordination, often use a long history of prior observations to model the behavior of other agents (or agent types) and to determine the ad hoc agent's behavior. In many practical domains, it is difficult to obtain large training datasets, and necessary to quickly revise the existing models to account for changes in team composition or domain attributes. Our architecture builds on the principles of step-wise refinement and ecological rationality to enable an ad hoc agent to perform non-monotonic logical reasoning with prior commonsense domain knowledge and models learned rapidly from limited examples to predict the behavior of other agents. In the simulated multiagent collaboration domain Fort Attack, we experimentally demonstrate that our architecture enables an ad hoc agent to adapt to changes in the behavior of other agents, and provides enhanced transparency and better performance than a state of the art data-driven baseline.

NMR Workshop 2023 Conference Paper

Collaborate and Explain on the Fly: Nonmonotonic Logical Reasoning and Incremental Learning for Ad Hoc Teamwork

• Hasra Dodampegama
• Mohan Sridharan

This paper describes an architecture for ad hoc teamwork, i. e. , to enable an agent to collaborate with other agents “on the fly". State of the art frameworks for ad hoc teamwork often pursue a data-driven approach, using a large labeled dataset of prior observations to model the behavior of other agents and to determine the ad hoc agent’s behavior. These models are computationally expensive to learn, lack transparency, and make it difficult to recognize and adapt to previously unseen changes. In a departure from existing work, we introduce an architecture for ad hoc teamwork that performs non-monotonic logical reasoning with prior commonsense domain knowledge and models that are learned and revised rapidly from limited examples to predict the behavior of other agents. In addition, the architecture enables the agent to provide relational descriptions as on-demand explanations of its decisions and beliefs in response to different types of questions. We evaluate the architecture’s capabilities in two benchmark multiagent collaboration domains: Fort Attack and Half field Offense, in comparison with data-driven and knowledge-driven baselines.

ICRA Conference 2023 Conference Paper

Sequence-Agnostic Multi-Object Navigation

• Nandiraju Gireesh
• Ayush Agrawal
• Ahana Datta
• Snehasis Banerjee
• Mohan Sridharan
• Brojeshwar Bhowmick
• K. Madhava Krishna

The Multi-Object Navigation (MultiON) task requires a robot to localize an instance (each) of multiple object classes. It is a fundamental task for an assistive robot in a home or a factory. Existing methods for MultiON have viewed this as a direct extension of Object Navigation (ON), the task of localising an instance of one object class, and are pre-sequenced, i. e. , the sequence in which the object classes are to be explored is provided in advance. This is a strong limitation in practical applications characterized by dynamic changes. This paper describes a deep reinforcement learning framework for sequence-agnostic MultiON based on an actor-critic architecture and a suitable reward specification. Our framework leverages past experiences and seeks to reward progress toward individual as well as multiple target object classes. We use photo-realistic scenes from the Gibson benchmark dataset in the AI Habitat 3D simulation environment to experimentally show that our method performs better than a pre-sequenced approach and a state of the art ON method extended to MultiON.

EUMAS Conference 2022 Conference Paper

A Survey of Ad Hoc Teamwork Research

• Reuth Mirsky
• Ignacio Carlucho
• Arrasy Rahman
• Elliot Fosong
• William Macke
• Mohan Sridharan
• Peter Stone 0001
• Stefano V. Albrecht

Abstract Ad hoc teamwork is the research problem of designing agents that can collaborate with new teammates without prior coordination. This survey makes a two-fold contribution: First, it provides a structured description of the different facets of the ad hoc teamwork problem. Second, it discusses the progress that has been made in the field so far, and identifies the immediate and long-term open problems that need to be addressed in ad hoc teamwork.

NMR Workshop 2022 Conference Paper

There and Back Again: Combining Non-monotonic Logical Reasoning and Deep Learning on an Assistive Robot

• Mohan Sridharan
• Chloé Benz
• Arthur Findelair
• Kévin Gloaguen

This paper describes the development of an architecture that combines non-monotonic logical reasoning and deep learning in virtual (simulated) and real (physical) environments for an assistive robot. As an illustrative example, we consider a robot assisting in a simulated restaurant environment. For any given goal, the architecture uses Answer Set Prolog to represent and reason with incomplete commonsense domain knowledge, providing a sequence of actions for the robot to execute. At the same time, reasoning directs the robot’s learning of deep neural network models for human face and hand gestures made in the real world. These learned models are used to recognize and translate human gestures to scenarios that mimic real-world situations in the simulated environment, and to goals that need to be achieved by the robot in the simulated environment. We report the challenges faced in the development of such an integrated architecture, as well as the insights learned from the design, implementation, and evaluation of this architecture by a distributed team of researchers during the ongoing pandemic.

JAAMAS Journal 2022 Journal Article

Towards combining commonsense reasoning and knowledge acquisition to guide deep learning

• Mohan Sridharan
• Tiago Mota

Abstract Algorithms based on deep network models are being used for many pattern recognition and decision-making tasks in robotics and AI. Training these models requires a large labeled dataset and considerable computational resources, which are not readily available in many domains. Also, it is difficult to explore the internal representations and reasoning mechanisms of these models. As a step towards addressing the underlying knowledge representation, reasoning, and learning challenges, the architecture described in this paper draws inspiration from research in cognitive systems. As a motivating example, we consider an assistive robot trying to reduce clutter in any given scene by reasoning about the occlusion of objects and stability of object configurations in an image of the scene. In this context, our architecture incrementally learns and revises a grounding of the spatial relations between objects and uses this grounding to extract spatial information from input images. Non-monotonic logical reasoning with this information and incomplete commonsense domain knowledge is used to make decisions about stability and occlusion. For images that cannot be processed by such reasoning, regions relevant to the tasks at hand are automatically identified and used to train deep network models to make the desired decisions. Image regions used to train the deep networks are also used to incrementally acquire previously unknown state constraints that are merged with the existing knowledge for subsequent reasoning. Experimental evaluation performed using simulated and real-world images indicates that in comparison with baselines based just on deep networks, our architecture improves reliability of decision making and reduces the effort involved in training data-driven deep network models.

IROS Conference 2021 Conference Paper

Towards an Online Framework for Changing-Contact Robot Manipulation Tasks

• Saif Sidhik
• Mohan Sridharan
• Dirk Ruiken

We describe a framework for changing-contact robot manipulation tasks, which require the robot to make and break contacts with objects and surfaces. The discontinuous interaction dynamics of such tasks make it difficult to construct and use a single dynamics model or control strategy for such tasks. For any target motion trajectory, our framework incrementally improves its prediction of when contacts will occur. This prediction and a model relating approach velocity to impact force modify the velocity profile of the motion sequence such that it is C ∞ smooth, and help achieve a desired force on impact. We implement this framework by building on our hybrid force-motion variable impedance controller for continuous-contact tasks. We evaluate our framework in the illustrative context of a robot manipulator performing sliding tasks involving multiple contact changes with surfaces of different properties.

IJCAI Conference 2020 Conference Paper

Commonsense Reasoning to Guide Deep Learning for Scene Understanding (Extended Abstract)

• Mohan Sridharan
• Tiago Mota

Our architecture uses non-monotonic logical reasoning with incomplete commonsense domain knowledge, and incremental inductive learning, to guide the construction of deep network models from a small number of training examples. Experimental results in the context of a robot reasoning about the partial occlusion of objects and the stability of object configurations in simulated images indicate an improvement in reliability and a reduction in computational effort in comparison with an architecture based just on deep networks.

EUMAS Conference 2020 Conference Paper

Integrated Commonsense Reasoning and Deep Learning for Transparent Decision Making in Robotics

• Tiago Mota
• Mohan Sridharan
• Ales Leonardis

Abstract A robot’s ability to provide explanatory descriptions of its decisions and beliefs promotes effective collaboration with humans. Providing such transparency in decision making is particularly challenging in integrated robot systems that include knowledge-based reasoning methods and data-driven learning algorithms. Towards addressing this challenge, our architecture couples the complementary strengths of non-monotonic logical reasoning with incomplete commonsense domain knowledge, deep learning, and inductive learning. During reasoning and learning, the architecture enables a robot to provide on-demand explanations of its decisions, beliefs, and the outcomes of hypothetical actions, in the form of relational descriptions of relevant domain objects, attributes, and actions. The architecture’s capabilities are illustrated and evaluated in the context of scene understanding tasks and planning tasks performed using simulated images and images from a physical robot manipulating tabletop objects. Experimental results indicate the ability to reliably acquire and merge new information about the domain in the form of constraints, and to provide accurate explanations in the presence of noisy sensing and actuation.

EUMAS Conference 2020 Conference Paper

Integrating Deep Learning and Non-monotonic Logical Reasoning for Explainable Visual Question Answering

• Mohan Sridharan
• Heather Riley

Abstract Deep learning algorithms represent the state of the art for many problems in robotics and AI. However, they require a large labeled dataset, are computationally expensive, and the learned models are difficult to understand. Our architecture draws inspiration from research in cognitive systems to address these limitations. In the context of answering explanatory questions about scenes and an underlying classification task, our architecture uses non-monotonic logical reasoning with incomplete commonsense domain knowledge, and the features extracted from input images, to answer the input queries. Features from images not processed by such reasoning are mapped to the desired answers using a learned deep network model. In addition, previously unknown state constraints of the domain are learned incrementally and used for subsequent reasoning. Experimental results show that in comparison with an “end to end” deep architecture, our architecture significantly improves accuracy and efficiency of decision making.

ECAI Conference 2020 Conference Paper

REBA-KRL: Refinement-Based Architecture for Knowledge Representation, Explainable Reasoning and Interactive Learning in Robotics

• Mohan Sridharan

EUMAS Conference 2020 Conference Paper

Towards a Theory of Intentions for Human-Robot Collaboration

• Rocío Gómez
• Mohan Sridharan
• Heather Riley

Abstract The architecture described in this paper encodes a theory of intentions based on the principles of non-procrastination, persistence, and relevance. The architecture reasons with transition diagrams at two different resolutions, with the fine-resolution description defined as a refinement of, and hence tightly-coupled with, a coarse-resolution description. For any given goal, non-monotonic logical reasoning with the coarse-resolution description computes an activity, i. e. , a plan, comprising a sequence of abstract actions to be executed to achieve the goal. Each abstract action is implemented as a sequence of concrete actions by automatically zooming to and reasoning with the part of the fine-resolution transition diagram relevant to the coarse-resolution transition and the goal. Each concrete action is executed using probabilistic models of the uncertainty in sensing and actuation, and the corresponding coarse-resolution observations are added to the coarse-resolution history. Experimental results in the context of simulated and physical robots indicate improvements in reliability and efficiency compared with an architecture that does not include the theory of intentions, and an architecture that does not include zooming for fine-resolution reasoning.

JAIR Journal 2019 Journal Article

REBA: A Refinement-Based Architecture for Knowledge Representation and Reasoning in Robotics

• Mohan Sridharan
• Michael Gelfond
• Shiqi Zhang
• Jeremy Wyatt

This article describes REBA, a knowledge representation and reasoning architecture for robots that is based on tightly-coupled transition diagrams of the domain at two different levels of granularity. An action language is extended to support non-boolean fluents and non-deterministic causal laws, and used to describe the domain's transition diagrams, with the fine-resolution transition diagram being defined as a refinement of the coarse-resolution transition diagram. The coarse-resolution system description, and a history that includes prioritized defaults, are translated into an Answer Set Prolog (ASP) program. For any given goal, inference in the ASP program provides a plan of abstract actions. To implement each such abstract action, the robot automatically zooms to the part of the fine-resolution transition diagram relevant to this action. The zoomed fine-resolution system description, and a probabilistic representation of the uncertainty in sensing and actuation, are used to construct a partially observable Markov decision process (POMDP). The policy obtained by solving the POMDP is invoked repeatedly to implement the abstract action as a sequence of concrete actions. The fine-resolution outcomes of executing these concrete actions are used to infer coarse-resolution outcomes that are added to the coarse-resolution history and used for subsequent coarse-resolution reasoning. The architecture thus combines the complementary strengths of declarative programming and probabilistic graphical models to represent and reason with non-monotonic logic-based and probabilistic descriptions of uncertainty and incomplete domain knowledge. In addition, we describe a general methodology for the design of software components of a robot based on these knowledge representation and reasoning tools, and provide a path for proving the correctness of these components. The architecture is evaluated in simulation and on a mobile robot finding and moving target objects to desired locations in indoor domains, to show that the architecture supports reliable and efficient reasoning with violation of defaults, noisy observations and unreliable actions, in complex domains.

IJCAI Conference 2018 Conference Paper

Incrementally Grounding Expressions for Spatial Relations between Objects

• Tiago Mota
• Mohan Sridharan

Recognizing, reasoning about, and providing understandable descriptions of spatial relations between objects is an important task for robots interacting with humans. This paper describes an architecture for incrementally learning and revising the grounding of spatial relations between objects. Answer Set Prolog, a declarative language, is used to represent and reason with incomplete knowledge that includes prepositional spatial relations between scene objects. A generic grounding of prepositions for spatial relations, human input (when available), and non-monotonic logical inference, are used to infer spatial relations between 3D point clouds in given scenes, incrementally acquiring a specialized metric grounding of the prepositions and the relative confidence associated with each grounding. The architecture is evaluated on a benchmark dataset of tabletop images and on complex simulated scenes of furniture.

IS Journal 2017 Journal Article

Cognitive Computing

• Mohan Sridharan
• Gerald Tesauro
• James Hendler

The guest editors of this special issue on cognitive computing discuss the field in general and the four articles they selected to represent it in particular.

ICAPS Conference 2017 Conference Paper

What Can I Not Do? Towards an Architecture for Reasoning about and Learning Affordances

• Mohan Sridharan
• Ben Meadows
• Rocío Gómez

This paper describes an architecture for an agent to learn and reason about affordances. In this architecture, Answer Set Prolog, a declarative language, is used to represent and reason with incomplete domain knowledge that includes a representation of affordances as relations defined jointly over objects and actions. Reinforcement learning and decision-tree induction based on this relational representation and observations of action outcomes are used to interactively and cumulatively (a) acquire knowledge of affordances of specific objects being operated upon by specific agents; and (b) generalize from these specific learned instances. The capabilities of this architecture are illustrated and evaluated in two simulated domains, a variant of the classic Blocks World domain, and a robot assisting humans in an office environment.

IJCAI Conference 2013 Conference Paper

Estimating Reference Evapotranspiration for Irrigation Management in the Texas High Plains

• Daniel Holman
• Mohan Sridharan
• Prasanna Gowda
• Dana Porter
• Thomas Marek
• Terry Howell
• Jerry Moorhead

Accurate estimates of daily crop evapotranspiration (ET) are needed for efficient irrigation management in regions where crop water demand exceeds rainfall. Daily grass or alfalfa reference ET values and crop coefficients are widely used to estimate crop water demand. Inaccurate reference ET estimates can hence have a tremendous impact on irrigation costs and the demands on freshwater resources. ET networks calculate reference ET using precise measurements of meteorological data. These networks are typically characterized by gaps in spatial coverage and lack of sufficient funding, creating an immediate need for alternative sources that can fill data gaps without high costs. Although non-agricultural weather stations provide publicly accessible meteorological data, there are concerns that the data may be unsuitable for estimating reference ET due to factors such as weather station siting, data formats and quality control issues. The objective of our research is to enable the use of alternative data sources, adapting sophisticated machine learning algorithms such as Gaussian process models and neural networks to discover and model the nonlinear relationships between non-ET weather station data and the reference ET computed by ET networks. Using data from the Texas High Plains region in the U. S. , we demonstrate significant improvement in estimation accuracy in comparison with baseline regression models typically used for irrigation management applications.

AAMAS Conference 2013 Conference Paper

Learning Visual Object Models on A Robot Using Context and Appearance Cues

• Xiang Li
• Mohan Sridharan
• Catie Meador

Visual object recognition is a key challenge to the deployment of robots in domains characterized by partial observability and unforeseen changes. Sophisticated algorithms developed for modeling and recognizing objects using different visual cues [3, 4] are computationally expensive, sensitive to changes in object configurations and environmental factors, and require many training samples and accurate domain knowledge to learn object models, making it difficult for robots to reliably and efficiently model and recognize objects. These challenges are partially offset by the fact that many objects possess unique characteristics (e. g. , color and shape) and motion patterns, although these characteristics and patterns are not known in advance and may change over time. Furthermore, only a subset of domain objects are relevant to any given task and a variety of cues can be extracted from images to represent objects. This paper presents an algorithm that enables robots to identify a set of interesting objects, using appearance-based and contextual cues extracted from a small number of images to efficiently learn models of these objects. Robots learn the domain map and consider objects that move to be interesting, using motion cues to identify the corresponding image regions. Object models learned automatically from these regions consist of spatial arrangement of gradient features, graph-based models of neighborhoods of gradient features, parts-based models of image segments, color distributions, and mixture models of local context. The learned models are used for object recognition in novel scenes based on energy minimization and a generative model for information fusion. All algorithms are evaluated on wheeled robots in indoor and outdoor domains.

AAMAS Conference 2012 Conference Paper

Active Visual Sensing and Collaboration on Mobile Robots using Hierarchical POMDPs

• Shiqi Zhang
• Mohan Sridharan

A key challenge to widespread deployment of mobile robots in the real-world is the ability to robustly and autonomously sense the environment and collaborate with teammates. Real-world domains are characterized by partial observability, non-deterministic action outcomes and unforeseen changes, making autonomous sensing and collaboration a formidable challenge. This paper poses vision-based sensing, information processing and collaboration as an instance of probabilistic planning using partially observable Markov decision processes. Reliable, efficient and autonomous operation is achieved using a hierarchical decomposition that includes: (a) convolutional policies to exploit the local symmetry of high-level visual search; (b) adaptive observation functions, policy re-weighting, automatic belief propagation and online updates of the domain map for autonomous adaptation to domain changes; and (c) a probabilistic strategy for a team of robots to robustly share beliefs. All algorithms are evaluated in simulation and on physical robots localizing target objects in dynamic indoor domains.

AAAI Conference 2012 Conference Paper

Combining Probabilistic Planning and Logic Programming on Mobile Robots

• Shiqi Zhang
• Forrest Bao
• Mohan Sridharan

ICRA Conference 2011 Conference Paper

Autonomous learning of vision-based layered object models on mobile robots

• Xiang Li 0102
• Mohan Sridharan
• Shiqi Zhang 0001

Although mobile robots are increasingly being used in real-world applications, the ability to robustly sense and interact with the environment is still missing. A key requirement for the widespread deployment of mobile robots is the ability to operate autonomously by learning desired environmental models and revising the learned models in response to environmental changes. This paper presents an approach that enables a mobile robot to autonomously learn layered models for environmental objects using temporal, local and global visual cues. A temporal assessment of image gradient features is used to detect candidate objects, which are then modeled using color distribution statistics and a spatial representation of gradient features. The robot incrementally revises the learned models and uses them for object recognition and tracking based on a matching scheme comprising a spatial similarity measure and second order distribution statistics. All algorithms are implemented and tested on a wheeled robot platform in dynamic indoor environments.

ICRA Conference 2011 Conference Paper

To look or not to look: A hierarchical representation for visual planning on mobile robots

• Shiqi Zhang 0001
• Mohan Sridharan
• Xiang Li 0102

Mobile robots are increasingly being used in real-world applications due to the ready availability of high-fidelity sensors and the development of sophisticated information processing algorithms. However, one key challenge to the widespread deployment of mobile robots equipped with multiple sensors and processing algorithms is the ability to autonomously tailor sensing and information processing to the task at hand. This paper poses this challenge as the task of planning under uncertainty, and more specifically as an instance of probabilistic sequential decision-making. A novel hierarchy of partially observable Markov decision processes (POMDPs) is incorporated, which uses constrained-convolutional policies and automatic belief propagation to achieve efficient and reliable operation on mobile robots. All algorithms are implemented and evaluated on simulated and physical robot platforms for the task of searching for target objects in dynamic indoor environments.

AIJ Journal 2010 Journal Article

Planning to see: A hierarchical approach to planning visual actions on a robot using POMDPs

• Mohan Sridharan
• Jeremy Wyatt
• Richard Dearden

Flexible, general-purpose robots need to autonomously tailor their sensing and information processing to the task at hand. We pose this challenge as the task of planning under uncertainty. In our domain, the goal is to plan a sequence of visual operators to apply on regions of interest (ROIs) in images of a scene, so that a human and a robot can jointly manipulate and converse about objects on a tabletop. We pose visual processing management as an instance of probabilistic sequential decision making, and specifically as a Partially Observable Markov Decision Process (POMDP). The POMDP formulation uses models that quantitatively capture the unreliability of the operators and enable a robot to reason precisely about the trade-offs between plan reliability and plan execution time. Since planning in practical-sized POMDPs is intractable, we partially ameliorate this intractability for visual processing by defining a novel hierarchical POMDP based on the cognitive requirements of the corresponding planning task. We compare our hierarchical POMDP planning system (HiPPo) with a non-hierarchical POMDP formulation and the Continual Planning (CP) framework that handles uncertainty in a qualitative manner. We show empirically that HiPPo and CP outperform the naive application of all visual operators on all ROIs. The key result is that the POMDP methods produce more robust plans than CP or the naive visual processing. In summary, visual processing problems represent a challenging domain for planning techniques and our hierarchical POMDP-based approach for visual processing management opens up a promising new line of research.

ICRA Conference 2010 Conference Paper

Toward autonomous scientific exploration of ice-covered lakes - Field experiments with the ENDURANCE AUV in an Antarctic Dry Valley

• Shilpa Gulati
• Kristof Richmond
• Christopher Flesher
• Bart P. Hogan
• Aniket Murarka
• Gregory Kuhlmann
• Mohan Sridharan
• William C. Stone

Chemical properties of lake water can provide valuable insight into its ecology. Lakes that are permanently frozen over with ice are generally inaccessible to comprehensive exploration by humans. This paper describes the integration of several novel and existing technologies into an autonomous underwater robot, ENDURANCE, that was successfully used for gathering scientific data in West Lake Bonney in Taylor Valley, Antarctica, in December 2008. This paper focuses on three novel technological and algorithmic solutions. First, a robust position estimation system that uses an acoustic beacon to complement traditional dead-reckoning is described. Second, a novel vision-based docking algorithm for locating and ascending a vertical shaft by tracking a blinking light source is presented. Third, a novel profiling system for measuring water properties while causing minimal water disturbance is described. Finally, experimental results from the scientific missions in 2008 in West Lake Bonney are presented.

IROS Conference 2008 Conference Paper

Detecting obstacles and drop-offs using stereo and motion cues for safe local motion

• Aniket Murarka
• Mohan Sridharan
• Benjamin Kuipers

A mobile robot operating in an urban environment has to navigate around obstacles and hazards. Though a significant amount of work has been done on detecting obstacles, not much attention has been given to the detection of drop-offs, e. g. , sidewalk curbs, downward stairs, and other hazards where an error could lead to disastrous consequences. In this paper, we propose algorithms for detecting both obstacles and drop-offs (also called negative obstacles) in an urban setting using stereo vision and motion cues. We propose a global color segmentation stereo method and compare its performance at detecting hazards against prior work using a local correlation stereo method. Furthermore, we introduce a novel drop-off detection scheme based on visual motion cues that adds to the performance of the stereo-vision methods. All algorithms are implemented and evaluated on data obtained by driving a mobile robot in urban environments.

ICAPS Conference 2008 Conference Paper

HiPPo: Hierarchical POMDPs for Planning Information Processing and Sensing Actions on a Robot

• Mohan Sridharan
• Jeremy L. Wyatt
• Richard Dearden

Flexible general purpose robots need to tailor their visual processing to their task, on the fly. We propose a new approach to this within a planning framework, where the goal is to plan a sequence of visual operators to apply to the regions of interest (ROIs) in a scene. We pose the visual processing problem as a Partially Observable Markov Decision Process (POMDP). This requires probabilistic models of operator effects to quantitatively capture the unreliability of the processing actions, and thus reason precisely about trade-offs between plan execution time and plan reliability. Since planning in practical sized POMDPs is intractable we show how to ameliorate this intractability somewhat for our domain by defining a hierarchical POMDP. We compare the hierarchical POMDP approach with a Continual Planning (CP) approach. On a real robot visual domain, we show empirically that all the planning methods outperform naive application of all visual operators. The key result is that the POMDP methods produce more robust plans than either naive visual processing or the CP approach. In summary, we believe that visual processing problems represent a challenging and worthwhile domain for planning techniques, and that our hierarchical POMDP based approach to them opens up a promising new line of research.

IJCAI Conference 2007 Conference Paper

• Mohan Sridharan
• Peter Stone

A central goal of robotics and AI is to be able to deploy an agent to act autonomously in the real world over an extended period of time. It is commonly asserted that in order to do so, the agent must be able to learn to deal with unexpected environmental conditions. However an ability to learn is not sufficient. For true extended autonomy, an agent must also be able to recognize when to abandon its current model in favor of learning a new one; and how to learn in its current situation. This paper presents a fully implemented example of such autonomy in the context of color map learning on a vision-based mobile robot for the purpose of image segmentation. Past research established the ability of a robot to learn a color map in a single fixed lighting condition when manually given a 'curriculum, ' an action sequence designed to facilitate learning. This paper introduces algorithms that enable a robot to i) devise its own curriculum; and ii) recognize when the lighting conditions have changed sufficiently to warrant learning a new color map.

IROS Conference 2007 Conference Paper

Global action selection for illumination invariant color modeling

• Mohan Sridharan
• Peter Stone 0001

A major challenge in the path of widespread use of mobile robots is the ability to function autonomously, learning useful features from the environment and using them to adapt to environmental changes. We propose an algorithm for mobile robots equipped with color cameras that allows for smooth operation under illumination changes. The robot uses image statistics and the environmental structure to autonomously detect and adapt to both major and minor illumination changes. Furthermore, the robot autonomously plans an action sequence that maximizes color learning opportunities while minimizing localization errors. Our approach is fully implemented and tested on the Sony AIBO robots.

AAAI Conference 2006 Short Paper

Robust Autonomous Structure-based Color Learning on a Mobile Robot

• Mohan Sridharan

This is a submission for the Doctoral Consortium - the paper itself is a 2-page extended abstract...

AAAI Conference 2005 Conference Paper

Autonomous Color Learning on a Mobile Robot

• Mohan Sridharan

Color segmentation is a challenging subtask in computer vision. Most popular approaches are computationally expensive, involve an extensive off-line training phase and/or rely on a stationary camera. This paper presents an approach for color learning on-board a legged robot with limited computational and memory resources. A key defining feature of the approach is that it works without any labeled training data. Rather, it trains autonomously from a color-coded model of its environment. The process is fully implemented, completely autonomous, and provides high degree of segmentation accuracy.

ICRA Conference 2005 Conference Paper

Practical Vision-Based Monte Carlo Localization on a Legged Robot

• Mohan Sridharan
• Gregory Kuhlmann
• Peter Stone 0001

Mobile robot localization, the ability of a robot to determine its global position and orientation, continues to be a major research focus in robotics. In most past cases, such localization has been studied on wheeled robots with range finding sensors such as sonar or lasers. In this paper, we consider the more challenging scenario of a legged robot localizing with a limited field-of-view camera as its primary sensory input. We begin with a baseline implementation adapted from the literature that provides a reasonable level of competence, but that exhibits some weaknesses in real-world tests. We propose a series of practical enhancements designed to improve the robot’s sensory and actuator models that enable our robots to achieve a 50% improvement in localization accuracy over the baseline implementation. We go on to demonstrate how the accuracy improvement is even more dramatic when the robot is subjected to large unmodeled movements. These enhancements are each individually straightforward, but together they provide a roadmap for avoiding potential pitfalls when implementing Monte Carlo Localization on vision-based and/or legged robots.

IROS Conference 2005 Conference Paper

Real-time vision on a mobile robot platform

• Mohan Sridharan
• Peter Stone 0001

Computer vision is a broad and significant ongoing research challenge, even when performed on an individual image or on streaming video from a high-quality stationary camera with abundant computational resources. When faced with streaming video from a lower-quality, rapidly moving camera and limited computational resources, the challenge increases. We present our implementation of a vision system on a mobile robot platform that uses a camera image as the primary sensory input. Having to perform all processing, including segmentation and object detection, in real-time on-board the robot, eliminates the possibility of using some state-of-the-art methods that otherwise might apply. We describe the methods that we developed to achieve a practical vision system within these constraints. Our approach is fully implemented and tested on a team of Sony AIBO robots.