Joydeep Biswas

IROS Conference 2025 Conference Paper

ConfigBot: Adaptive Resource Allocation for Robot Applications in Dynamic Environments

• Rohit Dwivedula
• Sadanand Modak
• Aditya Akella
• Joydeep Biswas
• Daehyeok Kim
• Christopher J. Rossbach

The growing use of service robots in dynamic environments requires flexible management of on-board compute resources to optimize the performance of diverse tasks such as navigation, localization, and perception. Current robot deployments often rely on static OS configurations and system over-provisioning. However, they are suboptimal because they ignore variations in resource usage, leading to system-wide issues like robot instability or inefficient resource utilization. This paper presents ConfigBot, a novel system designed to adaptively reconfigure robot applications to meet a predefined performance specification by leveraging runtime profiling and automated configuration tuning. Through experiments on multiple real robots, each running a different stack with diverse performance requirements, which could be context-dependent, we illustrate ConfigBot's efficacy in maintaining system stability and optimizing resource allocation. Our findings highlight the promise of automatic system configuration tuning for robot deployments, including adaptation to dynamic changes. Code available at: https://github.com/ldos-project/configbot

IROS Conference 2025 Conference Paper

Multi-Agent Inverse Reinforcement Learning in Real World Unstructured Pedestrian Crowds

• Rohan Chandra
• Haresh Karnan
• Negar Mehr
• Peter Stone 0001
• Joydeep Biswas

Social robot navigation in crowded public spaces such as university campuses, restaurants, grocery stores, and hospitals, is an increasingly important area of research. One of the core strategies for achieving this goal is to understand humans’ intent–underlying psychological factors that govern their motion–by learning how humans assign rewards to their actions, typically via inverse reinforcement learning (IRL). Despite significant progress in IRL, learning reward functions of multiple agents simultaneously in dense unstructured pedestrian crowds has remained intractable due to the nature of the tightly coupled social interactions that occur in these scenarios e. g. passing, intersections, swerving, weaving, etc. In this paper, we present a new multi-agent maximum entropy inverse reinforcement learning algorithm for real world unstructured pedestrian crowds. Key to our approach is a simple, but effective, mathematical trick which we name the so-called "tractability-rationality trade-off" trick that achieves tractability at the cost of a slight reduction in accuracy. We compare our approach to the classical single-agent MaxEnt IRL as well as state-of-the-art trajectory prediction methods on several datasets including the ETH, UCY, SCAND, JRDB, and a new dataset, called Speedway, collected at a busy intersection on a University campus focusing on dense, complex agent interactions. Our key findings show that, on the dense Speedway dataset, our approach ranks 1 st among top 7 baselines with > 2× improvement over single-agent IRL, and is competitive with state-of-the-art large transformer-based encoder-decoder models on sparser datasets such as ETH/UCY (ranks 3 rd among top 7 baselines).

ICRA Conference 2025 Conference Paper

ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigation

• Abrar Anwar
• John Welsh
• Joydeep Biswas
• Soha Pouya
• Yan Chang

Navigating and understanding complex environments over extended periods of time is a significant challenge for robots. People interacting with the robot may want to ask questions like where something happened, when it occurred, or how long ago it took place, which would require the robot to reason over a long history of their deployment. To address this problem, we introduce a Retrieval-augmented Memory for Embodied Robots, or ReMEmbR, a system designed for long-horizon video question answering for robot navigation. To evaluate ReMEmbR, we introduce the NaVQA dataset where we annotate spatial, temporal, and descriptive questions to long-horizon robot navigation videos. ReMEmbR employs a structured approach involving a memory building and a querying phase, leveraging temporal information, spatial information, and images to efficiently handle continuously growing robot histories. Our experiments demonstrate that ReMEmbR outperforms LLM and VLM baselines, allowing ReMEmbR to achieve effective long-horizon reasoning with low latency. Additionally, we deploy ReMEmbR on a robot and show that our approach can handle diverse queries. The dataset, code, videos, and other material can be found at the following link: https://nvidia-ai-iot.github.io/remembr.

ICRA Conference 2025 Conference Paper

SPOT: SE(3) Pose Trajectory Diffusion for Object-Centric Manipulation

• Cheng-Chun Hsu
• Bowen Wen
• Jie Xu 0028
• Yashraj S. Narang
• Xiaolong Wang 0004
• Yuke Zhu
• Joydeep Biswas
• Stan Birchfield

We introduce SPOT, an object-centric imitation learning framework. The key idea is to capture each task by an object-centric representation, specifically the SE(3) object pose trajectory relative to the target. This approach decouples embodiment actions from sensory inputs, facilitating learning from various demonstration types, including both action-based and action-less human hand demonstrations, as well as crossembodiment generalization. Additionally, object pose trajectories inherently capture planning constraints from demonstrations without the need for manually-crafted rules. To guide the robot in executing the task, the object trajectory is used to condition a diffusion policy. We systematically evaluate our method on simulation and real-world tasks. In real-world evaluation, using only eight demonstrations shot on an iPhone, our approach completed all tasks while fully complying with task constraints. Project page: https://nvlabs.github.io/object_centric_diffusion

AAAI Conference 2025 Conference Paper

SYNAPSE: SYmbolic Neural-Aided Preference Synthesis Engine

• Sadanand Modak
• Noah Tobias Patton
• Isil Dillig
• Joydeep Biswas

This paper addresses the problem of preference learning, which aims to align robot behaviors through learning user-specific preferences (e.g. “good pull-over location”) from visual demonstrations. Despite its similarity to learning factual concepts (e.g. “red door”), preference learning is a fundamentally harder problem due to its subjective nature and the paucity of person-specific training data. We address this problem using a novel framework called SYNAPSE, which is a neuro-symbolic approach designed to efficiently learn preferential concepts from limited data. SYNAPSE represents preferences as neuro-symbolic programs – facilitating inspection of individual parts for alignment – in a domain-specific language (DSL) that operates over images and leverages a novel combination of visual parsing, large language models, and program synthesis to learn programs representing individual preferences. We perform extensive evaluations on various preferential concepts as well as user case studies demonstrating its ability to align well with dissimilar user preferences. Our method significantly outperforms baselines, especially when it comes to out-of-distribution generalization. We show the importance of the design choices in the framework through multiple ablation studies.

ICRA Conference 2025 Conference Paper

X-MOBILITY: End-to-End Generalizable Navigation via World Modeling

• Wei Liu
• Huihua Zhao
• Chenran Li
• Joydeep Biswas
• Billy Okal
• Pulkit Goyal
• Yan Chang
• Soha Pouya

General-purpose navigation in challenging environments remains a significant problem in robotics, with current state-of-the-art approaches facing myriad limitations. Classical approaches struggle with cluttered settings and require extensive tuning, while learning-based methods face difficulties generalizing to out-of-distribution environments. This paper introduces X-Mobility, an end-to-end generalizable navigation model that overcomes existing challenges by leveraging three key ideas. First, X-Mobility employs an auto-regressive world modeling architecture with a latent state space to capture world dynamics. Second, a diverse set of multi-head decoders enables the model to learn a rich state representation that correlates strongly with effective navigation skills. Third, by decoupling world modeling from action policy, our architecture can train effectively on a variety of data sources, both with and without expert policies-off-policy data allows the model to learn world dynamics, while on-policy data with supervisory control enables optimal action policy learning. Through extensive experiments, we demonstrate that X-Mobility not only generalizes effectively but also surpasses current state-of-the-art navigation approaches. Additionally, X-Mobility also achieves zero-shot Sim2Real transferability and shows strong potential for crossembodiment generalization. Project page: https://nvlabs.github.io/X-MOBILITY.

NeurIPS Conference 2024 Conference Paper

Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning

• Arko Banerjee
• Kia Rahmani
• Joydeep Biswas
• Isil Dillig

Among approaches for provably safe reinforcement learning, Model Predictive Shielding (MPS) has proven effective at complex tasks in continuous, high-dimensional state spaces, by leveraging a backup policy to ensure safety when the learned policy attempts to take risky actions. However, while MPS can ensure safety both during and after training, it often hinders task progress due to the conservative and task-oblivious nature of backup policies. This paper introduces Dynamic Model Predictive Shielding (DMPS), which optimizes reinforcement learning objectives while maintaining provable safety. DMPS employs a local planner to dynamically select safe recovery actions that maximize both short-term progress as well as long-term rewards. Crucially, the planner and the neural policy play a synergistic role in DMPS. When planning recovery actions for ensuring safety, the planner utilizes the neural policy to estimate long-term rewards, allowing it to observe beyond its short-term planning horizon. Conversely, the neural policy under training learns from the recovery plans proposed by the planner, converging to policies that are both high-performing and safe in practice. This approach guarantees safety during and after training, with bounded recovery regret that decreases exponentially with planning horizon depth. Experimental results demonstrate that DMPS converges to policies that rarely require shield interventions after training and achieve higher rewards compared to several state-of-the-art baselines.

ICRA Conference 2024 Conference Paper

Looking Inside Out: Anticipating Driver Intent From Videos

• Yung-Chi Kung
• Arthur Zhang
• Junmin Wang
• Joydeep Biswas

Anticipating driver intention is an important task when vehicles of mixed and varying levels of human/machine autonomy share roadways. Driver intention can be leveraged to improve road safety, such as warning surrounding vehicles in the event the driver is attempting a dangerous maneuver. In this work, we propose a novel method of utilizing both in-cabin and external camera data to improve state-of-the-art performance in predicting future driver actions. Compared to existing methods, our approach explicitly extracts object and road-level features from external camera data, which we demonstrate are important features for predicting driver intention. Using our handcrafted features as inputs for both a transformer and a long-short-term-memory-based architecture, we empirically show that jointly utilizing in-cabin and external features improves performance compared to using in-cabin features alone. Furthermore, our models predict driver maneuvers more accurately and sooner than existing approaches, with an accuracy of 87. 5% and an average prediction time of 4. 35 seconds before the maneuver takes place. We release our model configurations and training scripts on https://github.com/ykung83/Driver-Intent-Prediction.

ICRA Conference 2024 Conference Paper

Rethinking Social Robot Navigation: Leveraging the Best of Two Worlds

• Amir Hossain Raj
• Zichao Hu
• Haresh Karnan
• Rohan Chandra
• Amirreza Payandeh
• Luisa Mao
• Peter Stone 0001
• Joydeep Biswas

Empowering robots to navigate in a socially compliant manner is essential for the acceptance of robots moving in human-inhabited environments. Previously, roboticists have developed geometric navigation systems with decades of empirical validation to achieve safety and efficiency. However, the many complex factors of social compliance make geometric navigation systems hard to adapt to social situations, where no amount of tuning enables them to be both safe (people are too unpredictable) and efficient (the frozen robot problem). With recent advances in deep learning approaches, the common reaction has been to entirely discard these classical navigation systems and start from scratch, building a completely new learning-based social navigation planner. In this work, we find that this reaction is unnecessarily extreme: using a large-scale real-world social navigation dataset, SCAND, we find that geometric systems can produce trajectory plans that align with the human demonstrations in a large number of social situations. We, therefore, ask if we can rethink the social robot navigation problem by leveraging the advantages of both geometric and learning-based methods. We validate this hybrid paradigm through a proof-of-concept experiment, in which we develop a hybrid planner that switches between geometric and learning-based planning. Our experiments on both SCAND and two physical robots show that the hybrid planner can achieve better social compliance compared to using either the geometric or learning-based approach alone.

AAAI Conference 2024 System Paper

SOCIALGYM 2.0: Simulator for Multi-Robot Learning and Navigation in Shared Human Spaces

• Rohan Chandra
• Zayne Sprague
• Joydeep Biswas

We present Social Gym 2.0, a simulator for multi-agent navigation research. Our simulator enables navigation for multiple autonomous agents, replicating real-world dynamics in complex indoor environments, including doorways, hallways, intersections, and roundabouts. Unlike current simulators that concentrate on single robots in open spaces, Social Gym 2.0 employs multi-agent reinforcement learning (MARL) to develop optimal navigation policies for multiple robots with diverse, dynamic constraints in complex environments. Social Gym 2.0 also departs from the accepted software design standards by employing a configuration-over-convention paradigm providing the capability to benchmark different MARL algorithms, as well as customize observation and reward functions. Users can additionally create their own environments and evaluate various algorithms, based on both deep reinforcement learning as well as classical navigation, using a broad range of social navigation metrics.

ICRA Conference 2024 Conference Paper

Wait, That Feels Familiar: Learning to Extrapolate Human Preferences for Preference-Aligned Path Planning

• Haresh Karnan
• Elvin Yang
• Garrett Warnell
• Joydeep Biswas
• Peter Stone 0001

Autonomous mobility tasks such as last-mile delivery require reasoning about operator-indicated preferences over terrains on which the robot should navigate to ensure both robot safety and mission success. However, coping with out of distribution data from novel terrains or appearance changes due to lighting variations remains a fundamental problem in visual terrain-adaptive navigation. Existing solutions either require labor-intensive manual data re-collection and labeling or use hand-coded reward functions that may not align with operator preferences. In this work, we posit that operator preferences for visually novel terrains, which the robot should adhere to, can often be extrapolated from established terrain preferences within the inertial-proprioceptive-tactile domain. Leveraging this insight, we introduce Preference extrApolation for Terrain-awarE Robot Navigation (PATERN), a novel framework for extrapolating operator terrain preferences for visual navigation. PATERN learns to map inertial-proprioceptive-tactile measurements from the robot’s observations to a representation space and performs nearest-neighbor search in this space to estimate operator preferences over novel terrains. Through physical robot experiments in outdoor environments, we assess PATERN’s capability to extrapolate preferences and generalize to novel terrains and challenging lighting conditions. Compared to baseline approaches, our findings indicate that PATERN 1 robustly generalizes to diverse terrains and varied lighting conditions, while navigating in a preference-aligned manner.

AIJ Journal 2023 Journal Article

Competence-aware systems

• Connor Basich
• Justin Svegliato
• Kyle H. Wray
• Stefan Witwicki
• Joydeep Biswas
• Shlomo Zilberstein

AIJ Journal 2023 Journal Article

Introspective perception for mobile robots

• Sadegh Rabiee
• Joydeep Biswas

ICRA Conference 2023 Conference Paper

System Configuration and Navigation of a Guide Dog Robot: Toward Animal Guide Dog-Level Guiding Work

• Hochul Hwang
• Tim Xia
• Ibrahima Keita
• Ken Suzuki
• Joydeep Biswas
• Sunghoon Ivan Lee
• Donghyun Kim 0002

A robot guide dog has compelling advantages over animal guide dogs for its cost-effectiveness, the potential for mass production, and low maintenance burden. However, despite the long history of guide dog robot research, previous studies were conducted with little or no consideration of how the guide dog handler and the guide dog work as a team for navigation. To develop a robotic guiding system that genuinely benefits blind or visually impaired individuals, we performed qualitative research, including interviews with guide dog handlers, trainers, and first-hand blindfold walking experiences with various guide dogs. We build a collaborative indoor navigation scheme for a guide dog robot that includes preferred features such as speed and directional control. For collaborative navigation, we propose a semantic-aware local path planner that enables safe and efficient guiding work by utilizing semantic information about the environment and considering the handler's position and directional cues to determine the collision-free path. We evaluate our integrated robotic system by testing blindfolded walking in indoor settings and demonstrate guide dog-like navigation behavior by avoiding obstacles at typical gait speed (0. 7m/s). The following demonstration video link includes an audio description: https://youtu.be/YxlcMeaL7GA

IROS Conference 2022 Conference Paper

High-Speed Accurate Robot Control using Learned Forward Kinodynamics and Non-linear Least Squares Optimization

• Pranav Atreya
• Haresh Karnan
• Kavan Singh Sikand
• Xuesu Xiao
• Sadegh Rabiee
• Joydeep Biswas

Accurate control of robots at high speeds requires a control system that can take into account the kinodynamic interactions of the robot with the environment. Prior works on learning inverse kinodynamic (IKD) models of robots have shown success in capturing the complex kinodynamic effects. However, the types of control problems these approaches can be applied to are limited only to that of following pre-computed kinodynamically feasible trajectories. In this paper we present Optim-FKD, a new formulation for accurate, high-speed robot control that makes use of a learned forward kinodynamic (FKD) model and non-linear least squares optimization. Optim-FKD can be used for accurate, high speed control on any control task specifiable by a non-linear least squares objective. Optim-FKD can solve for control objectives such as path following and time-optimal control in real time, without needing access to pre-computed kinodynamically feasible trajectories. We empirically demonstrate these abilities of our approach through experiments on a scale one-tenth autonomous car. Our results show that Optim-FKD can follow desired trajectories more accurately and can find better solutions to optimal control problems than baseline approaches.

IROS Conference 2022 Conference Paper

Probabilistic Object Maps for Long-Term Robot Localization

• Amanda Adkins
• Taijing Chen
• Joydeep Biswas

Robots deployed in settings such as warehouses and parking lots must cope with frequent and substantial changes when localizing in their environments. While many previous localization and mapping algorithms have explored methods of identifying and focusing on long-term features to handle change in such environments, we propose a different approach - can a robot understand the distribution of movable objects and relate it to observations of such objects to reason about global localization? In this paper, we present probabilistic object maps (POMs), which represent the distributions of movable objects using pose-likelihood sample pairs derived from prior trajectories through the environment and use a Gaussian process classifier to generate the likelihood of an object at a query pose. We also introduce POM-Localization, which uses an observation model based on POMs to perform inference on a factor graph for globally consistent long-term localization. We present empirical results showing that POM-Localization is indeed effective at producing globally consistent localization estimates in challenging real-world environments and that POM-Localization improves trajectory estimates even when the POM is formed from partially incorrect data.

IROS Conference 2022 Conference Paper

SocialGym: A Framework for Benchmarking Social Robot Navigation

• Jarrett Holtz
• Joydeep Biswas

Robots moving safely and in a socially compliant manner in dynamic human environments is an essential benchmark for long-term robot autonomy. However, it is not feasible to learn and benchmark social navigation behaviors entirely in the real world, as learning is data-intensive, and it is challenging to make safety guarantees during training. Therefore, simulation-based benchmarks that provide abstractions for social navigation are required. A framework for these benchmarks would need to support a wide variety of learning approaches, be extensible to the broad range of social navigation scenarios, and abstract away the perception problem to focus on social navigation explicitly. While there have been many proposed solutions, including high fidelity 3D simulators and grid world approximations, no existing solution satisfies all of the aforementioned properties for learning and evaluating social navigation behaviors. In this work, we propose SocialGym, a lightweight 2D simulation environment for robot social navigation designed with extensibility in mind, and a benchmark scenario built on SocialGym. Further, we present benchmark results that compare and contrast human-engineered and model-based learning approaches to a suite of off-the-shelf Learning from Demonstration (LfD) and Reinforcement Learning (RL) approaches applied to social robot navigation. These results demonstrate the data efficiency, task performance, social compliance, and environment transfer capabilities for each of the policies evaluated to provide a solid grounding for future social navigation research.

AAMAS Conference 2022 Conference Paper

State Supervised Steering Function for Sampling-based Kinodynamic Planning

• Pranav Atreya
• Joydeep Biswas

Sampling-based motion planners such as RRT* and BIT*, when applied to kinodynamic motion planning, rely on steering functions to generate time-optimal solutions connecting sampled states. Implementing exact steering functions requires either analytical solutions to the time-optimal control problem, or nonlinear programming (NLP) solvers to solve the boundary value problem given the system’s kinodynamic equations. Unfortunately, analytical solutions are unavailable for many real-world domains, and NLP solvers are prohibitively computationally expensive, hence fast and optimal kinodynamic motion planning remains an open problem. We provide a solution to this problem by introducing State Supervised Steering Function (S3F), a novel approach to learn time-optimal steering functions. S3F is able to produce near-optimal solutions to the steering function orders of magnitude faster than its NLP counterpart. Experiments conducted on three challenging robot domains show that RRT* using S3F significantly outperforms stateof-the-art planning approaches on both solution cost and runtime. We further provide a proof of probabilistic completeness of RRT* modified to use S3F.

IROS Conference 2022 Conference Paper

STEADY: Simultaneous State Estimation and Dynamics Learning from Indirect Observations

• Jiayi Wei
• Jarrett Holtz
• Isil Dillig
• Joydeep Biswas

Accurate kinodynamic models play a crucial role in many robotics applications such as off-road navigation and high-speed driving. Many state-of-the-art approaches for learning stochastic kinodynamic models, however, require precise measurements of robot states as labeled input/output examples, which can be hard to obtain in outdoor settings due to limited sensor capabilities and the absence of ground truth. In this work, we propose a new technique for learning neural stochastic kinodynamic models from noisy and indirect observations by performing simultaneous state estimation and dynamics learning. The proposed technique iteratively improves the kinodynamic model in an expectation-maximization loop, where the E Step samples posterior state trajectories using particle filtering, and the M Step updates the dynamics to be more consistent with the sampled trajectories via stochastic gradient ascent. We evaluate our approach on both simulation and real-world benchmarks and compare it with several baseline techniques. Our approach not only achieves significantly higher accuracy but is also more robust to observation noise, thereby showing promise for boosting the performance of many other robotics applications.

IROS Conference 2022 Conference Paper

VI-IKD: High-Speed Accurate Off-Road Navigation using Learned Visual-Inertial Inverse Kinodynamics

• Haresh Karnan
• Kavan Singh Sikand
• Pranav Atreya
• Sadegh Rabiee
• Xuesu Xiao
• Garrett Warnell
• Peter Stone 0001
• Joydeep Biswas

One of the key challenges in high-speed off-road navigation on ground vehicles is that the kinodynamics of the vehicle-terrain interaction can differ dramatically depending on the terrain. Previous approaches to addressing this challenge have considered learning an inverse kinodynamics (IKD) model, conditioned on inertial information of the vehicle to sense the kinodynamic interactions. In this paper, we hypothesize that to enable accurate high-speed off-road navigation using a learned IKD model, in addition to inertial information from the past, one must also anticipate the kinodynamic interactions of the vehicle with the terrain in the future. To this end, we introduce Visual-Inertial Inverse Kinodynamics (VI-IKD), a novel learning based IKD model that is conditioned on visual information from a terrain patch ahead of the robot in addition to past inertial information, enabling it to anticipate kinodynamic interactions in the future. We validate the effectiveness of VI-IKD in accurate high-speed off-road navigation experimentally on a scale 1/5 UT-AlphaTruck off-road autonomous vehicle in both indoor and outdoor environments and show that compared to other state-of-the-art approaches, VI-IKD enables more accurate and robust off-road navigation on a variety of different terrains at speeds of up to 3. 5m/s.

ICRA Conference 2022 Conference Paper

Visual Representation Learning for Preference-Aware Path Planning

• Kavan Singh Sikand
• Sadegh Rabiee
• Adam Uccello
• Xuesu Xiao
• Garrett Warnell
• Joydeep Biswas

Autonomous mobile robots deployed in outdoor environments must reason about different types of terrain for both safety (e. g. , prefer dirt over mud) and deployer preferences (e. g. , prefer dirt path over flower beds). Most existing solutions to this preference-aware path planning problem use semantic segmentation to classify terrain types from camera images, and then ascribe costs to each type. Unfortunately, there are three key limitations of such approaches - they 1) require preenumeration of the discrete terrain types, 2) are unable to handle hybrid terrain types (e. g. , grassy dirt), and 3) require expensive labelled data to train visual semantic segmentation. We introduce Visual Representation Learning for Preference-Aware Path Planning (VRL-PAP), an alternative approach that overcomes all three limitations: VRL-PAP leverages un-labelled human demonstrations of navigation to autonomously generate triplets for learning visual representations of terrain that are viewpoint invariant and encode terrain types in a continuous representation space. The learned representations are then used along with the same unlabelled human navigation demonstrations to learn a mapping from the representation space to terrain costs. At run time, VRL-PAP maps from images to representations and then representations to costs to perform preference-aware path planning. We present empirical results from challenging outdoor settings that demonstrate VRL-PAP 1) is successfully able to pick paths that reflect demonstrated preferences, 2) is comparable in execution to geometric navigation with a highly detailed manually annotated map (without requiring such annotations), 3) is able to generalize to novel terrain types with minimal additional unlabeled demonstrations.

IROS Conference 2021 Conference Paper

Iterative Program Synthesis for Adaptable Social Navigation

• Jarrett Holtz
• Simon Andrews
• Arjun Guha
• Joydeep Biswas

Robot social navigation is influenced by human preferences and environment-specific scenarios such as elevators and doors, thus necessitating end-user adaptability. State-of-the-art approaches to social navigation fall into two categories: model-based social constraints and learning-based approaches. While effective, these approaches have fundamental limitations – model-based approaches require constraint and parameter tuning to adapt to preferences and new scenarios, while learning-based approaches require reward functions, significant training data, and are hard to adapt to new social scenarios or new domains with limited demonstrations. In this work, we propose Iterative Dimension Informed Program Synthesis (IDIPS) to address these limitations by learning and adapting social navigation in the form of human-readable symbolic programs. IDIPS works by combining pro-gram synthesis, parameter optimization, predicate repair, and iterative human demonstration to learn and adapt model-free action selection policies from orders of magnitude less data than learning-based approaches. We introduce a novel predicate repair technique that can accommodate previously unseen social scenarios or preferences by growing existing policies. We present experimental results showing that IDIPS: 1) synthesizes effective policies that model user preference, 2) can adapt existing policies to changing preferences, 3) can extend policies to handle novel social scenarios such as locked doors, and 4) generates policies that can be transferred from simulation to real-world robots with minimal effort.

IROS Conference 2021 Conference Paper

OneVision: Centralized to Distributed Controller Synthesis with Delay Compensation

• Jiayi Wei
• Tongrui Li
• Swarat Chaudhuri
• Isil Dillig
• Joydeep Biswas

We propose a new algorithm to simplify the controller development for distributed robotic systems subject to external observations, disturbances, and communication delays. Unlike prior approaches that propose specialized solutions to handling communication latency for specific robotic applications, our algorithm uses an arbitrary centralized controller as the specification and automatically generates distributed controllers with communication management and delay compensation. We formulate our goal as nonlinear optimal control— using a regret minimizing objective that measures how much the distributed agents behave differently from the delay-free centralized response—and solve for optimal actions w. r. t. local estimations of this objective using gradient-based optimization. We analyze our proposed algorithm’s behavior under a linear time-invariant special case and prove that the closed-loop dynamics satisfy a form of input-to-state stability w. r. t. unexpected disturbances and observations. Our experimental results on both simulated and real-world robotic tasks demonstrate the practical usefulness of our approach and show significant improvement over several baseline approaches.

IROS Conference 2021 Conference Paper

Robofleet: Open Source Communication and Management for Fleets of Autonomous Robots

• Kavan Singh Sikand
• Logan Zartman
• Sadegh Rabiee
• Joydeep Biswas

Long-term deployment of a fleet of mobile robots requires reliable and secure two-way communication channels between individual robots and remote human operators for supervision and tasking. Existing open-source solutions to this problem degrade in performance in challenging real-world situations such as intermittent and low-bandwidth connectivity, do not provide security control options, and can be computationally expensive on hardware-constrained mobile robot platforms. In this paper, we present Robofleet, a lightweight open-source system which provides inter-robot communication, remote monitoring, and remote tasking for a heterogenous fleet of ROS-enabled service-mobile robots that is designed with the practical goals of resilience to network variance and security control in mind. Robofleet supports multi-user, multi-robot communication via a central server. This architecture deduplicates network traffic between robots, significantly reducing overall network load when compared with native ROS communication. This server also functions as a single entrypoint into the system, enabling security control and user authentication. Individual robots run the lightweight Robofleet client, which is responsible for exchanging messages with the Robofleet server. It automatically adapts to adverse network conditions through backpressure monitoring as well as topic-level priority control, ensuring that safety-critical messages are successfully transmitted. Finally, the system includes a web-based visualization tool that can be run on any internet-connected, browser-enabled device to monitor and control the fleet. We compare Robofleet to existing methods of robotic communication, and demonstrate that it provides superior resilience to network variance while maintaining performance that exceeds that of widely-used systems.

AIJ Journal 2021 Journal Article

X*: Anytime Multi-Agent Path Finding for Sparse Domains using Window-Based Iterative Repairs

• Kyle Vedder
• Joydeep Biswas

ICRA Conference 2019 Conference Paper

A Friction-Based Kinematic Model for Skid-Steer Wheeled Mobile Robots

• Sadegh Rabiee
• Joydeep Biswas

Skid-steer drive systems are widely used in mobile robot platforms. Such systems are subject to significant slippage and skidding during normal operation due to their nature. The ability to predict and compensate for such slippages in the forward kinematics of these types of robots is of great importance and provides the means for accurate control and safe navigation. In this work, we propose a new kinematic model capable of slip prediction for skid-steer wheeled mobile robots (SSWMRs). The proposed model outperforms the state-of-the-art in terms of both translational and rotational prediction error on a dataset composed of more than 6 km worth of trajectories traversed by a skid-steer robot. We also publicly release our dataset to serve as a benchmark for system identification and model learning research for SSWMRs.

IROS Conference 2019 Conference Paper

Belief Space Metareasoning for Exception Recovery

• Justin Svegliato
• Kyle Hollins Wray
• Stefan J. Witwicki
• Joydeep Biswas
• Shlomo Zilberstein

Due to the complexity of the real world, autonomous systems use decision-making models that rely on simplifying assumptions to make them computationally tractable and feasible to design. However, since these limited representations cannot fully capture the domain of operation, an autonomous system may encounter unanticipated scenarios that cannot be resolved effectively. We first formally introduce an introspective autonomous system that uses belief space metareasoning to recover from exceptions by interleaving a main decision process with a set of exception handlers. We then apply introspective autonomy to autonomous driving. Finally, we demonstrate that an introspective autonomous vehicle is effective in simulation and on a fully operational prototype.

IROS Conference 2019 Conference Paper

IVOA: Introspective Vision for Obstacle Avoidance

• Sadegh Rabiee
• Joydeep Biswas

Vision, as an inexpensive yet information rich sensor, is commonly used for perception on autonomous mobile robots. Unfortunately, accurate vision-based perception requires a number of assumptions about the environment to hold – some examples of such assumptions, depending on the perception algorithm at hand, include purely lambertian surfaces, texture-rich scenes, absence of aliasing features, and refractive surfaces. In this paper, we present an approach for introspective vision for obstacle avoidance (IVOA) – by leveraging a supervisory sensor that is occasionally available, we detect failures of stereo vision-based perception from divergence in plans generated by vision and the supervisory sensor. By projecting the 3D coordinates where the plans agree and disagree onto the images used for vision-based perception, IVOA generates a training set of reliable and unreliable image patches for perception. We then use this training dataset to learn a model of which image patches are likely to cause failures of the vision-based perception algorithm. Using this model, IVOA is then able to predict whether the relevant image patches in the observed images are likely to cause failures due to vision (both false positives and false negatives). We empirically demonstrate with extensive real-world data from both indoor and outdoor environments, the ability of IVOA to accurately predict the failures of two distinct vision algorithms.

IJCAI Conference 2019 Conference Paper

The Quest For " Always-On" Autonomous Mobile Robots

• Joydeep Biswas

Building ``always-on'' robots to be deployed over extended periods of time in real human environments is challenging for several reasons. Some fundamental questions that arise in the process include: 1) How can the robot reconcile unexpected differences between its observations and its outdated map of the world? 2) How can we scalably test robots for long-term autonomy? 3) Can a robot learn to predict its own failures, and their corresponding causes? 4) When the robot fails and is unable to recover autonomously, can it utilize partially specified, approximate human corrections to overcome its failures? We summarize our research towards addressing all of these questions. We present 1) Episodic non-Markov Localization to maintain the belief of the robot's location while explicitly reasoning about unmapped observations; 2) a 1, 000km challenge to test for long-term autonomy; 3) feature-based and learning-based approaches to predicting failures; and 4) human-in-the-loop SLAM to overcome robot mapping errors, and SMT-based robot transition repair to overcome state machine failures.

AAMAS Conference 2019 Conference Paper

X*: Anytime Multiagent Planning With Bounded Search

• Kyle Vedder
• Joydeep Biswas

Multi-agent planning in dynamic domains is a challenging problem: the size of the configuration space increases exponentially in the number of agents, and plans need to be re-evaluated periodically to account for moving obstacles. However, we have two key insights that hold in several domains: 1) conflicts between multi-agent plans often have geometrically local resolutions within a small repair window, even if such local resolutions are not globally optimal; and 2) the partial search tree for such local resolutions can then be iteratively improved over successively larger windows to eventually compute the global optimal plan. Building upon these two insights, we introduce 1) a class of anytime multiagent planning solvers, 2) a naïve solver in this class, and 3) an efficient solver in this class which reuses prior search information when improving a solution.

IROS Conference 2018 Conference Paper

A Real- Time Solver for Time-Optimal Control of Omnidirectional Robots with Bounded Acceleration

• David Balaban
• Alexander Fischer
• Joydeep Biswas

We are interested in the problem of time-optimal control of omnidirectional robots with bounded acceleration (TOC-ORBA). While there exist approximate solutions for such problems, and exact solutions with unbounded acceleration, exact solvers to the TOC-ORBA problem have remained elusive until now. In this paper, we present a real-time solver for true time-optimal control of omnidirectional robots with bounded acceleration. We first derive the general parameterized form of the solution to the TOC-ORBA problem by application of Pontryagin's maximum principle. We then frame the boundary value problem of TOC-ORBA as an optimization problem over the parameterized control space. To overcome local minima and poor initial guesses to the optimization problem, we introduce a two-stage optimal control solver (TSOCS): The first stage computes an upper bound to the total time for the TOC-ORBA problem and holds the time constant while optimizing the parameters of the trajectory to approach the boundary value conditions. The second stage uses the parameters found by the first stage, and relaxes the constraint on the total time to solve for the parameters of the complete TOC-ORBA problem. Furthermore, we implement TSOCS as a closed loop controller to overcome actuation errors on real robots in realtime. We empirically demonstrate the effectiveness of TSOCS in simulation and on real robots, showing that 1) it runs in real time, generating solutions in less than 0. 5ms on average; 2) it generates faster trajectories compared to an approximate solver; and 3) it is able to solve TOC-ORBA problems with nonzero final velocities that were previously unsolvable in real-time.

AAMAS Conference 2018 Conference Paper

Demo: Interactive Robot Transition Repair

• Jarrett Holtz
• Arjun Guha
• Joydeep Biswas

Complex robot behaviors are often structured as state machines, where states encapsulate actions and a transition function switches between states. Since transitions depend on physical parameters, when the environment changes, a roboticist has to painstakingly readjust the parameters to work in the new environment. In this demo we present Interactive SMT-based Robot Transition Repair (SRTR): instead of manually adjusting parameters, we ask users to identify a few instances where the robot is in a wrong state and what the right state should be. A lightweight automated analysis of the transition function’s source code then 1) identifies adjustable parameters, 2) converts the transition function into a system of logical constraints, and 3) formulates the constraints and usersupplied corrections as a MaxSMT problem that yields adjustments to parameter values. This demo uses a simulated RoboCup Small Size League platform, allows users to correct faulty behaviors, and then uses SRTR to adjust parameters automatically.

AAAI Conference 2018 Conference Paper

Human-in-the-Loop SLAM

• Samer Nashed
• Joydeep Biswas

Building large-scale, globally consistent maps is a challenging problem, made more difficult in environments with limited access, sparse features, or when using data collected by novice users. For such scenarios, where state-of-the-art mapping algorithms produce globally inconsistent maps, we introduce a systematic approach to incorporating sparse human corrections, which we term Human-in-the-Loop Simultaneous Localization and Mapping (HitL-SLAM). Given an initial factor graph for pose graph SLAM, HitL-SLAM accepts approximate, potentially erroneous, and rank-deficient human input, infers the intended correction via expectation maximization (EM), back-propagates the extracted corrections over the pose graph, and finally jointly optimizes the factor graph including the human inputs as human correction factor terms, to yield globally consistent large-scale maps. We thus contribute an EM formulation for inferring potentially rank-deficient human corrections to mapping, and human correction factor extensions to the factor graphs for pose graph SLAM that result in a principled approach to joint optimization of the pose graph while simultaneously accounting for multiple forms of human correction. We present empirical results showing the effectiveness of HitL-SLAM at generating globally accurate and consistent maps even when given poor initial estimates of the map.

IJCAI Conference 2018 Conference Paper

Interactive Robot Transition Repair With SMT

• Jarrett Holtz
• Arjun Guha
• Joydeep Biswas

Complex robot behaviors are often structured as state machines, where states encapsulate actions and a transition function switches between states. Since transitions depend on physical parameters, when the environment changes, a roboticist has to painstakingly readjust the parameters to work in the new environment. We present interactive SMT- based Robot Transition Repair (SRTR): instead of manually adjusting parameters, we ask the roboticist to identify a few instances where the robot is in a wrong state and what the right state should be. An automated analysis of the transition function 1) identifies adjustable parameters, 2) converts the transition function into a system of logical constraints, and 3) formulates the constraints and user-supplied corrections as a MaxSMT problem that yields new parameter values. We show that SRTR finds new parameters 1) quickly, 2) with few corrections, and 3) that the parameters generalize to new scenarios. We also show that a SRTR-corrected state machine can outperform a more complex, expert-tuned state machine.

ICRA Conference 2018 Conference Paper

Localization Under Topological Uncertainty for Lane Identification of Autonomous Vehicles

• Samer B. Nashed
• David M. Ilstrup
• Joydeep Biswas

Autonomous vehicles (AVs) require accurate metric and topological location estimates for safe, effective navigation and decision-making. Although many high-definition (HD) roadmaps exist, they are not always accurate since public roads are dynamic, shaped unpredictably by both human activity and nature. Thus, AVs must be able to handle situations in which the topology specified by the map does not agree with reality. We present the Variable Structure Multiple Hidden Markov Model (VSM-HMM) as a framework for localizing in the presence of topological uncertainty, and demonstrate its effectiveness on an AV where lane membership is modeled as a topological localization process. VSM-HMMs use a dynamic set of HMMs to simultaneously reason about location within a set of most likely current topologies and therefore may also be applied to topological structure estimation as well as AV lane estimation. In addition, we present an extension to the Earth Mover's Distance which allows uncertainty to be taken into account when computing the distance between belief distributions on simplices of arbitrary relative sizes.

IROS Conference 2017 Conference Paper

Automatic extrinsic calibration of depth sensors with ambiguous environments and restricted motion

• Jarrett Holtz
• Joydeep Biswas

Autonomous mobile robots that use multiple depth sensors to perceive their environments, rely on extrinsic calibration to combine the individual views from each sensor into a single coherent view of the surroundings. Such extrinsic calibration is tedious to perform manually, and requires that specific scenes to calibrate. Current state of the art automatic approaches do not consider the content of scenes used for calibration, and thus are not robust to partially informative scenes in long-term deployments. In this paper, we present Delta-Calibration, an automated extrinsic calibration technique that takes into account the information in a scene for calibration. Delta-Calibration relies on constrained sensor motion to minimize the effects of desynchronization, and ego-motion estimation from each depth camera to detect significant changes in pose, which we term Delta-Transforms. We derive a solution to the extrinsic calibration using such Delta-Transforms taking into account uncertain axes of motion in the environment, and further infer necessary and sufficient conditions on the Delta-Transforms such that Delta-Calibration results in a unique, non-singular, and numerically stable extrinsic calibration. We present quantitative and qualitative results demonstrating the effectiveness of Delta-Calibration at computing extrinsic calibration over different arrangements of depth sensors.

IROS Conference 2017 Conference Paper

Joint perception and planning for efficient obstacle avoidance using stereo vision

• Sourish Ghosh
• Joydeep Biswas

Stereo vision is commonly used for local obstacle avoidance of autonomous mobile robots: stereo images are first processed to yield a dense 3D reconstruction of the observed scene, which is then used for navigation planning. Such an approach, which we term Sequential Perception and Planning (SPP), results in significant unnecessary computations as the navigation planner only needs to explore a small part of the scene to compute the shortest obstacle-free path. In this paper, we introduce an approach to Joint Perception and Planning (JPP) using stereo vision, which performs disparity checks on demand, only as necessary while searching on a planning graph. Furthermore, obstacle checks for navigation planning do not require full 3D reconstruction: we present in this paper how obstacle queries can be decomposed into a sequence of confident positive stereo matches and confident negative stereo matches, which are significantly faster to compute than the exact depth of points. The resulting complete JPP formulation is significantly faster than SPP, while still maintaining correctness of planning. We also show how the JPP works with different planners, including search-based and sampling-based planners. We present extensive experimental results from real robot data and simulation experiments, demonstrating that the JPP requires less than 10% of the disparity computations required by SPP.

IROS Conference 2016 Conference Paper

Curating Long-Term Vector Maps

• Samer B. Nashed
• Joydeep Biswas

Autonomous service mobile robots need to consistently, accurately, and robustly localize in human environments despite changes to such environments over time. Episodic non-Markov Localization addresses the challenge of localization in such changing environments by classifying observations as arising from Long-Term, Short-Term, or Dynamic Features. However, in order to do so, EnML relies on an estimate of the Long-Term Vector Map (LTVM) that does not change over time. In this paper, we introduce a recursive algorithm to build and update the LTVM over time by reasoning about visibility constraints of objects observed over multiple robot deployments. We use a signed distance function (SDF) to filter out observations of short-term and dynamic features from multiple deployments of the robot. The remaining long-term observations are used to build a vector map by robust local linear regression. The uncertainty in the resulting LTVM is computed via Monte Carlo resampling the observations arising from long-term features. By combining occupancy-grid based SDF filtering of observations with continuous space regression of the filtered observations, our proposed approach builds, updates, and amends LTVMs over time, reasoning about all observations from all robot deployments in an environment. We present experimental results demonstrating the accuracy, robustness, and compact nature of the extracted LTVMs from several long-term robot datasets.

AAAI Conference 2016 Conference Paper

Selectively Reactive Coordination for a Team of Robot Soccer Champions

• Juan Pablo Mendoza
• Joydeep Biswas
• Philip Cooksey
• Richard Wang
• Steven Klee
• Danny Zhu
• Manuela Veloso

CMDragons 2015 is the champion of the RoboCup Small Size League of autonomous robot soccer. The team won all of its six games, scoring a total of 48 goals and conceding 0. This unprecedented dominant performance is the result of various features, but we particularly credit our novel offense multi-robot coordination. This paper thus presents our Selectively Reactive Coordination (SRC) algorithm, consisting of two layers: A coordinated opponent-agnostic layer enables the team to create its own plans, setting the pace of the game in offense. An individual opponent-reactive action selection layer enables the robots to maintain reactivity to different opponents. We demonstrate the effectiveness of our coordination through results from RoboCup 2015, and through controlled experiments using a physics-based simulator and an automated referee.

IS Journal 2016 Journal Article

The 1,000-km Challenge: Insights and Quantitative and Qualitative Results

• Joydeep Biswas
• Manuela Veloso

On 18 November 2014, a team of four autonomous CoBot robots reached 1, 000-km of overall autonomous navigation, as a result of a 1, 000-km challenge that the authors had set three years earlier. The authors are frequently asked for the lessons learned, as well as the performance results. In this article, they introduce the challenge and contribute a detailed presentation of technical insights as well as quantitative and qualitative results. They have previously presented the algorithms for the individual technical contributions, namely robot localization, symbiotic robot autonomy, and robot task scheduling. In this article, they present the data collected over the 1, 000-km challenge and analyze it to evaluate the accuracy and robustness of the localization algorithms on the CoBots. Furthermore, they present technical insights into the algorithms, which they believe are responsible for the robots' continuous robust performance.

IJCAI Conference 2015 Conference Paper

CoBots: Robust Symbiotic Autonomous Mobile Service Robots

• Manuela Veloso
• Joydeep Biswas
• Brian Coltin
• Stephanie Rosenthal

We research and develop autonomous mobile service robots as Collaborative Robots, i. e. , CoBots. For the last three years, our four CoBots have autonomously navigated in our multi-floor office buildings for more than 1, 000km, as the result of the integration of multiple perceptual, cognitive, and actuations representations and algorithms. In this paper, we identify a few core aspects of our CoBots underlying their robust functionality. The reliable mobility in the varying indoor environments comes from a novel episodic non-Markov localization. Service tasks requested by users are the input to a scheduler that can consider different types of constraints, including transfers among multiple robots. With symbiotic autonomy, the CoBots proactively seek external sources of help to fill-in for their inevitable occasional limitations. We present sampled results from a deployment and conclude with a brief review of other features of our service robots.

ICRA Conference 2014 Conference Paper

Episodic non-Markov localization: Reasoning about short-term and long-term features

• Joydeep Biswas
• Manuela Veloso

Markov localization and its variants are widely used for localization of mobile robots. These methods assume Markov independence of observations, implying that observations made by a robot correspond to a static map. However, in real human environments, observations include occlusions due to unmapped objects like chairs and tables, and dynamic objects like humans. We introduce an episodic non-Markov localization algorithm that maintains estimates of the belief over the trajectory of the robot while explicitly reasoning about observations and their correlations arising from unmapped static objects, moving objects, as well as objects from the static map. Observations are classified as arising from long-term features, short-term features, or dynamic features, which correspond to mapped objects, unmapped static objects, and unmapped dynamic objects respectively. By detecting time steps along the robot's trajectory where unmapped observations prior to such time steps are unrelated to those afterwards, non-Markov localization limits the history of observations and pose estimates to “episodes” over which the belief is computed. We demonstrate non-Markov localization in challenging real world indoor and outdoor environments over multiple datasets, comparing it with alternative state-of-the-art approaches, showing it to be robust as well as accurate.

ICRA Conference 2013 Conference Paper

Fast human detection for indoor mobile robots using depth images

• Benjamin Choi
• Çetin Meriçli
• Joydeep Biswas
• Manuela Veloso

A human detection algorithm running on an indoor mobile robot has to address challenges including occlusions due to cluttered environments, changing backgrounds due to the robot's motion, and limited on-board computational resources. We introduce a fast human detection algorithm for mobile robots equipped with depth cameras. First, we segment the raw depth image using a graph-based segmentation algorithm. Next, we apply a set of parameterized heuristics to filter and merge the segmented regions to obtain a set of candidates. Finally, we compute a Histogram of Oriented Depth (HOD) descriptor for each candidate, and test for human presence with a linear SVM. We experimentally evaluate our approach on a publicly available dataset of humans in an open area as well as our own dataset of humans in a cluttered cafe environment. Our algorithm performs comparably well on a single CPU core against another HOD-based algorithm that runs on a GPU even when the number of training examples is decreased by half. We discuss the impact of the number of training examples on performance, and demonstrate that our approach is able to detect humans in different postures (e. g. standing, walking, sitting) and with occlusions.

IROS Conference 2012 Conference Paper

CoBots: Collaborative robots servicing multi-floor buildings

• Manuela Veloso
• Joydeep Biswas
• Brian Coltin
• Stephanie Rosenthal
• Thomas Kollar
• Çetin Meriçli
• Mehdi Samadi
• Susana Brandão

In this video we briefly illustrate the progress and contributions made with our mobile, indoor, service robots CoBots (Collaborative Robots), since their creation in 2009. Many researchers, present authors included, aim for autonomous mobile robots that robustly perform service tasks for humans in our indoor environments. The efforts towards this goal have been numerous and successful, and we build upon them. However, there are clearly many research challenges remaining until we can experience intelligent mobile robots that are fully functional and capable in our human environments.

ICRA Conference 2012 Conference Paper

Depth camera based indoor mobile robot localization and navigation

• Joydeep Biswas
• Manuela Veloso

The sheer volume of data generated by depth cameras provides a challenge to process in real time, in particular when used for indoor mobile robot localization and navigation. We introduce the Fast Sampling Plane Filtering (FSPF) algorithm to reduce the volume of the 3D point cloud by sampling points from the depth image, and classifying local grouped sets of points as belonging to planes in 3D (the “plane filtered” points) or points that do not correspond to planes within a specified error margin (the “outlier” points). We then introduce a localization algorithm based on an observation model that down-projects the plane filtered points on to 2D, and assigns correspondences for each point to lines in the 2D map. The full sampled point cloud (consisting of both plane filtered as well as outlier points) is processed for obstacle avoidance for autonomous navigation. All our algorithms process only the depth information, and do not require additional RGB data. The FSPF, localization and obstacle avoidance algorithms run in real time at full camera frame rates (30Hz) with low CPU requirements (16%). We provide experimental results demonstrating the effectiveness of our approach for indoor mobile robot localization and navigation. We further compare the accuracy and robustness in localization using depth cameras with FSPF vs. alternative approaches that simulate laser rangefinder scans from the 3D data.

IROS Conference 2012 Conference Paper

Planar polygon extraction and merging from depth images

• Joydeep Biswas
• Manuela Veloso

There has been considerable interest recently in building 3D maps of environments using inexpensive depth cameras like the Microsoft Kinect sensor. We exploit the fact that typical indoor scenes have an abundance of planar features by modeling environments as sets of plane polygons. To this end, we build upon the Fast Sampling Plane Filtering (FSPF) algorithm that extracts points belonging to local neighborhoods of planes from depth images, even in the presence of clutter. We introduce an algorithm that uses the FSPF-generated plane filtered point clouds to generate convex polygons from individual observed depth images. We then contribute an approach of merging these detected polygons across successive frames while accounting for a complete history of observed plane filtered points without explicitly maintaining a list of all observed points. The FSPF and polygon merging algorithms run in real time at full camera frame rates with low CPU requirements: in a real world indoor environment scene, the FSPF and polygon merging algorithms take 2. 5 ms on average to process a single 640 × 480 depth image. We provide experimental results demonstrating the computational efficiency of the algorithm and the accuracy of the detected plane polygons by comparing with ground truth.

IROS Conference 2011 Conference Paper

Corrective gradient refinement for mobile robot localization

• Joydeep Biswas
• Brian Coltin
• Manuela Veloso

Particle filters for mobile robot localization must balance computational requirements and accuracy of localization. Increasing the number of particles in a particle filter improves accuracy, but also increases the computational requirements. Hence, we investigate a different paradigm to better utilize particles than to increase their numbers. To this end, we introduce the Corrective Gradient Refinement (CGR) algorithm that uses the state space gradients of the observation model to improve accuracy while maintaining low computational requirements. We develop an observation model for mobile robot localization using point cloud sensors (LIDAR and depth cameras) with vector maps. This observation model is then used to analytically compute the state space gradients necessary for CGR. We show experimentally that the resulting complete localization algorithm is more accurate than the Sampling/Importance Resampling Monte Carlo Localization algorithm, while requiring fewer particles.

AAMAS Conference 2010 Conference Paper

An Effective Personal Mobile Robot Agent Through Symbiotic Human-Robot Interaction

• Stephanie Rosenthal
• Joydeep Biswas
• Manuela Veloso

Several researchers, present authors included, envision personalmobile robot agents that can assist humans in their dailytasks. Despite many advances in robotics, such mobile robot agentsstill face many limitations in their perception, cognition, and actioncapabilities. In this work, we propose a symbiotic interaction betweenrobot agents and humans to overcome the robot limitations whileallowing robots to also help humans. We introduce a visitor'scompanion robot agent, as a natural task for such symbioticinteraction, e. g. , the visitor lacks knowledge of the environment butcan easily open a door or read a door label, while the mobile robotwith no arms cannot open a door and may be confused about its exactlocation, but can plan paths well through the building and can provideuseful relevant information to the visitor. We present this visitorcompanion task in detail with an enumeration and formalization of theactions of the robot agent in its interaction with the human. Webriefly describe the wifi-based robot localization algorithm and showresults of the different levels of human help to the robot during the robotnavigation. We then model the tradeoffs of the value of the robot help to the human and present illustrative experiments. Our work has been fully implemented in a mobile robot agent, CoBot, which has successfully navigated for several hours and continues to navigate in our indoor environment.

ICRA Conference 2010 Conference Paper

WiFi localization and navigation for autonomous indoor mobile robots

• Joydeep Biswas
• Manuela Veloso

Building upon previous work that demonstrates the effectiveness of WiFi localization information per se, in this paper we contribute a mobile robot that autonomously navigates in indoor environments using WiFi sensory data. We model the world as a WiFi signature map with geometric constraints and introduce a continuous perceptual model of the environment generated from the discrete graph-based WiFi signal strength sampling. We contribute our WiFi localization algorithm which continuously uses the perceptual model to update the robot location in conjunction with its odometry data. We then briefly introduce a navigation approach that robustly uses the WiFi location estimates. We present the results of our exhaustive tests of the WiFi localization independently and in conjunction with the navigation of our custom-built mobile robot in extensive long autonomous runs.