Jonathan P. How

ICRA Conference 2025 Conference Paper

PRIMER: Perception-Aware Robust Learning-Based Multiagent Trajectory Planner

• Kota Kondo
• Claudius T. Tewari
• Andrea Tagliabue
• Jesus Tordesillas
• Parker C. Lusk
• Mason B. Peterson
• Jonathan P. How

In decentralized multiagent trajectory planners, agents need to communicate and exchange their positions to generate collision-free trajectories. However, due to localization errors/uncertainties, trajectory deconfliction can fail even if trajectories are perfectly shared between agents. To address this issue, we first present PARM and PARM*, perception-aware, decentralized, asynchronous multiagent trajectory planners that enable a team of agents to navigate uncertain environments while deconflicting trajectories and avoiding obstacles using perception information. PARM* differs from PARM as it is less conservative, using more computation to find closer-to-optimal solutions. While these methods achieve state-of-the-art performance, they suffer from high computational costs as they need to solve large optimization problems onboard, making it difficult for agents to replan at high rates. To overcome this challenge, we present our second key contribution, PRIMER, a learning-based planner trained with imitation learning (IL) using PARM* as the expert demonstrator. PRIMER leverages the low computational requirements at deployment of neural networks and achieves a computation speed up to 5614 times faster than optimization-based approaches.

ICRA Conference 2025 Conference Paper

TCAFF: Temporal Consistency for Robot Frame Alignment

• Mason B. Peterson
• Parker C. Lusk
• Antonio Avila
• Jonathan P. How

In the field of collaborative robotics, the ability to communicate spatial information like planned trajectories and shared environment information is crucial. When no global position information is available (e. g. , indoor or GPS-denied environments), agents must align their coordinate frames before shared spatial information can be properly expressed and interpreted. Coordinate frame alignment is particularly difficult when robots have no initial alignment and are affected by odometry drift. To this end, we develop a novel multiple hypothesis algorithm, called TCAFF, for aligning the coordinate frames of neighboring robots. TCAFF considers potential alignments from associating sparse open-set object maps and leverages temporal consistency to determine an initial alignment and correct for drift, all without any initial knowledge of neighboring robot poses. We demonstrate TCAFF being used for frame alignment in a collaborative object tracking application on a team of four robots tracking six pedestrians and show that TCAFF enables robots to achieve a tracking accuracy similar to that of a system with ground truth localization. The code and hardware dataset are available at https://github.com/mit-acl/tcaff.

AIJ Journal 2024 Journal Article

Finding the optimal exploration-exploitation trade-off online through Bayesian risk estimation and minimization

• Stewart Jamieson
• Jonathan P. How
• Yogesh Girdhar

We propose endogenous Bayesian risk minimization (EBRM) over policy sets as an approach to online learning across a wide range of settings. Many real-world online learning problems have complexities such as action- and belief-dependent rewards, time-discounting of reward, and heterogeneous costs for actions and feedback; we find that existing online learning heuristics cannot leverage most problem-specific information, to the detriment of their performance. We introduce a belief-space Markov decision process (BMDP) model that can capture these complexities, and further apply the concepts of aleatoric, epistemic, and process risks to online learning. These risk functions describe the risk inherent to the learning problem, the risk due to the agent's lack of knowledge, and the relative quality of its policy, respectively. We demonstrate how computing and minimizing these risk functions guides the online learning agent towards the optimal exploration-exploitation trade-off in any stochastic online learning problem, constituting the basis of the EBRM approach. We also show how Bayes' risk, the minimization objective in stochastic online learning problems, can be decomposed into the aforementioned aleatoric, epistemic, and process risks. In simulation experiments, EBRM algorithms achieve state-of-the-art performance across various classical online learning problems, including Gaussian and Bernoulli multi-armed bandits, best-arm identification, mixed objectives with action- and belief-dependent rewards, and dynamic pricing, a finite partial monitoring problem. To our knowledge, it is also the first computationally efficient online learning approach that can provide online bounds on an algorithm's Bayes' risk. Finally, because the EBRM approach is parameterized by a set of policy algorithms, it can be extended to incorporate new developments in online learning algorithms, and is thus well-suited as the foundation for developing real-world learning agents.

IROS Conference 2024 Conference Paper

Look Before You Leap: Socially Acceptable High-Speed Ground Robot Navigation in Crowded Hallways

• Lakshay Sharma
• Nicolaniello Buono
• Ashton Flather
• Xiaoyi Cai
• Jonathan P. How

To operate safely and efficiently, autonomous warehouse/delivery robots must be able to accomplish tasks while navigating in dynamic environments and handling the large uncertainties associated with the motions/behaviors of other robots and/or humans. A key scenario in such environments is the hallway problem, where robots must operate in the same narrow corridor as human traffic going in one or both directions. Traditionally, robot planners have tended to focus on socially acceptable behavior in the hallway scenario at the expense of performance. This paper proposes a planner that aims to address the consequent "robot freezing problem" in hallways by allowing for "peek-and-pass" maneuvers. We then go on to demonstrate in simulation how this planner improves robot time to goal without violating social norms. Finally, we show initial hardware demonstrations of this planner in the real world, along with a novel STAR (Socially Trained Agile Robot) platform designed with human comfort in mind.

ICRA Conference 2024 Conference Paper

Online Data-Driven Safety Certification for Systems Subject to Unknown Disturbances

• Nicholas Rober
• Karan Mahesh
• Tyler M. Paine
• Max L. Greene
• Steven Lee
• Sildomar T. Monteiro
• Michael R. Benjamin
• Jonathan P. How

Deploying autonomous systems in safety critical settings necessitates methods to verify their safety properties. This is challenging because real-world systems may be subject to disturbances that affect their performance, but are unknown a priori. This work develops a safety-verification strategy wherein data is collected online and incorporated into a reachability analysis approach to check in real-time that the system avoids dangerous regions of the state space. Specifically, we employ an optimization-based moving horizon estimator (MHE) to characterize the disturbance affecting the system, which is incorporated into an online reachability calculation. Reachable sets are calculated using a computational graph analysis tool to predict the possible future states of the system and verify that they satisfy safety constraints. We include theoretical arguments proving our approach generates reachable sets that bound the future states of the system, as well as numerical results demonstrating how it can be used for safety verification. Finally, we present results from hardware experiments demonstrating our approach’s ability to perform online reachability calculations for an unmanned surface vehicle subject to currents and actuator failures.

ICRA Conference 2024 Conference Paper

PUMA: Fully Decentralized Uncertainty-aware Multiagent Trajectory Planner with Real-time Image Segmentation-based Frame Alignment

• Kota Kondo
• Claudius T. Tewari
• Mason B. Peterson
• Annika Thomas
• Jouko Kinnari
• Andrea Tagliabue
• Jonathan P. How

Fully decentralized, multiagent trajectory planners enable complex tasks like search and rescue or package delivery by ensuring safe navigation in unknown environments. However, deconflicting trajectories with other agents and ensuring collision-free paths in a fully decentralized setting is complicated by dynamic elements and localization uncertainty. To this end, this paper presents (1) an uncertainty-aware multiagent trajectory planner and (2) an image segmentation-based frame alignment pipeline. The uncertainty-aware planner propagates uncertainty associated with the future motion of detected obstacles, and by incorporating this propagated uncertainty into optimization constraints, the planner effectively navigates around obstacles. Unlike conventional methods that emphasize explicit obstacle tracking, our approach integrates implicit tracking. Moreover, sharing trajectories between agents can cause potential collisions due to frame misalignment. Addressing this, we introduce a novel frame alignment pipeline that rectifies inter-agent frame misalignment. This method leverages a zero-shot image segmentation model for detecting objects in the environment and a data association framework based on geometric consistency for map alignment. Our approach accurately aligns frames with only 0. 18 m and 2. 7° of mean frame alignment error in our most challenging simulation scenario. In addition, we conducted hardware experiments and successfully achieved 0. 29 m and 2. 59° of frame alignment error. Together with the alignment framework, our planner ensures safe navigation in unknown environments and collision avoidance in decentralized settings.

IROS Conference 2024 Conference Paper

SOS-Match: Segmentation for Open-Set Robust Correspondence Search and Robot Localization in Unstructured Environments

• Annika Thomas
• Jouko Kinnari
• Parker C. Lusk
• Kota Kondo
• Jonathan P. How

We present SOS-Match, a novel framework for detecting and matching objects in unstructured environments. Our system consists of 1) a front-end mapping pipeline using a zero-shot segmentation model to extract object masks from images and track them across frames and 2) a frame alignment pipeline that uses the geometric consistency of object relationships to efficiently localize across a variety of conditions. We evaluate SOS-Match on the Båtvik seasonal dataset which includes drone flights collected over a coastal plot of southern Finland during different seasons and lighting conditions. Results show that our approach is more robust to changes in lighting and appearance than classical image feature-based approaches or global descriptor methods, and it provides more viewpoint invariance than learning-based feature detection and description approaches. SOS-Match localizes within a reference map up to 46x faster than other feature-based approaches and has a map size less than 0. 5% the size of the most compact other maps. SOS-Match is a promising new approach for landmark detection and correspondence search in unstructured environments that is robust to changes in lighting and appearance and is more computationally efficient than other approaches, suggesting that the geometric arrangement of segments is a valuable localization cue in unstructured environments. We release our datasets at https://acl.mit.edu/SOS-Match/.

ICRA Conference 2023 Conference Paper

DeepSeeColor: Realtime Adaptive Color Correction for Autonomous Underwater Vehicles via Deep Learning Methods

• Stewart Jamieson
• Jonathan P. How
• Yogesh A. Girdhar

Successful applications of complex vision-based behaviours underwater have lagged behind progress in terrestrial and aerial domains. This is largely due to the degraded image quality resulting from the physical phenomena involved in underwater image formation. Spectrally-selective light attenuation drains some colors from underwater images while backscattering adds others, making it challenging to perform vision-based tasks underwater. State-of-the-art methods for underwater color correction optimize the parameters of image formation models to restore the full spectrum of color to underwater imagery. However, these methods have high computational complexity that is unfavourable for realtime use by autonomous underwater vehicles (AUVs), as a result of having been primarily designed for offline color correction. Here, we present DeepSeeColor, a novel algorithm that combines a state-of-the-art underwater image formation model with the computational efficiency of deep learning frameworks. In our experiments, we show that DeepSeeColor offers comparable performance to the popular “Sea-Thru” algorithm [1] while being able to rapidly process images at up to 60Hz, thus making it suitable for use onboard AUVs as a preprocessing step to enable more robust vision-based behaviours.

IROS Conference 2023 Conference Paper

Efficient Deep Learning of Robust, Adaptive Policies using Tube MPC-Guided Data Augmentation

• Tong Zhao
• Andrea Tagliabue
• Jonathan P. How

The deployment of agile autonomous systems in challenging, unstructured environments requires adaptation capabilities and robustness to uncertainties. Existing robust and adaptive controllers, such as those based on model predictive control (MPC), can achieve impressive performance at the cost of heavy online onboard computations. Strategies that efficiently learn robust and onboard-deployable policies from MPC have emerged, but they still lack fundamental adaptation capabilities. In this work, we extend an existing efficient Imitation Learning (IL) algorithm for robust policy learning from MPC with the ability to learn policies that adapt to challenging model/environment uncertainties. The key idea of our approach consists in modifying the IL procedure by conditioning the policy on a learned lower-dimensional model/environment representation that can be efficiently estimated online. We tailor our approach to the task of learning an adaptive position and attitude control policy to track trajectories under challenging disturbances on a multirotor. Evaluations in simulation show that a high-quality adaptive policy can be obtained in about 1. 3 hours. We additionally empirically demonstrate rapid adaptation to in- and out-of-training-distribution uncertainties, achieving a 6. 1 cm average position error under wind disturbances that correspond to about 50% of the weight of the robot, and that are 36% larger than the maximum wind seen during training.

IROS Conference 2023 Conference Paper

Global Localization in Unstructured Environments Using Semantic Object Maps Built from Various Viewpoints

• Jacqueline Ankenbauer
• Parker C. Lusk
• Annika Thomas
• Jonathan P. How

We present a novel framework for global localization and guided relocalization of a vehicle in an unstructured environment. Compared to existing methods, our pipeline does not rely on cues from urban fixtures (e. g. , lane markings, buildings), nor does it make assumptions that require the vehicle to be navigating on a road network. Instead, we achieve localization in both urban and non-urban environments by robustly associating and registering the vehicle's local semantic object map with a compact semantic reference map, potentially built from other viewpoints, time periods, and/or modalities. Robustness to noise, outliers, and missing objects is achieved through our graph-based data association algorithm. Further, the guided relocalization capability of our pipeline mitigates drift inherent in odometry-based localization after the initial global localization. We evaluate our pipeline on two publicly-available, real-world datasets to demonstrate its effectiveness at global localization in both non-urban and urban environments. The Katwijk Beach Planetary Rover dataset [1] is used to show our pipeline's ability to perform accurate global localization in unstructured environments. Demonstrations on the KITTI dataset [2] achieve an average pose error of 3. 8 m across all 35 localization events on Sequence 00 when localizing in a reference map created from aerial images. Compared to existing works, our pipeline is more general because it can perform global localization in unstructured environments using maps built from different viewpoints.

IROS Conference 2023 Conference Paper

MOTLEE: Distributed Mobile Multi-Object Tracking with Localization Error Elimination

• Mason B. Peterson
• Parker C. Lusk
• Jonathan P. How

We present MOTLEE, a distributed mobile multi-object tracking algorithm that enables a team of robots to collaboratively track moving objects in the presence of localization error. Existing approaches to distributed tracking make limiting assumptions regarding the relative spatial relationship of sensors, including assuming a static sensor network or that perfect localization is available. Instead, we develop an algorithm based on the Kalman-Consensus filter for distributed tracking that properly leverages localization uncertainty in collaborative tracking. Further, our method allows the team to maintain an accurate understanding of dynamic objects in the environment by realigning robot frames and incorporating frame alignment uncertainty into our object tracking formulation. We evaluate our method in hardware on a team of three mobile ground robots tracking four people. Compared to previous works that do not account for localization error, we show that MOTLEE is resilient to localization uncertainties, enabling accurate tracking in distributed, dynamic settings with mobile tracking sensors.

IROS Conference 2023 Conference Paper

Probabilistic Traversability Model for Risk-Aware Motion Planning in Off-Road Environments

• Xiaoyi Cai
• Michael Everett
• Lakshay Sharma
• Philip R. Osteen
• Jonathan P. How

A key challenge in off-road navigation is that even visually similar terrains or ones from the same semantic class may have substantially different traction properties. Existing work typically assumes no wheel slip or uses the expected traction for motion planning, where the predicted trajectories provide a poor indication of the actual performance if the terrain traction has high uncertainty. In contrast, this work proposes to analyze terrain traversability with the empirical distribution of traction parameters in unicycle dynamics, which can be learned by a neural network in a self-supervised fashion. The probabilistic traction model leads to two risk-aware cost formulations that account for the worst-case expected cost and traction. To help the learned model generalize to unseen environment, terrains with features that lead to unreliable predictions are detected via a density estimator fit to the trained network's latent space and avoided via auxiliary penalties during planning. Simulation results demonstrate that the proposed approach outperforms existing work that assumes no slip or uses the expected traction in both navigation success rate and completion time. Furthermore, avoiding terrains with low density-based confidence score achieves up to 30% improvement in success rate when the learned traction model is used in a novel environment.

ICRA Conference 2023 Conference Paper

RAMP: A Risk-Aware Mapping and Planning Pipeline for Fast Off-Road Ground Robot Navigation

• Lakshay Sharma
• Michael Everett
• Donggun Lee
• Xiaoyi Cai
• Philip R. Osteen
• Jonathan P. How

A key challenge in fast ground robot navigation in 3D terrain is balancing robot speed and safety. Recent work has shown that 2. 5D maps (2D representations with additional 3D information) are ideal for real-time safe and fast planning. However, the prevalent approach of generating 2D occupancy grids through raytracing makes the generated map unsafe to plan in, due to inaccurate representation of unknown space. Additionally, existing planners such as MPPI do not consider speeds in known free and unknown space separately, leading to slower overall plans. The RAMP pipeline proposed here solves these issues using new mapping and planning methods. This work first presents ground point inflation with persistent spatial memory as a way to generate accurate occupancy grid maps from classified pointclouds. Then we present an MPPI-based planner with embedded variability in horizon, to maximize speed in known free space while retaining cautionary penetration into unknown space. Finally, we integrate this mapping and planning pipeline with risk constraints arising from 3D terrain, and verify that it enables fast and safe navigation using simulations and hardware demonstrations.

IROS Conference 2023 Conference Paper

Resilient and Distributed Multi-Robot Visual SLAM: Datasets, Experiments, and Lessons Learned

• Yulun Tian
• Yun Chang
• Long Quang
• Arthur Schang
• Carlos Nieto-Granda
• Jonathan P. How
• Luca Carlone

This paper revisits Kimera-Multi, a distributed multi-robot Simultaneous Localization and Mapping (SLAM) system, towards the goal of deployment in the real world. In particular, this paper has three main contributions. First, we describe improvements to Kimera-Multi to make it resilient to large-scale real-world deployments, with particular emphasis on handling intermittent and unreliable communication. Second, we collect and release challenging multi-robot benchmarking datasets obtained during live experiments conducted on the MIT campus, with accurate reference trajectories and maps for evaluation. The datasets include up to 8 robots traversing long distances (up to 8 km) and feature many challenging elements such as severe visual ambiguities (e. g. , in underground tunnels and hallways), mixed indoor and outdoor trajectories with different lighting conditions, and dynamic entities (e. g. , pedestrians and cars). Lastly, we evaluate the resilience of Kimera-Multi under different communication scenarios, and provide a quantitative comparison with a centralized baseline system. Based on the results from both live experiments and subsequent analysis, we discuss the strengths and weaknesses of Kimera-Multi, and suggest future directions for both algorithm and system design. We release the source code of Kimera-Multi and all datasets to facilitate further research towards the reliable real-world deployment of multi-robot SLAM systems.

ICRA Conference 2023 Conference Paper

Robust MADER: Decentralized and Asynchronous Multiagent Trajectory Planner Robust to Communication Delay

• Kota Kondo
• Jesus Tordesillas
• Reinaldo Figueroa
• Juan Rached
• Joseph Merkel
• Parker C. Lusk
• Jonathan P. How

Although communication delays can disrupt multiagent systems, most of the existing multiagent trajectory planners lack a strategy to address this issue. State-of-the-art approaches typically assume perfect communication environments, which is hardly realistic in real-world experiments. This paper presents Robust MADER (RMADER), a decentralized and asynchronous multiagent trajectory planner that can handle communication delays among agents. By broadcasting both the newly optimized trajectory and the committed trajectory, and by performing a delay check step, RMADER is able to guarantee safety even under communication delay. RMADER was validated through extensive simulation and hardware flight experiments and achieved a 100% success rate of collision-free trajectory generation, outperforming state-of-the-art approaches.

ICRA Conference 2023 Conference Paper

Robust, High-Rate Trajectory Tracking on Insect-Scale Soft-Actuated Aerial Robots with Deep-Learned Tube MPC

• Andrea Tagliabue
• Yi Hsuan Hsiao
• Urban Fasel
• J. Nathan Kutz
• Steven L. Brunton
• Yufeng Chen 0003
• Jonathan P. How

Accurate and agile trajectory tracking in sub-gram Micro Aerial Vehicles (MAVs) is challenging, as the small scale of the robot induces large model uncertainties, demanding robust feedback controllers, while the fast dynamics and computational constraints prevent the deployment of computationally expensive strategies. In this work, we present an approach for agile and computationally efficient trajectory tracking on the MIT SoftFly [1], a sub-gram MAV (0. 7 grams). Our strategy employs a cascaded control scheme, where an adaptive attitude controller is combined with a neural network (NN) policy trained to imitate a trajectory tracking robust tube model predictive controller (RTMPC). The NN policy is obtained using our recent work [2], which enables the policy to preserve the robustness of RTMPC, but at a fraction of its computational cost. We experimentally evaluate our approach, achieving position Root Mean Square Errors (RMSEs) lower than 1. 8 cm even in the more challenging maneuvers, obtaining a 60% reduction in maximum position error compared to [3], and demonstrating robustness to large external disturbances.

ICRA Conference 2023 Conference Paper

Wide-Area Geolocalization with a Limited Field of View Camera

• Lena M. Downes
• Ted J. Steiner
• Rebecca L. Russell
• Jonathan P. How

Cross-view geolocalization, a supplement or replacement for GPS, localizes an agent within a search area by matching images taken from a ground-view camera to overhead images taken from satellites or aircraft. Although the viewpoint disparity between ground and overhead images makes crossview geolocalization challenging, significant progress has been made assuming that the ground agent has access to a panoramic camera. For example, our prior work (WAG) introduced changes in search area discretization, training loss, and particle filter weighting that enabled city-scale panoramic cross-view geolocalization. However, panoramic cameras are not widely used in existing robotic platforms due to their complexity and cost. Non-panoramic cross-view geolocalization is more applicable for robotics, but is also more challenging. This paper presents Restricted FOV Wide-Area Geolocalization (ReWAG), a cross-view geolocalization approach that generalizes WAG for use with standard, non-panoramic ground cameras by creating pose-aware embeddings and providing a strategy to incorporate particle pose into the Siamese network. ReWAG is a neural network and particle filter system that is able to globally localize a mobile agent in a GPS-denied environment with only odometry and a 90° FOV camera, achieving similar localization accuracy as what WAG achieved with a panoramic camera and improving localization accuracy by a factor of 100 compared to a baseline vision transformer (ViT) approach.

IROS Conference 2022 Conference Paper

City-wide Street-to-Satellite Image Geolocalization of a Mobile Ground Agent

• Lena M. Downes
• Dong-Ki Kim
• Ted J. Steiner
• Jonathan P. How

Cross-view image geolocalization provides an estimate of an agent's global position by matching a local ground image to an overhead satellite image without the need for GPS. It is challenging to reliably match a ground image to the correct satellite image since the images have significant viewpoint differences. Existing works have demonstrated localization in constrained scenarios over small areas but have not demonstrated wider-scale localization. Our approach, called Wide-Area Geolocalization (WAG), combines a neural network with a particle filter to achieve global position estimates for agents moving in GPS-denied environments, scaling efficiently to city-scale regions. WAG introduces a trinomial loss function for a Siamese network to robustly match non-centered image pairs and thus enables the generation of a smaller satellite image database by coarsely discretizing the search area. A modified particle filter weighting scheme is also presented to improve localization accuracy and convergence. Taken together, WAG's network training and particle filter weighting approach achieves city-scale position estimation accuracies on the order of 20 meters, a 98% reduction compared to a baseline training and weighting approach. Applied to a smaller-scale testing area, WAG reduces the final position estimation error by 64% compared to a state-of-the-art baseline from the literature. WAG's search space discretization additionally significantly reduces storage and processing requirements. We include in our submission a video demonstrating particle filter convergence results for WAG compared to the baseline for the Chicago test area.

AAAI Conference 2022 Conference Paper

Context-Specific Representation Abstraction for Deep Option Learning

• Marwa Abdulhai
• Dong-Ki Kim
• Matthew Riemer
• Miao Liu
• Gerald Tesauro
• Jonathan P. How

Hierarchical reinforcement learning has focused on discovering temporally extended actions, such as options, that can provide benefits in problems requiring extensive exploration. One promising approach that learns these options end-to-end is the option-critic (OC) framework. We examine and show in this paper that OC does not decompose a problem into simpler sub-problems, but instead increases the size of the search over policy space with each option considering the entire state space during learning. This issue can result in practical limitations of this method, including sample inefficient learning. To address this problem, we introduce Context-Specific Representation Abstraction for Deep Option Learning (CRADOL), a new framework that considers both temporal abstraction and context-specific representation abstraction to effectively reduce the size of the search over policy space. Specifically, our method learns a factored belief state representation that enables each option to learn a policy over only a subsection of the state space. We test our method against hierarchical, nonhierarchical, and modular recurrent neural network baselines, demonstrating significant sample efficiency improvements in challenging partially observable environments.

ICRA Conference 2022 Conference Paper

Demonstration-Efficient Guided Policy Search via Imitation of Robust Tube MPC

• Andrea Tagliabue
• Dong-Ki Kim
• Michael Everett
• Jonathan P. How

We propose a demonstration-efficient strategy to compress a computationally expensive Model Predictive Controller (MPC) into a more computationally efficient representation based on a deep neural network and Imitation Learning (IL). By generating a Robust Tube variant (RTMPC) of the MPC and leveraging properties from the tube, we introduce a data augmentation method that enables high demonstration-efficiency, capable of compensating the distribution shifts typically encountered in IL. Our approach opens the possibility of zero-shot transfer from a single demonstration collected in a nominal domain, such as a simulation or a robot in a lab/controlled environment, to a domain with bounded model errors/perturbations. Numerical and experimental evaluations performed on a trajectory tracking MPC for a multirotor show that our method outperforms strategies commonly employed in IL, such as DAgger and Domain Randomization, in terms of demonstration-efficiency and robustness to perturbations unseen during training.

IROS Conference 2022 Conference Paper

Distributed Riemannian Optimization with Lazy Communication for Collaborative Geometric Estimation

• Yulun Tian
• Amrit Singh Bedi
• Alec Koppel
• Miguel Calvo-Fullana
• David M. Rosen
• Jonathan P. How

We present the first distributed optimization al-gorithm with lazy communication for collaborative geometric estimation, the backbone of modern collaborative simultaneous localization and mapping (SLAM) and structure-from-motion (SfM) applications. Our method allows agents to cooperatively reconstruct a shared geometric model on a central server by fusing individual observations, but without the need to transmit potentially sensitive information about the agents themselves (such as their locations). Furthermore, to alleviate the burden of communication during iterative optimization, we design a set of communication triggering conditions that enable agents to selectively upload a targeted subset of local information that is useful to global optimization. Our approach thus achieves significant communication reduction with minimal impact on optimization performance. As our main theoretical contribution, we prove that our method converges to first-order critical points with a global sublinear convergence rate. Numerical evaluations on bundle adjustment problems from collaborative SLAM and SfM datasets show that our method performs competitively against existing distributed techniques, while achieving up to 78% total communication reduction.

IROS Conference 2022 Conference Paper

Global Data Association for SLAM with 3D Grassmannian Manifold Objects

• Parker C. Lusk
• Jonathan P. How

Using pole and plane objects in lidar SLAM can increase accuracy and decrease map storage requirements compared to commonly-used point cloud maps. However, place recognition and geometric verification using these landmarks is challenging due to the requirement for global matching without an initial guess. Existing works typically only leverage either pole or plane landmarks, limiting application to a restricted set of environments. We present a global data association method for loop closure in lidar scans using 3D line and plane objects simultaneously and in a unified manner. The main novelty of this paper is in the representation of line and plane objects extracted from Iidar scans on the manifold of affine subspaces, known as the affine Grassmannian. Line and plane correspondences are matched using our graph-based data association framework and subsequently registered in the least-squares sense. Compared to pole-only approaches and plane-only approaches, our 3D affine Grassmannian method yields a 71 % and 325 % increase respectively to loop closure recall at 100 % precision on the KITTI dataset and can provide frame alignment with less than 10 cm and 1 deg of error.

NeurIPS Conference 2022 Conference Paper

Influencing Long-Term Behavior in Multiagent Reinforcement Learning

• Dong-Ki Kim
• Matthew Riemer
• Miao Liu
• Jakob Foerster
• Michael Everett
• Chuangchuang Sun
• Gerald Tesauro
• Jonathan P. How

The main challenge of multiagent reinforcement learning is the difficulty of learning useful policies in the presence of other simultaneously learning agents whose changing behaviors jointly affect the environment's transition and reward dynamics. An effective approach that has recently emerged for addressing this non-stationarity is for each agent to anticipate the learning of other agents and influence the evolution of future policies towards desirable behavior for its own benefit. Unfortunately, previous approaches for achieving this suffer from myopic evaluation, considering only a finite number of policy updates. As such, these methods can only influence transient future policies rather than achieving the promise of scalable equilibrium selection approaches that influence the behavior at convergence. In this paper, we propose a principled framework for considering the limiting policies of other agents as time approaches infinity. Specifically, we develop a new optimization objective that maximizes each agent's average reward by directly accounting for the impact of its behavior on the limiting set of policies that other agents will converge to. Our paper characterizes desirable solution concepts within this problem setting and provides practical approaches for optimizing over possible outcomes. As a result of our farsighted objective, we demonstrate better long-term performance than state-of-the-art baselines across a suite of diverse multiagent benchmark domains.

IROS Conference 2022 Conference Paper

Multi-Agent Relative Pose Estimation with UWB and Constrained Communications

• Andrew Fishberg
• Jonathan P. How

Inter-agent relative localization is critical for any multi-robot system operating in the absence of external positioning infrastructure or prior environmental knowledge. We propose a novel inter-agent relative 2D pose estimation system where each participating agent is equipped with several ultra-wideband (UWB) ranging tags. Prior work typically supplements noisy UWB range measurements with additional continuously transmitted data, such as odometry, making these approaches scale poorly with increased swarm size or decreased communication throughput. This approach addresses these concerns by using only locally collected UWB measurements with no additionally transmitted data. By modeling observed ranging biases and systematic antenna obstructions in our proposed optimization solution, our experimental results demonstrate an improved mean position error (while remaining competitive in other metrics) over a similar state-of-the-art approach that additionally relies on continuously transmitted odometry.

IROS Conference 2022 Conference Paper

Output Feedback Tube MPC-Guided Data Augmentation for Robust, Efficient Sensorimotor Policy Learning

• Andrea Tagliabue
• Jonathan P. How

Imitation learning (IL) can generate computationally efficient sensorimotor policies from demonstrations provided by computationally expensive model-based sensing and control algorithms. However, commonly employed IL methods are often data-inefficient, requiring the collection of a large number of demonstrations and producing policies with limited robustness to uncertainties. In this work, we combine IL with an output feedback robust tube model predictive controller (RTMPC) to co-generate demonstrations and a data augmentation strategy to efficiently learn neural network-based sensorimotor policies. Thanks to the augmented data, we reduce the computation time and the number of demonstrations needed by IL, while providing robustness to sensing and process uncertainty. We tailor our approach to the task of learning a trajectory tracking visuomotor policy for an aerial robot, leveraging a 3D mesh of the environment as part of the data augmentation process. We numerically demonstrate that our method can learn a robust visuomotor policy from a single demonstration—a two-orders of magnitude improvement in demonstration efficiency compared to existing IL methods.

IROS Conference 2022 Conference Paper

Risk-Aware Off-Road Navigation via a Learned Speed Distribution Map

• Xiaoyi Cai
• Michael Everett
• Jonathan Fink
• Jonathan P. How

Motion planning in off-road environments re-quires reasoning about both the geometry and semantics of the scene (e. g. , a robot may be able to drive through soft bushes but not a fallen log). In many recent works, the world is classified into a finite number of semantic categories that often are not sufficient to capture the ability (i. e. , the speed) with which a robot can traverse off-road terrain. Instead, this work proposes a new representation of traversability based exclusively on robot speed that can be learned from data, offers interpretability and intuitive tuning, and can be easily integrated with a variety of planning paradigms in the form of a costmap. Specifically, given a dataset of experienced trajectories, the proposed algorithm learns to predict a distribution of speeds the robot could achieve, conditioned on the environment semantics and commanded speed. The learned speed distribution map is converted into costmaps with a risk-aware cost term based on conditional value at risk (CVaR). Numerical simulations demonstrate that the proposed risk-aware planning algorithm leads to faster average time-to-goals compared to a method that only considers expected behavior, and the planner can be tuned for slightly slower, but less variable behavior. Furthermore, the approach is integrated into a full autonomy stack and demonstrated in a high-fidelity Unity environment and is shown to provide a 30% improvement in the success rate of navigation.

ICRA Conference 2022 Conference Paper

ROMAX: Certifiably Robust Deep Multiagent Reinforcement Learning via Convex Relaxation

• Chuangchuang Sun
• Dong-Ki Kim
• Jonathan P. How

In a multirobot system, a number of cyber-physical attacks (e. g. , communication hijack, observation per-turbations) can challenge the robustness of agents. This robust-ness issue worsens in multiagent reinforcement learning because there exists the non-stationarity of the environment caused by simultaneously learning agents whose changing policies affect the transition and reward functions. In this paper, we propose a minimax MARL approach to infer the worst-case policy update of other agents. As the minimax formulation is computationally intractable to solve, we apply the convex relaxation of neural networks to solve the inner minimization problem. Such convex relaxation enables robustness in interacting with peer agents that may have significantly different behaviors and also achieves a certified bound of the original optimization problem. We eval-uate our approach on multiple mixed cooperative-competitive tasks and show that our method outperforms the previous state of the art approaches on this topic.

IROS Conference 2022 Conference Paper

Safe adaptation in multiagent competition

• Macheng Shen
• Jonathan P. How

Achieving the capability of adapting to ever-changing environments is a critical step towards building fully autonomous robots that operate safely in complicated scenarios. In multiagent competitive scenarios, agents may have to adapt to new opponents with previously unseen behaviors by learning from the interaction experiences between the ego-agent and the opponent. However, this adaptation is susceptible to opponent exploitation. As the ego-agent updates its own behavior to exploit the opponent, its own behavior could become more exploitable as a result of overfitting to this specific opponent's behavior. To overcome this difficulty, we developed a safe adaptation approach in which the ego-agent is trained against a regularized opponent model, which effectively avoids overfitting and consequently improves the robustness of the ego-agent's policy. We evaluated our approach in the Mujoco domain with two competing agents. The experiment results suggest that our approach effectively achieves both adaptation to the specific opponent that the ego-agent is interacting with and maintaining low exploitability to other possible opponent exploitation.

ICML Conference 2021 Conference Paper

A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning

• Dong-Ki Kim
• Miao Liu 0001
• Matthew Riemer
• Chuangchuang Sun
• Marwa Abdulhai
• Golnaz Habibi
• Sebastian Lopez-Cot
• Gerald Tesauro

A fundamental challenge in multiagent reinforcement learning is to learn beneficial behaviors in a shared environment with other simultaneously learning agents. In particular, each agent perceives the environment as effectively non-stationary due to the changing policies of other agents. Moreover, each agent is itself constantly learning, leading to natural non-stationarity in the distribution of experiences encountered. In this paper, we propose a novel meta-multiagent policy gradient theorem that directly accounts for the non-stationary policy dynamics inherent to multiagent learning settings. This is achieved by modeling our gradient updates to consider both an agent’s own non-stationary policy dynamics and the non-stationary policy dynamics of other agents in the environment. We show that our theoretically grounded approach provides a general solution to the multiagent learning problem, which inherently comprises all key aspects of previous state of the art approaches on this topic. We test our method on a diverse suite of multiagent benchmarks and demonstrate a more efficient ability to adapt to new agents as they learn than baseline methods across the full spectrum of mixed incentive, competitive, and cooperative domains.

ICRA Conference 2021 Conference Paper

Airflow-Inertial Odometry for Resilient State Estimation on Multirotors

• Andrea Tagliabue
• Jonathan P. How

We present a dead reckoning strategy for increased resilience to position estimation failures on multirotors, using only data from a low-cost IMU and novel, bio-inspired airflow sensors. The goal is challenging, since low-cost IMUs are subject to large noise and drift, while 3D airflow sensing is made difficult by the interference caused by the propellers and by the wind. Our approach relies on a deep-learning strategy to interpret the measurements of the bio-inspired sensors, a map of the wind speed to compensate for position-dependent wind, and a filter to fuse the information and generate a pose and velocity estimate. Our results show that the approach reduces the drift with respect to IMU-only dead reckoning by up to an order of magnitude over 30 seconds after a position sensor failure in non-windy environments, and it can compensate for the challenging effects of turbulent, and spatially varying wind.

ICRA Conference 2021 Conference Paper

CLIPPER: A Graph-Theoretic Framework for Robust Data Association

• Parker C. Lusk
• Kaveh Fathian
• Jonathan P. How

We present CLIPPER (Consistent LInking, Pruning, and Pairwise Error Rectification), a framework for robust data association in the presence of noise and outliers. We formulate the problem in a graph-theoretic framework using the notion of geometric consistency. State-of-the-art techniques that use this framework utilize either combinatorial optimization techniques that do not scale well to large-sized problems, or use heuristic approximations that yield low accuracy in high-noise, high-outlier regimes. In contrast, CLIPPER uses a relaxation of the combinatorial problem and returns solutions that are guaranteed to correspond to the optima of the original problem. Low time complexity is achieved with an efficient projected gradient ascent approach. Experiments indicate that CLIPPER maintains a consistently low runtime of 15 ms where exact methods can require up to 24 s at their peak, even on small-sized problems with 200 associations. When evaluated on noisy point cloud registration problems, CLIPPER achieves 100% precision and 98% recall in 90% outlier regimes while competing algorithms begin degrading by 70% outliers. In an instance of associating noisy points of the Stanford Bunny with 990 outlier associations and only 10 inlier associations, CLIPPER successfully returns 8 inlier associations with 100% precision in 138 ms. Code is available at https://mit-acl.github.io/clipper.

ICRA Conference 2021 Conference Paper

Efficient Reachability Analysis of Closed-Loop Systems with Neural Network Controllers

• Michael Everett
• Golnaz Habibi
• Jonathan P. How

Neural Networks (NNs) can provide major empirical performance improvements for robotic systems, but they also introduce challenges in formally analyzing those systems’ safety properties. In particular, this work focuses on estimating the forward reachable set of closed-loop systems with NN controllers. Recent work provides bounds on these reachable sets, yet the computationally efficient approaches provide overly conservative bounds (thus cannot be used to verify useful properties), whereas tighter methods are too intensive for online computation. This work bridges the gap by formulating a convex optimization problem for reachability analysis for closed-loop systems with NN controllers. While the solutions are less tight than prior semidefinite program-based methods, they are substantially faster to compute, and some of the available computation time can be used to refine the bounds through input set partitioning, which more than overcomes the tightness gap. The proposed framework further considers systems with measurement and process noise, thus being applicable to realistic systems with uncertainty. Finally, numerical comparisons show that our approach based on linear programming and partitioning can give 10× reduction in conservatism in $\frac{1}{2}$ of the computation time compared to the state-of-the-art, and the ability to handle various sources of uncertainty is highlighted on a quadrotor model.

ICRA Conference 2021 Conference Paper

FISAR: Forward Invariant Safe Reinforcement Learning with a Deep Neural Network-Based Optimizer

• Chuangchuang Sun
• Dong-Ki Kim
• Jonathan P. How

This paper investigates reinforcement learning with constraints, which are indispensable in safety-critical environments. To drive the constraint violation to decrease monotonically, we take the constraints as Lyapunov functions and impose new linear constraints on the policy parameters’ updating dynamics. As a result, the original safety set can be forward-invariant. However, because the new guaranteed-feasible constraints are imposed on the updating dynamics instead of the original policy parameters, classic optimization algorithms are no longer applicable. To address this, we propose to learn a generic deep neural network (DNN)-based optimizer to optimize the objective while satisfying the linear constraints. The constraint-satisfaction is achieved via projection onto a polytope formulated by multiple linear inequality constraints, which can be solved analytically with our newly designed metric. To the best of our knowledge, this is the first DNN-based optimizer for constrained optimization with the forward invariance guarantee. We show that our optimizer trains a policy to decrease the constraint violation and maximize the cumulative reward monotonically. Results on numerical constrained optimization and obstacle-avoidance navigation validate the theoretical findings.

ICRA Conference 2021 Conference Paper

Kimera-Multi: a System for Distributed Multi-Robot Metric-Semantic Simultaneous Localization and Mapping

• Yun Chang
• Yulun Tian
• Jonathan P. How
• Luca Carlone

We present the first fully distributed multi-robot system for dense metric-semantic Simultaneous Localization and Mapping (SLAM). Our system, dubbed Kimera-Multi, is implemented by a team of robots equipped with visual-inertial sensors, and builds a 3D mesh model of the environment in real-time, where each face of the mesh is annotated with a semantic label (e. g. , building, road, objects). In Kimera-Multi, each robot builds a local trajectory estimate and a local mesh using Kimera [1]. Then, when two robots are within communication range, they initiate a distributed place recognition and robust pose graph optimization protocol with a novel incremental maximum clique outlier rejection; the protocol allows the robots to improve their local trajectory estimates by leveraging inter-robot loop closures. Finally, each robot uses its improved trajectory estimate to correct the local mesh using mesh deformation techniques. We demonstrate Kimera-Multi in photo-realistic simulations and real data. Kimera-Multi (i) is able to build accurate 3D metric-semantic meshes, (ii) is robust to incorrect loop closures while requiring less computation than state-of-the-art distributed SLAM backends, and (iii) is efficient, both in terms of computation at each robot as well as communication bandwidth.

ICRA Conference 2021 Conference Paper

Multi-Robot Distributed Semantic Mapping in Unfamiliar Environments through Online Matching of Learned Representations

• Stewart Jamieson
• Kaveh Fathian
• Kasra Khosoussi
• Jonathan P. How
• Yogesh A. Girdhar

We present a solution to multi-robot distributed semantic mapping of novel and unfamiliar environments. Most state-of-the-art semantic mapping systems are based on supervised learning algorithms that cannot classify novel observations online. While unsupervised learning algorithms can invent labels for novel observations, approaches to detect when multiple robots have independently developed their own labels for the same new class are prone to erroneous or inconsistent matches. These issues worsen as the number of robots in the system increases and prevent fusing the local maps produced by each robot into a consistent global map, which is crucial for cooperative planning and joint mission summarization. Our proposed solution overcomes these obstacles by having each robot learn an unsupervised semantic scene model online and use a multiway matching algorithm to identify consistent sets of matches between learned semantic labels belonging to different robots. Compared to the state of the art, the proposed solution produces 20-60% higher quality global maps that do not degrade even as many more local maps are fused.

ICRA Conference 2021 Conference Paper

NF-iSAM: Incremental Smoothing and Mapping via Normalizing Flows

• Qiangqiang Huang
• Can Pu
• Dehann Fourie
• Kasra Khosoussi
• Jonathan P. How
• John J. Leonard

This paper presents a novel non-Gaussian inference algorithm, Normalizing Flow iSAM (NF-iSAM), for solving SLAM problems with non-Gaussian factors and/or non-linear measurement models. NF-iSAM exploits the expressive power of neural networks, and trains normalizing flows to draw samples from the joint posterior of non-Gaussian factor graphs. By leveraging the Bayes tree, NF-iSAM is able to exploit the sparsity structure of SLAM, thus enabling efficient incremental updates similar to iSAM2, albeit in the more challenging non- Gaussian setting. We demonstrate the performance of NF-iSAM and compare it against the state-of-the-art algorithms such as iSAM2 (Gaussian) and mm-iSAM (non-Gaussian) in synthetic and real range-only SLAM datasets.

ICRA Conference 2021 Conference Paper

Non-Monotone Energy-Aware Information Gathering for Heterogeneous Robot Teams

• Xiaoyi Cai
• Brent Schlotfeldt
• Kasra Khosoussi
• Nikolay Atanasov 0001
• George J. Pappas
• Jonathan P. How

This paper considers the problem of planning trajectories for a team of sensor-equipped robots to reduce uncertainty about a dynamical process. Optimizing the trade-off between information gain and energy cost (e. g. , control effort, distance travelled) is desirable but leads to a non-monotone objective function in the set of robot trajectories. Therefore, common multi-robot planning algorithms based on techniques such as coordinate descent lose their performance guarantees. Methods based on local search provide performance guarantees for optimizing a non-monotone submodular function, but require access to all robots’ trajectories, making it not suitable for distributed execution. This work proposes a distributed planning approach based on local search and shows how lazy/greedy methods can be adopted to reduce the computation and communication of the approach. We demonstrate the efficacy of the proposed method by coordinating robot teams composed of both ground and aerial vehicles with different sensing/control profiles and evaluate the algorithm’s performance in two target tracking scenarios. Compared to the naive distributed execution of local search, our approach saves up to 60% communication and 80–92% computation on average when coordinating up to 10 robots, while outperforming the coordinate descent based algorithm in achieving a desirable trade-off between sensing and energy cost.

ICAPS Conference 2021 Conference Paper

Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning

• Macheng Shen
• Jonathan P. How

This paper studies decision-making in two-player scenarios where the type (e. g. adversary, neutral, or teammate) of the other agent (opponent) is uncertain to the decision-making agent (protagonist), which is an abstraction of security-domain applications. In these settings, the reward for the protagonist agent depends on the type of the opponent, but this is private information known only to the opponent itself, and thus hidden from the protagonist. In contrast, as is often the case, the type of the protagonist agent is assumed to be known to the opponent, and this information-asymmetry significantly complicates the protagonist's decision-making. In particular, to determine the best actions to take, the protagonist agent must infer the opponent type from the observations and agent modeling. To address this problem, this paper presents an opponent-type deduction module based on Bayes' rule. This inference module takes as input the imagined opponent's decision-making rule (opponent model) as well as the observed opponent's history of actions and states, and outputs a belief over the opponent's hidden type. A multiagent reinforcement learning approach is used to develop this game-theoretic opponent model through self-play, which avoids the expensive data collection step that requires interaction with a real opponent. Besides, this multiagent approach also captures the strategy interaction and reasoning between agents. In addition, we apply ensemble training to avoid over-fitting to a single opponent model during the training. As a result, the learned protagonist policy is also effective against unseen opponents. Experimental results show that the proposed game-theoretic modeling, explicit opponent type inference and the ensemble training significantly improves the decision-making performance over baseline approaches, and generalizes well against adversaries that have not been seen during the training.

IROS Conference 2020 Conference Paper

A Whisker-inspired Fin Sensor for Multi-directional Airflow Sensing

• Suhan Kim
• Regan Kubicek
• Aleix Paris
• Andrea Tagliabue
• Jonathan P. How
• Sarah Bergbreiter

This work presents the design, fabrication, and characterization of an airflow sensor inspired by the whiskers of animals. The body of the whisker was replaced with a fin structure in order to increase the air resistance. The fin was suspended by a micro-fabricated spring system at the bottom. A permanent magnet was attached beneath the spring, and the motion of fin was captured by a readily accessible and low- cost 3D magnetic sensor located below the magnet. The sensor system was modeled in terms of the dimension parameters of fin and the spring stiffness, which were optimized to improve the performance of the sensor. The system response was then characterized using a commercial wind tunnel and the results were used for sensor calibration. The sensor was integrated into a micro aerial vehicle (MAV) and demonstrated the capability of capturing the velocity of the MAV by sensing the relative airflow during flight.

ICRA Conference 2020 Conference Paper

Active Reward Learning for Co-Robotic Vision Based Exploration in Bandwidth Limited Environments

• Stewart Jamieson
• Jonathan P. How
• Yogesh A. Girdhar

We present a novel POMDP problem formulation for a robot that must autonomously decide where to go to collect new and scientifically relevant images given a limited ability to communicate with its human operator. From this formulation we derive constraints and design principles for the observation model, reward model, and communication strategy of such a robot, exploring techniques to deal with the very high-dimensional observation space and scarcity of relevant training data. We introduce a novel active reward learning strategy based on making queries to help the robot minimize path "regret" online, and evaluate it for suitability in autonomous visual exploration through simulations. We demonstrate that, in some bandwidth-limited environments, this novel regret-based criterion enables the robotic explorer to collect up to 17% more reward per mission than the next-best criterion.

JAAMAS Journal 2020 Journal Article

Crossmodal attentive skill learner: learning in Atari and beyond with audio–video inputs

• Dong-Ki Kim
• Shayegan Omidshafiei
• Jonathan P. How

Abstract This paper introduces the Crossmodal Attentive Skill Learner (CASL), integrated with the recently-introduced Asynchronous Advantage Option-Critic architecture [Harb et al. in When waiting is not an option: learning options with a deliberation cost. arXiv preprint arXiv: 1709. 04571, 2017] to enable hierarchical reinforcement learning across multiple sensory inputs. Agents trained using our approach learn to attend to their various sensory modalities (e. g. , audio, video) at the appropriate moments, thereby executing actions based on multiple sensory streams without reliance on supervisory data. We demonstrate empirically that the sensory attention mechanism anticipates and identifies useful latent features, while filtering irrelevant sensor modalities during execution. Further, we provide concrete examples in which the approach not only improves performance in a single task, but accelerates transfer to new tasks. We modify the Arcade Learning Environment [Bellemare et al. in J Artif Intell Res 47: 253–279, 2013] to support audio queries (ALE-audio code available at https: //github. com/shayegano/Arcade-Learning-Environment ), and conduct evaluations of crossmodal learning in the Atari 2600 games H. E. R. O. and Amidar. Finally, building on the recent work of Babaeizadeh et al. [in: International conference on learning representations (ICLR), 2017], we open-source a fast hybrid CPU–GPU implementation of CASL (CASL code available at https: //github. com/shayegano/CASL ).

ICRA Conference 2020 Conference Paper

Dynamic Landing of an Autonomous Quadrotor on a Moving Platform in Turbulent Wind Conditions

• Aleix Paris
• Brett T. Lopez
• Jonathan P. How

Autonomous landing on a moving platform presents unique challenges for multirotor vehicles, including the need to accurately localize the platform, fast trajectory planning, and precise/robust control. Previous works studied this problem but most lack explicit consideration of the wind disturbance, which typically leads to slow descents onto the platform. This work presents a fully autonomous vision-based system that addresses these limitations by tightly coupling the localization, planning, and control, thereby enabling fast and accurate landing on a moving platform. The platform's position, orientation, and velocity are estimated by an extended Kalman filter using simulated GPS measurements when the quadrotor-platform distance is large, and by a visual fiducial system when the platform is nearby. The landing trajectory is computed online using receding horizon control and is followed by a boundary layer sliding controller that provides tracking performance guarantees in the presence of unknown, but bounded, disturbances. To improve the performance, the characteristics of the turbulent conditions are accounted for in the controller. The landing trajectory is fast, direct, and does not require hovering over the platform, as is typical of most stateof-the-art approaches. Simulations and hardware experiments are presented to validate the robustness of the approach.

ICRA Conference 2020 Conference Paper

Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning

• Arpan Kusari
• Jonathan P. How

A common approach for defining a reward function for multi-objective reinforcement learning (MORL) problems is the weighted sum of the multiple objectives. The weights are then treated as design parameters dependent on the expertise (and preference) of the person performing the learning, with the typical result that a new solution is required for any change in these settings. This paper investigates the relationship between the reward function and the optimal value function for MORL; specifically addressing the question of how to approximate the optimal value function well beyond the set of weights for which the optimization problem was actually solved, thereby avoiding the need to recompute for any particular choice. We prove that the value function transforms smoothly given a transformation of weights of the reward function (and thus a smooth interpolation in the policy space). A Gaussian process is used to obtain a smooth interpolation over the reward function weights of the optimal value function for three well-known examples: Gridworld, Objectworld and Pendulum. The results show that the interpolation can provide robust values for sample states and actions in both discrete and continuous domain problems. Significant advantages arise from utilizing this interpolation technique in the domain of autonomous vehicles: easy, instant adaptation of user preferences while driving and true randomization of obstacle vehicle behavior preferences during training.

IROS Conference 2020 Conference Paper

Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph

• Chuangchuang Sun
• Macheng Shen
• Jonathan P. How

The complexity of multiagent reinforcement learning (MARL) in multiagent systems increases exponentially with respect to the agent number. This scalability issue prevents MARL from being applied in large-scale multiagent systems. However, one critical feature in MARL that is often neglected is that the interactions between agents are quite sparse. Without exploiting this sparsity structure, existing works aggregate information from all of the agents and thus have a high sample complexity. To address this issue, we propose an adaptive sparse attention mechanism by generalizing a sparsity-inducing activation function. Then a sparse communication graph in MARL is learned by graph neural networks based on this new attention mechanism. Through this sparsity structure, the agents can communicate in an effective as well as efficient way via only selectively attending to agents that matter the most and thus the scale of the MARL problem is reduced with little optimality compromised. Comparative results show that our algorithm can learn an interpretable sparse structure and outperforms previous works by a significant margin on applications involving a large-scale multiagent system.

IROS Conference 2020 Conference Paper

Touch the Wind: Simultaneous Airflow, Drag and Interaction Sensing on a Multirotor

• Andrea Tagliabue
• Aleix Paris
• Suhan Kim
• Regan Kubicek
• Sarah Bergbreiter
• Jonathan P. How

Disturbance estimation for Micro Aerial Vehicles (MAVs) is crucial for robustness and safety. In this paper, we use novel, bio-inspired airflow sensors to measure the airflow acting on a MAV, and we fuse this information in an Unscented Kalman filter (UKF) to simultaneously estimate the three-dimensional wind vector, the drag force, and other interaction forces (e. g. due to collisions, interaction with a human) acting on the robot. To this end, we present and compare a fully model-based and a deep learning-based strategy. The model-based approach considers the MAV and airflow sensor dynamics and its interaction with the wind, while the deep learning-based strategy uses a Long Short-Term Memory (LSTM) to obtain an estimate of the relative airflow, which is then fused in the proposed filter. We validate our methods in hardware experiments, showing that we can accurately estimate relative airflow of up to 4 m/s, and we can differentiate drag and interaction force.

ICRA Conference 2019 Conference Paper

Active Perception in Adversarial Scenarios using Maximum Entropy Deep Reinforcement Learning

• Macheng Shen
• Jonathan P. How

We pose an active perception problem where an autonomous agent actively interacts with a second agent with potentially adversarial behaviors. Given the uncertainty in the intent of the other agent, the objective is to collect further evidence to help discriminate potential threats. The main technical challenges are the partial observability of the agent intent, the adversary modeling, and the corresponding uncertainty modeling. Note that an adversary agent may act to mislead the autonomous agent by using a deceptive strategy that is learned from past experiences. We propose an approach that combines belief space planning, generative adversary modeling, and maximum entropy reinforcement learning to obtain a stochastic belief space policy. By accounting for various adversarial behaviors in the simulation framework and minimizing the predictability of the autonomous agent's action, the resulting policy is more robust to unmodeled adversarial strategies. This improved robustness is empirically shown against an adversary that adapts to and exploits the autonomous agent's policy when compared with a standard Chance-Constraint Partially Observable Markov Decision Process robust approach.

ICRA Conference 2019 Conference Paper

Efficient Constellation-Based Map-Merging for Semantic SLAM

• Kristoffer M. Frey
• Ted J. Steiner
• Jonathan P. How

Data association in SLAM is fundamentally challenging, and handling ambiguity well is crucial to achieve robust operation in real-world environments. When ambiguous measurements arise, conservatism often mandates that the measurement is discarded or a new landmark is initialized rather than risking an incorrect association. To address the inevitable “duplicate” landmarks that arise, we present an efficient map-merging framework to detect duplicate constellations of landmarks, providing a high-confidence loop-closure mechanism well-suited for object-level SLAM. This approach uses an incrementally-computable approximation of landmark uncertainty that only depends on local information in the SLAM graph, avoiding expensive recovery of the full system covariance matrix. This enables a search based on geometric consistency (GC) (rather than full joint compatibility (JC)) that inexpensively reduces the search space to a handful of “best” hypotheses. Furthermore, we reformulate the commonly-used interpretation tree to allow for more efficient integration of clique-based pairwise compatibility, accelerating the branch-and-bound max-cardinality search. Our method is demonstrated to match the performance of full JC methods at significantly-reduced computational cost, facilitating robust object-based loop-closure over large SLAM problems.

IROS Conference 2019 Conference Paper

FASTER: Fast and Safe Trajectory Planner for Flights in Unknown Environments

• Jesus Tordesillas
• Brett T. Lopez
• Jonathan P. How

High-speed trajectory planning through unknown environments requires algorithmic techniques that enable fast reaction times while maintaining safety as new information about the operating environment is obtained. The requirement of computational tractability typically leads to optimization problems that do not include the obstacle constraints (collision checks are done on the solutions) or use a convex decomposition of the free space and then impose an ad-hoc time allocation scheme for each interval of the trajectory. Moreover, safety guarantees are usually obtained by having a local planner that plans a trajectory with a final “stop” condition in the freeknown space. However, these two decisions typically lead to slow and conservative trajectories. We propose FASTER (Fast and Safe Trajectory Planner) to overcome these issues. FASTER obtains high-speed trajectories by enabling the local planner to optimize in both the free-known and unknown spaces. Safety guarantees are ensured by always having a feasible, safe back-up trajectory in the free-known space at the start of each replanning step. Furthermore, we present a Mixed Integer Quadratic Program formulation in which the solver can choose the trajectory interval allocation, and where a time allocation heuristic is computed efficiently using the result of the previous replanning iteration. This proposed algorithm is tested extensively both in simulation and in real hardware, showing agile flights in unknown cluttered environments with velocities up to 3. 6 $\mathrm{m}/\mathrm{s}$.

AAAI Conference 2019 Conference Paper

Learning to Teach in Cooperative Multiagent Reinforcement Learning

• Shayegan Omidshafiei
• Dong-Ki Kim
• Miao Liu
• Gerald Tesauro
• Matthew Riemer
• Christopher Amato
• Murray Campbell
• Jonathan P. How

Collective human knowledge has clearly benefited from the fact that innovations by individuals are taught to others through communication. Similar to human social groups, agents in distributed learning systems would likely benefit from communication to share knowledge and teach skills. The problem of teaching to improve agent learning has been investigated by prior works, but these approaches make assumptions that prevent application of teaching to general multiagent problems, or require domain expertise for problems they can apply to. This learning to teach problem has inherent complexities related to measuring long-term impacts of teaching that compound the standard multiagent coordination challenges. In contrast to existing works, this paper presents the first general framework and algorithm for intelligent agents to learn to teach in a multiagent environment. Our algorithm, Learning to Coordinate and Teach Reinforcement (LeCTR), addresses peer-to-peer teaching in cooperative multiagent reinforcement learning. Each agent in our approach learns both when and what to advise, then uses the received advice to improve local learning. Importantly, these roles are not fixed; these agents learn to assume the role of student and/or teacher at the appropriate moments, requesting and providing advice in order to improve teamwide performance and learning. Empirical comparisons against state-of-the-art teaching methods show that our teaching agents not only learn significantly faster, but also learn to coordinate in tasks where existing methods fail.

JAIR Journal 2019 Journal Article

Modeling and Planning with Macro-Actions in Decentralized POMDPs

• Christopher Amato
• George Konidaris
• Leslie P. Kaelbling
• Jonathan P. How

Decentralized partially observable Markov decision processes (Dec-POMDPs) are general models for decentralized multi-agent decision making under uncertainty. However, they typically model a problem at a low level of granularity, where each agent's actions are primitive operations lasting exactly one time step. We address the case where each agent has macro-actions: temporally extended actions that may require different amounts of time to execute. We model macro-actions as options in a Dec-POMDP, focusing on actions that depend only on information directly available to the agent during execution. Therefore, we model systems where coordination decisions only occur at the level of deciding which macro-actions to execute. The core technical difficulty in this setting is that the options chosen by each agent no longer terminate at the same time. We extend three leading Dec-POMDP algorithms for policy generation to the macro-action case, and demonstrate their effectiveness in both standard benchmarks and a multi-robot coordination problem. The results show that our new algorithms retain agent coordination while allowing high-quality solutions to be generated for significantly longer horizons and larger state-spaces than previous Dec-POMDP methods. Furthermore, in the multi-robot domain, we show that, in contrast to most existing methods that are specialized to a particular problem class, our approach can synthesize control policies that exploit opportunities for coordination while balancing uncertainty, sensor information, and information about other agents.

IROS Conference 2019 Conference Paper

Planning Beyond The Sensing Horizon Using a Learned Context

• Michael Everett
• Justin Miller
• Jonathan P. How

Last-mile delivery systems commonly propose the use of autonomous robotic vehicles to increase scalability and efficiency. The economic inefficiency of collecting accurate prior maps for navigation motivates the use of planning algorithms that operate in unmapped environments. However, these algorithms typically waste time exploring regions that are unlikely to contain the delivery destination. Context is key information about structured environments that could guide exploration toward the unknown goal location, but the abstract idea is difficult to quantify for use in a planning algorithm. Some approaches specifically consider contextual relationships between objects, but would perform poorly in object-sparse environments like outdoors. Recent deep learningbased approaches consider context too generally, making training/transferability difficult. Therefore, this work proposes a novel formulation of utilizing context for planning as an image-to-image translation problem, which is shown to extract terrain context from semantic gridmaps, into a metric that an exploration-based planner can use. The proposed framework has the benefit of training on a static dataset instead of requiring a time-consuming simulator. Across 42 test houses with layouts from satellite images, the trained algorithm enables a robot to reach its goal 189% faster than with a context-unaware planner, and within 63% of the optimal path computed with a prior map. The proposed algorithm is also implemented on a vehicle with a forward-facing camera in a high-fidelity, Unreal simulation of neighborhood houses.

IROS Conference 2019 Conference Paper

Policy Distillation and Value Matching in Multiagent Reinforcement Learning

• Samir Wadhwania
• Dong-Ki Kim
• Shayegan Omidshafiei
• Jonathan P. How

Multiagent reinforcement learning (MARL) algorithms have been demonstrated on complex tasks that require the coordination of a team of multiple agents to complete. Existing works have focused on sharing information between agents via centralized critics to stabilize learning or through communication to improve performance, but do not generally consider how information can be shared between agents to address the curse of dimensionality in MARL. We posit that a multiagent problem can be decomposed into a multi-task problem where each agent explores a subset of the state space instead of exploring the entire state space. This paper introduces a multiagent actor-critic algorithm for combining knowledge from homogeneous agents through distillation and value-matching that outperforms policy distillation alone and allows further learning in discrete and continuous action spaces.

ICRA Conference 2019 Conference Paper

Real-Time Planning with Multi-Fidelity Models for Agile Flights in Unknown Environments

• Jesus Tordesillas
• Brett T. Lopez
• John Carter
• John Ware
• Jonathan P. How

Autonomous navigation through unknown environments is a challenging task that entails real-time localization, perception, planning, and control. UAVs with this capability have begun to emerge in the literature with advances in lightweight sensing and computing. Although the planning methodologies vary from platform to platform, many algorithms adopt a hierarchical planning architecture where a slow, low-fidelity global planner guides a fast, high-fidelity local planner. However, in unknown environments, this approach can lead to erratic or unstable behavior due to the interaction between the global planner, whose solution is changing constantly, and the local planner; a consequence of not capturing higher-order dynamics in the global plan. This work proposes a planning framework in which multi-fidelity models are used to reduce the discrepancy between the local and global planner. Our approach uses high-, medium-, and low-fidelity models to compose a path that captures higher-order dynamics while remaining computationally tractable. In addition, we address the interaction between a fast planner and a slower mapper by considering the sensor data not yet fused into the map during the collision check. This novel mapping and planning framework for agile flights is validated in simulation and hardware experiments, showing replanning times of 5-40 ms in cluttered environments.

ICRA Conference 2019 Conference Paper

Robust Object-based SLAM for High-speed Autonomous Navigation

• Kyel Ok
• Katherine Liu
• Kristoffer M. Frey
• Jonathan P. How
• Nicholas Roy

We present Robust Object-based SLAM for High-speed Autonomous Navigation (ROSHAN), a novel approach to object-level mapping suitable for autonomous navigation. In ROSHAN, we represent objects as ellipsoids and infer their parameters using three sources of information - bounding box detections, image texture, and semantic knowledge - to overcome the observability problem in ellipsoid-based SLAM under common forward-translating vehicle motions. Each bounding box provides four planar constraints on an object surface and we add a fifth planar constraint using the texture on the objects along with a semantic prior on the shape of ellipsoids. We demonstrate ROSHAN in simulation where we outperform the baseline, reducing the median shape error by 83% and the median position error by 72% in a forward-moving camera sequence. We demonstrate similar qualitative result on data collected on a fast-moving autonomous quadrotor.

ICRA Conference 2019 Conference Paper

Safe Reinforcement Learning With Model Uncertainty Estimates

• Björn Lütjens
• Michael Everett
• Jonathan P. How

Many current autonomous systems are being designed with a strong reliance on black box predictions from deep neural networks (DNNs). However, DNNs tend to be overconfident in predictions on unseen data and can give unpredictable results for far-from-distribution test data. The importance of predictions that are robust to this distributional shift is evident for safety-critical applications, such as collision avoidance around pedestrians. Measures of model uncertainty can be used to identify unseen data, but the state-of-the-art extraction methods such as Bayesian neural networks are mostly intractable to compute. This paper uses MC-Dropout and Bootstrapping to give computationally tractable and parallelizable uncertainty estimates. The methods are embedded in a Safe Reinforcement Learning framework to form uncertainty-aware navigation around pedestrians. The result is a collision avoidance policy that knows what it does not know and cautiously avoids pedestrians that exhibit unseen behavior. The policy is demonstrated in simulation to be more robust to novel observations and take safer actions than an uncertainty-unaware baseline.

ICRA Conference 2018 Conference Paper

Complexity Analysis and Efficient Measurement Selection Primitives for High-Rate Graph SLAM

• Kristoffer M. Frey
• Ted J. Steiner
• Jonathan P. How

Sparsity has been widely recognized as crucial for efficient optimization in graph-based SLAM. Because the sparsity and structure of the SLAM graph reflect the set of incorporated measurements, many methods for sparsification have been proposed in hopes of reducing computation. These methods often focus narrowly on reducing edge count without regard for structure at a global level. Such structurally-naïve techniques can fail to produce significant computational savings, even after aggressive pruning. In contrast, simple heuristics such as measurement decimation and keyframing are known empirically to produce significant computation reductions. To demonstrate why, we propose a quantitative metric called elimination complexity (EC) that bridges the existing analytic gap between graph structure and computation. EC quantifies the complexity of the primary computational bottleneck: the factorization step of a Gauss-Newton iteration. Using this metric, we show rigorously that decimation and keyframing impose favorable global structures and therefore achieve computation reductions on the order of r 2 /9 and r 3, respectively, where r is the pruning rate. We additionally present numerical results showing EC provides a good approximation of computation in both batch and incremental (iSAM2) optimization and demonstrate that pruning methods promoting globally-efficient structure outperform those that do not.

AAMAS Conference 2018 Conference Paper

Crossmodal Attentive Skill Learner

• Shayegan Omidshafiei
• Dong-Ki Kim
• Jason Pazis
• Jonathan P. How

This paper introduces the Crossmodal Attentive Skill Learner (CASL), integrated with the recently-introduced Asynchronous Advantage Option-Critic (A2OC) architecture [15] to enable hierarchical reinforcement learning across multiple sensory inputs. We provide concrete examples where the approach not only improves performance in a single task, but accelerates transfer to new tasks. We demonstrate the attention mechanism anticipates and identifies useful latent features, while filtering irrelevant sensor modalities during execution. We modify the Arcade Learning Environment [7] to support audio queries, and conduct evaluations of crossmodal learning in the Atari 2600 games H. E. R. O. and Amidar. Finally, building on the recent work of Babaeizadeh et al. [4], we open-source a fast hybrid CPU-GPU implementation of CASL. 1

IROS Conference 2018 Conference Paper

Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learning

• Michael Everett
• Yu Fan Chen
• Jonathan P. How

Robots that navigate among pedestrians use collision avoidance algorithms to enable safe and efficient operation. Recent works present deep reinforcement learning as a framework to model the complex interactions and cooperation. However, they are implemented using key assumptions about other agents' behavior that deviate from reality as the number of agents in the environment increases. This work extends our previous approach to develop an algorithm that learns collision avoidance among a variety of types of dynamic agents without assuming they follow any particular behavior rules. This work also introduces a strategy using LSTM that enables the algorithm to use observations of an arbitrary number of other agents, instead of previous methods that have a fixed observation size. The proposed algorithm outperforms our previous approach in simulation as the number of agents increases, and the algorithm is demonstrated on a fully autonomous robotic vehicle traveling at human walking speed.

ICRA Conference 2018 Conference Paper

Near-Optimal Adversarial Policy Switching for Decentralized Asynchronous Multi-Agent Systems

• Trong Nghia Hoang
• Yuchen Xiao
• Kavinayan Sivakumar
• Christopher Amato
• Jonathan P. How

A key challenge in multi-robot and multi-agent systems is generating solutions that are robust to other self-interested or even adversarial parties who actively try to prevent the agents from achieving their goals. The practicality of existing works addressing this challenge is limited to only small-scale synchronous decision-making scenarios or a single agent planning its best response against a single adversary with fixed, procedurally characterized strategies. In contrast this paper considers a more realistic class of problems where a team of asynchronous agents with limited observation and communication capabilities need to compete against multiple strategic adversaries with changing strategies. This problem necessitates agents that can coordinate to detect changes in adversary strategies and plan the best response accordingly. Our approach first optimizes a set of stratagems that represent these best responses. These optimized stratagems are then integrated into a unified policy that can detect and respond when the adversaries change their strategies. The near-optimality of the proposed framework is established theoretically as well as demonstrated empirically in simulation and hardware.

ICRA Conference 2018 Conference Paper

Robust Collision Avoidance via Sliding Control

• Brett T. Lopez
• Jean-Jacques E. Slotine
• Jonathan P. How

Recent advances in perception and planning algorithms have enabled robots to navigate autonomously through unknown, cluttered environments at high-speeds. A key component of these systems is the ability to identify, select, and execute a safe trajectory around obstacles. Many of these systems, however, lack performance guarantees because model uncertainty and external disturbances are ignored when a trajectory is selected for execution. This work leverages results from nonlinear control theory to establish a bound on tracking performance that can be used to select a provably safe trajectory. The Composite Adaptive Sliding Controller (CASC) provides robustness to disturbances and reduces model uncertainty through high-rate parameter estimation. CASC is demonstrated in simulation and hardware to significantly improve the performance of a quadrotor navigating through unknown environments with external disturbances and unknown model parameters.

ICRA Conference 2018 Conference Paper

Talk Resource-Efficiently to Me: Optimal Communication Planning for Distributed Loop Closure Detection

• Matthew Giamou
• Kasra Khosoussi
• Jonathan P. How

Due to the distributed nature of cooperative simultaneous localization and mapping (CSLAM), detecting inter-robot loop closures necessitates sharing sensory data with other robots. A naïve approach to data sharing can easily lead to a waste of mission-critical resources. This paper investigates the logistical aspects of CSLAM. Particularly, we present a general resource-efficient communication planning framework that takes into account both the total amount of exchanged data and the induced division of labor between the participating robots. Compared to other state-of-the-art approaches, our framework is able to verify the same set of potential inter-robot loop closures while exchanging considerably less data and influencing the induced workloads. We develop a fast algorithm for finding globally optimal communication policies, and present theoretical analysis to characterize the necessary and sufficient conditions under which simpler strategies are optimal. The proposed framework is extensively evaluated with data from the KITTI odometry benchmark datasets.

IROS Conference 2018 Conference Paper

Transferable Pedestrian Motion Prediction Models at Intersections

• Macheng Shen
• Golnaz Habibi
• Jonathan P. How

One desirable capability of autonomous cars is to accurately predict the pedestrian motion near intersections for safe and efficient trajectory planning. We are interested in developing transfer learning algorithms that can be trained on the pedestrian trajectories collected at one intersection and yet still provide accurate predictions of the trajectories at another, previously unseen intersection. We first discussed the feature selection for transferable pedestrian motion models in general. Following this discussion, we developed one transferable pedestrian motion prediction algorithm based on Inverse Reinforcement Learning (IRL) that infers pedestrian intentions and predicts future trajectories based on observed trajectory. We evaluated our algorithm at three intersections. We used the accuracy of augmented semi-nonnegative sparse coding (ASNSC), trained and tested at the same intersection as a baseline. The result shows that the proposed algorithm improves the baseline accuracy by a statistically significant percentage in both non-transfer task and transfer task.

ICRA Conference 2017 Conference Paper

Aggressive 3-D collision avoidance for high-speed navigation

• Brett T. Lopez
• Jonathan P. How

Autonomous robot navigation through unknown, cluttered environments at high-speeds is still an open problem. Quadrotor platforms with this capability have only begun to emerge with the advancements in light-weight, small form factor sensing and computing. Many of the existing platforms, however, require excessive computation time to perform collision avoidance, which ultimately limits the vehicle's top speed. This work presents an efficient perception and planning approach that significantly reduces the computation time by using instantaneous perception data for collision avoidance. Minimum-time, state and input constrained motion primitives are generated by sampling terminal states until a collision-free path is found. The worst case performance of the Triple Integrator Planner (TIP) is nearly an order of magnitude faster than the state-of-the-art. Experimental results demonstrate the algorithm's ability to plan and execute aggressive collision avoidance maneuvers in highly cluttered environments.

IROS Conference 2017 Conference Paper

Aggressive collision avoidance with limited field-of-view sensing

• Brett T. Lopez
• Jonathan P. How

Quadrotors that navigate through unknown, cluttered environments have only recently begun to emerge following the development of small form-factor sensing and computing hardware and computationally efficient collision avoidance algorithms. Computation time of planning algorithms has significantly decreased in part to using local information as opposed to using a global map for collision avoidance. Safe planning with local information, however, restricts the direction of travel to remain within the perception system's field-of-view (FOV). The vehicle's motion becomes more constrained with body-mounted narrow FOV sensors, reducing vehicle maneuverability and speed. This work presents a relaxed-constraint Model Predictive Control framework that allows motions outside the perception FOV with guaranteed safety. The key aspect of this approach is the ability to safely choose a motion primitive generated in the past. Simulation and hardware results shows the new framework improves time to goal and flight path efficiency in environments with varying levels of clutter.

ICRA Conference 2017 Conference Paper

Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning

• Yu Fan Chen
• Miao Liu 0001
• Michael Everett
• Jonathan P. How

Finding feasible, collision-free paths for multiagent systems can be challenging, particularly in non-communicating scenarios where each agent's intent (e. g. goal) is unobservable to the others. In particular, finding time efficient paths often requires anticipating interaction with neighboring agents, the process of which can be computationally prohibitive. This work presents a decentralized multiagent collision avoidance algorithm based on a novel application of deep reinforcement learning, which effectively offloads the online computation (for predicting interaction patterns) to an offline learning procedure. Specifically, the proposed approach develops a value network that encodes the estimated time to the goal given an agent's joint configuration (positions and velocities) with its neighbors. Use of the value network not only admits efficient (i. e. , real-time implementable) queries for finding a collision-free velocity vector, but also considers the uncertainty in the other agents' motion. Simulation results show more than 26% improvement in paths quality (i. e. , time to reach the goal) when compared with optimal reciprocal collision avoidance (ORCA), a state-of-the-art collision avoidance strategy.

ICML Conference 2017 Conference Paper

Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability

• Shayegan Omidshafiei
• Jason Pazis
• Christopher Amato
• Jonathan P. How
• John Vian

Many real-world tasks involve multiple agents with partial observability and limited communication. Learning is challenging in these settings due to local viewpoints of agents, which perceive the world as non-stationary due to concurrently-exploring teammates. Approaches that learn specialized policies for individual tasks face problems when applied to the real world: not only do agents have to learn and store distinct policies for each task, but in practice identities of tasks are often non-observable, making these approaches inapplicable. This paper formalizes and addresses the problem of multi-task multi-agent reinforcement learning under partial observability. We introduce a decentralized single-task learning approach that is robust to concurrent interactions of teammates, and present an approach for distilling single-task policies into a unified policy that performs well across multiple related tasks, without explicit provision of task identity.

IROS Conference 2017 Conference Paper

Demand estimation and chance-constrained fleet management for ride hailing

• Justin Miller
• Jonathan P. How

In autonomous Mobility on Demand (MOD) systems, customers request rides from a fleet of shared vehicles that can be automatically positioned in response to customer demand. Recent approaches to MOD systems have focused on environments where customers can only request rides through an app or by waiting at a station. This paper develops MOD fleet management approaches for ride hailing, where customers may instead request rides simply by hailing a passing vehicle, an approach of particular importance for campus MOD systems. The challenge for ride hailing is that customer demand is not explicitly provided as it would be with an app, but rather customers are only served if a vehicle happens to be located at the arrival location. This work focuses on maximizing the number of served hailing customers in an MOD system by learning and utilizing customer demand. A Bayesian framework is used to define a novel customer demand model which incorporates observed pedestrian traffic to estimate customer arrival locations with a quantification of uncertainty. An exploration planner is proposed which routes MOD vehicles in order to reduce arrival rate uncertainty. A robust ride hailing fleet management planner is proposed which routes vehicles under the presence of uncertainty using a chance-constrained formulation. Simulation of a real-world MOD system on MIT's campus demonstrates the effectiveness of the planners. The customer demand model and exploration planner are demonstrated to reduce estimation error over time and the ride hailing planner is shown to improve the fraction of served customers in the system by 73% over a baseline exploration approach.

ICRA Conference 2017 Conference Paper

Duckietown: An open, inexpensive and flexible platform for autonomy education and research

• Liam Paull
• Jacopo Tani
• Heejin Ahn
• Javier Alonso-Mora
• Luca Carlone
• Michal Cáp
• Yu Fan Chen
• Changhyun Choi

Duckietown is an open, inexpensive and flexible platform for autonomy education and research. The platform comprises small autonomous vehicles (“Duckiebots”) built from off-the-shelf components, and cities (“Duckietowns”) complete with roads, signage, traffic lights, obstacles, and citizens (duckies) in need of transportation. The Duckietown platform offers a wide range of functionalities at a low cost. Duckiebots sense the world with only one monocular camera and perform all processing onboard with a Raspberry Pi 2, yet are able to: follow lanes while avoiding obstacles, pedestrians (duckies) and other Duckiebots, localize within a global map, navigate a city, and coordinate with other Duckiebots to avoid collisions. Duckietown is a useful tool since educators and researchers can save money and time by not having to develop all of the necessary supporting infrastructure and capabilities. All materials are available as open source, and the hope is that others in the community will adopt the platform for education and research.

IROS Conference 2017 Conference Paper

Learning for multi-robot cooperation in partially observable stochastic environments with macro-actions

• Miao Liu 0001
• Kavinayan Sivakumar
• Shayegan Omidshafiei
• Christopher Amato
• Jonathan P. How

This paper presents a data-driven approach for multi-robot coordination in partially-observable domains based on Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) and macro-actions (MAs). Dec-POMDPs provide a general framework for cooperative sequential decision making under uncertainty and MAs allow temporally extended and asynchronous action execution. To date, most methods assume the underlying Dec-POMDP model is known a priori or a full simulator is available during planning time. Previous methods which aim to address these issues suffer from local optimality and sensitivity to initial conditions. Additionally, few hardware demonstrations involving a large team of heterogeneous robots and with long planning horizons exist. This work addresses these gaps by proposing an iterative sampling based Expectation-Maximization algorithm (iSEM) to learn polices using only trajectory data containing observations, MAs, and rewards. Our experiments show the algorithm is able to achieve better solution quality than the state-of-the-art learning-based methods. We implement two variants of multi-robot Search and Rescue (SAR) domains (with and without obstacles) on hardware to demonstrate the learned policies can effectively control a team of distributed robots to cooperate in a partially observable stochastic environment.

ICRA Conference 2017 Conference Paper

Predictive positioning and quality of service ridesharing for campus mobility on demand systems

• Justin Miller
• Jonathan P. How

Autonomous Mobility On Demand (MOD) systems can utilize fleet management strategies in order to provide a high customer quality of service (QoS). Previous works on autonomous MOD systems have developed methods for rebalancing single capacity vehicles, where QoS is maintained through large fleet sizing. This work focuses on MOD systems utilizing a small number of vehicles, such as those found on a campus, where additional vehicles cannot be introduced as demand for rides increases. A predictive positioning method is presented for improving customer QoS by identifying key locations to position the fleet in order to minimize expected customer wait time. Ridesharing is introduced as a means for improving customer QoS as arrival rates increase. However, with ridesharing perceived QoS is dependent on an often unknown customer preference. To address this challenge, a customer ratings model, which learns customer preference from a 5-star rating, is developed and incorporated directly into a ridesharing algorithm. The predictive positioning and ridesharing methods are applied to simulation of a real-world campus MOD system. A combined predictive positioning and ridesharing approach is shown to reduce customer service times by up to 29%. and the customer ratings model is shown to provide the best overall MOD fleet management performance over a range of customer preferences.

ICRA Conference 2017 Conference Paper

Scalable accelerated decentralized multi-robot policy search in continuous observation spaces

• Shayegan Omidshafiei
• Christopher Amato
• Miao Liu 0001
• Michael Everett
• Jonathan P. How
• John Vian

This paper presents the first ever approach for solving continuous-observation Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) and their semi-Markovian counterparts, Dec-POSMDPs. This contribution is especially important in robotics, where a vast number of sensors provide continuous observation data. A continuous-observation policy representation is introduced using Stochastic Kernel-based Finite State Automata (SK-FSAs). An SK-FSA search algorithm titled Entropy-based Policy Search using Continuous Kernel Observations (EPSCKO) is introduced and applied to the first ever continuous-observation Dec-POMDP/Dec-POSMDP domain, where it significantly outperforms state-of-the-art discrete approaches. This methodology is equally applicable to Dec-POMDPs and Dec-POSMDPs, though the empirical analysis presented focuses on Dec-POSMDPs due to their higher scalability. To improve convergence, an entropy injection policy search acceleration approach for both continuous and discrete observation cases is also developed and shown to improve convergence rates without degrading policy quality.

ICRA Conference 2017 Conference Paper

Semantic-level decentralized multi-robot decision-making using probabilistic macro-observations

• Shayegan Omidshafiei
• Shih-Yuan Liu
• Michael Everett
• Brett T. Lopez
• Christopher Amato
• Miao Liu 0001
• Jonathan P. How
• John Vian

Robust environment perception is essential for decision-making on robots operating in complex domains. Intelligent task execution requires principled treatment of uncertainty sources in a robot's observation model. This is important not only for low-level observations (e. g. , accelerom-eter data), but also for high-level observations such as semantic object labels. This paper formalizes the concept of macro-observations in Decentralized Partially Observable Semi-Markov Decision Processes (Dec-POSMDPs), allowing scalable semantic-level multi-robot decision making. A hierarchical Bayesian approach is used to model noise statistics of low-level classifier outputs, while simultaneously allowing sharing of domain noise characteristics between classes. Classification accuracy of the proposed macro-observation scheme, called Hierarchical Bayesian Noise Inference (HBNI), is shown to exceed existing methods. The macro-observation scheme is then integrated into a Dec-POSMDP planner, with hardware experiments running onboard a team of dynamic quadrotors in a challenging domain where noise-agnostic filtering fails. To the best of our knowledge, this is the first demonstration of a real-time, convolutional neural net-based classification framework running fully onboard a team of quadrotors in a multi-robot decision-making domain.

IROS Conference 2017 Conference Paper

Socially aware motion planning with deep reinforcement learning

• Yu Fan Chen
• Michael Everett
• Miao Liu 0001
• Jonathan P. How

For robotic vehicles to navigate safely and efficiently in pedestrian-rich environments, it is important to model subtle human behaviors and navigation rules (e. g. , passing on the right). However, while instinctive to humans, socially compliant navigation is still difficult to quantify due to the stochasticity in people's behaviors. Existing works are mostly focused on using feature-matching techniques to describe and imitate human paths, but often do not generalize well since the feature values can vary from person to person, and even run to run. This work notes that while it is challenging to directly specify the details of what to do (precise mechanisms of human navigation), it is straightforward to specify what not to do (violations of social norms). Specifically, using deep reinforcement learning, this work develops a time-efficient navigation policy that respects common social norms. The proposed method is shown to enable fully autonomous navigation of a robotic vehicle moving at human walking speed in an environment with many pedestrians.

IROS Conference 2017 Conference Paper

Stable laser interest point selection for place recognition in a forest

• Matthew Giamou
• Yaroslav Babich
• Golnaz Habibi
• Jonathan P. How

Place recognition is an essential part of robot localization and mapping problems. Using lower data-rate sensors like 2D scanning laser rangefinders enables the robots to use less memory and computation in building maps. However, place recognition by a vehicle with 6-DOF dynamics like a quadrotor in unstructured, 3D environments like forests is challenging, especially with a sensor that only measures a planar slice of the environment. This paper extends the 2D geometry-based place recognition system of [1] to a challenging forest envirnoment with a novel procedure for selecting stable and salient 2D laser interest points using Dirichlet process clustering (DP-means). This method is tested on both synthetic and real data from a forest trail and compared with [1]. The result reveals the importance of salient interest point selection in allowing accurate and fast place recognition. Our approach also ensures a low bandwidth representation of visited areas, making it suitable for real-time, multi-agent SLAM applications.

ICRA Conference 2016 Conference Paper

Augmented dictionary learning for motion prediction

• Yu Fan Chen
• Miao Liu 0001
• Jonathan P. How

Developing accurate models and efficient representations of multivariate trajectories is important for understanding the behavior patterns of mobile agents. This work presents a dictionary learning algorithm for developing a part-based trajectory representation, which combines merits of the existing Markovian-based and clustering-based approaches. In particular, this work presents the augmented semi-nonnegative sparse coding (ASNSC) algorithm for solving a constrained dictionary learning problem, and shows that the proposed method would converge to a local optimum given a convexity condition. We consider a trajectory modeling application, in which the learned dictionary atoms correspond to local motion patterns. Classical semi-nonnegative sparse coding approaches would add dictionary atoms with opposite signs to reduce the representational error, which can lead to learning noisy dictionary atoms that correspond poorly to local motion patterns. ASNSC addresses this problem and learns a concise set of intuitive motion patterns. ASNSC shows significant improvement over existing trajectory modeling methods in both prediction accuracy and computational time, as revealed by extensive numerical analysis on real datasets.

ICRA Conference 2016 Conference Paper

Autonomous drifting using simulation-aided reinforcement learning

• Mark Cutler
• Jonathan P. How

We introduce a framework that combines simple and complex continuous state-action simulators with a real-world robot to efficiently find good control policies, while minimizing the number of samples needed from the physical robot. The framework combines the strengths of various simulation levels by first finding optimal policies in a simple model, and then using that solution to initialize a gradient-based learner in a more complex simulation. The policy and transition dynamics from the complex simulation are in turn used to guide the learning in the physical world. A method is developed for transferring information gathered in the physical world back to the learning agent in the simulation. The new information is used to re-evaluate whether the original simulated policy is still optimal given the updated knowledge from the real-world. This reverse transfer is critical to minimizing samples from the physical world. The new framework is demonstrated on a robotic car learning to perform controlled drifting maneuvers. A video of the car's performance can be found at https://youtu.be/opsmd5yuBF0.

IROS Conference 2016 Conference Paper

Dynamic arrival rate estimation for campus Mobility On Demand network graphs

• Justin Miller
• Andres Hasfura
• Shih-Yuan Liu
• Jonathan P. How

Mobility On Demand (MOD) systems are revolutionizing transportation in urban settings by improving vehicle utilization and reducing parking congestion. A key factor in the success of an MOD system is the ability to measure and respond to real-time customer arrival data. Real time traffic arrival rate data is traditionally difficult to obtain due to the need to install fixed sensors throughout the MOD network. This paper presents a framework for measuring pedestrian traffic arrival rates using sensors onboard the vehicles that make up the MOD fleet. A novel distributed fusion algorithm is presented which combines onboard LIDAR and camera sensor measurements to detect trajectories of pedestrians with a 90% detection hit rate with 1. 5 false positives per minute. A novel moving observer method is introduced to estimate pedestrian arrival rates from pedestrian trajectories collected from mobile sensors. The moving observer method is evaluated in both simulation and hardware and is shown to achieve arrival rate estimates comparable to those that would be obtained with multiple stationary sensors.

ICRA Conference 2016 Conference Paper

Graph-based Cross Entropy method for solving multi-robot decentralized POMDPs

• Shayegan Omidshafiei
• Ali-Akbar Agha-Mohammadi
• Christopher Amato
• Shih-Yuan Liu
• Jonathan P. How
• John Vian

This paper introduces a probabilistic algorithm for multi-robot decision-making under uncertainty, which can be posed as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP). Dec-POMDPs are inherently synchronous decision-making frameworks which require significant computational resources to be solved, making them infeasible for many real-world robotics applications. The Decentralized Partially Observable Semi-Markov Decision Process (Dec-POSMDP) was recently introduced as an extension of the Dec-POMDP that uses high-level macro-actions to allow large-scale, asynchronous decision-making. However, existing Dec-POSMDP solution methods have limited scalability or perform poorly as the problem size grows. This paper proposes a cross-entropy based Dec-POSMDP algorithm motivated by the combinatorial optimization literature. The algorithm is applied to a constrained package delivery domain, where it significantly outperforms existing Dec-POSMDP solution methods.

IROS Conference 2016 Conference Paper

Motion planning with diffusion maps

• Yu Fan Chen
• Shih-Yuan Liu
• Miao Liu 0001
• Justin Miller
• Jonathan P. How

Many robotic applications require repeated, on-demand motion planning in mapped environments. In addition, the presence of other dynamic agents, such as people, often induces frequent, dynamic changes in the environment. Having a potential function that encodes pairwise cost-to-go can be useful for improving the computational speed of finding feasible paths, and for guiding local searches around dynamic obstacles. However, since storing pairwise potential can be impractical given the O(|V| 2 ) memory requirement, existing work often needs to compute a potential function for each query to a new goal, which would require a substantial online computation. This work addresses the problem by using diffusion maps, a machine learning algorithm, to learn the map's geometry and develop a memory-efficient parametrization (O(|V|)) of pairwise potentials. Specially, each state in the map is transformed to a diffusion coordinate, in which pairwise Euclidean distance is shown to be a meaningful similarity metric. We develop diffusion-based motion planning algorithms and, through extensive numerical evaluation, show that the proposed algorithms find feasible paths of similar quality with orders of magnitude improvement in computational speed compared with single-query methods. The proposed algorithms are implemented on hardware to enable real-time autonomous navigation in an indoor environment with frequent interactions with pedestrians.

IROS Conference 2016 Conference Paper

SLAM with objects using a nonparametric pose graph

• Beipeng Mu
• Shih-Yuan Liu
• Liam Paull
• John J. Leonard
• Jonathan P. How

Mapping and self-localization in unknown environments are fundamental capabilities in many robotic applications. These tasks typically involve the identification of objects as unique features or landmarks, which requires the objects both to be detected and then assigned a unique identifier that can be maintained when viewed from different perspectives and in different images. The data association and simultaneous localization and mapping (SLAM) problems are, individually, well-studied in the literature. But these two problems are inherently tightly coupled, and that has not been well-addressed. Without accurate SLAM, possible data associations are combinatorial and become intractable easily. Without accurate data association, the error of SLAM algorithms diverge easily. This paper proposes a novel nonparametric pose graph that models data association and SLAM in a single framework. An algorithm is further introduced to alternate between inferring data association and performing SLAM. Experimental results show that our approach has the new capability of associating object detections and localizing objects at the same time, leading to significantly better performance on both the data association and SLAM problems than achieved by considering only one and ignoring imperfections in the other.

ICRA Conference 2015 Conference Paper

Decentralized control of Partially Observable Markov Decision Processes using belief space macro-actions

• Shayegan Omidshafiei
• Ali-Akbar Agha-Mohammadi
• Christopher Amato
• Jonathan P. How

The focus of this paper is on solving multi-robot planning problems in continuous spaces with partial observability. Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) are general models for multi-robot coordination problems, but representing and solving Dec-POMDPs is often intractable for large problems. To allow for a high-level representation that is natural for multi-robot problems and scalable to large discrete and continuous problems, this paper extends the Dec-POMDP model to the Decentralized Partially Observable Semi-Markov Decision Process (Dec-POSMDP). The Dec-POSMDP formulation allows asynchronous decision-making by the robots, which is crucial in multi-robot domains. We also present an algorithm for solving this Dec-POSMDP which is much more scalable than previous methods since it can incorporate closed-loop belief space macro-actions in planning. These macro-actions are automatically constructed to produce robust solutions. The proposed method's performance is evaluated on a complex multi-robot package delivery problem under uncertainty, showing that our approach can naturally represent multi-robot problems and provide high-quality solutions for large-scale problems.

ICRA Conference 2015 Conference Paper

Decoupled multiagent path planning via incremental sequential convex programming

• Yu Fan Chen
• Mark Cutler
• Jonathan P. How

This paper presents a multiagent path planning algorithm based on sequential convex programming (SCP) that finds locally optimal trajectories. Previous work using SCP efficiently computes motion plans in convex spaces with no static obstacles. In many scenarios where the spaces are non-convex, previous SCP-based algorithms failed to find feasible solutions because the convex approximation of collision constraints leads to forming a sequence of infeasible optimization problems. This paper addresses this problem by tightening collision constraints incrementally, thus forming a sequence of more relaxed, feasible intermediate optimization problems. We show that the proposed algorithm increases the probability of finding feasible trajectories by 33% for teams of more than three vehicles in non-convex environments. Further, we show that decoupling the multiagent optimization problem to a number of single-agent optimization problems leads to significant improvement in computational tractability. We develop a decoupled implementation of the proposed algorithm, abbreviated dec-iSCP. We show that dec-iSCP runs 14% faster and finds feasible trajectories with higher probability than a decoupled implementation of previous SCP-based algorithms. The proposed algorithm is real-time implementable and is validated through hardware experiments on a team of quadrotors.

ICRA Conference 2015 Conference Paper

Efficient reinforcement learning for robots using informative simulated priors

• Mark Cutler
• Jonathan P. How

Autonomous learning through interaction with the physical world is a promising approach to designing controllers and decision-making policies for robots. Unfortunately, learning on robots is often difficult due to the large number of samples needed for many learning algorithms. Simulators are one way to decrease the samples needed from the robot by incorporating prior knowledge of the dynamics into the learning algorithm. In this paper we present a novel method for transferring data from a simulator to a robot, using simulated data as a prior for real-world learning. A Bayesian nonparametric prior is learned from a potentially black-box simulator. The mean of this function is used as a prior for the Probabilistic Inference for Learning Control (PILCO) algorithm. The simulated prior improves the convergence rate and performance of PILCO by directing the policy search in areas of the state-space that have not yet been observed by the robot. Simulated and hardware results show the benefits of using the prior knowledge in the learning framework.

IROS Conference 2015 Conference Paper

Online heterogeneous multiagent learning under limited communication with applications to forest fire management

• Nazim Kemal Ure
• Shayegan Omidshafiei
• Brett T. Lopez
• Ali-Akbar Agha-Mohammadi
• Jonathan P. How
• John Vian

Many robotic missions require online estimation of the unknown state transition models associated with uncertainty that stems from mission dynamics. The learning problem is usually distributed among agents in multiagent scenarios, either due to the absence of a centralized processing unit or because of the large size of the joint learning problem. This paper addresses the problem of multiagent learning in the likely scenario that agents estimate different models from their measured data, but they can share information by communicating model parameters. Previous approaches either consider homogeneous scenarios or perform model transfer in an open-loop manner, which hinders the convergence rate. We develop a closed-loop multiagent learning algorithm, Collaborative Filtering-Decentralized Incremental Feature Dependency Discovery (CF-Dec-iFDD), which enables agents to learn and share models in heterogeneous scenarios. Each agent learns a linear function approximation of the actual model, and the number of features is increased incrementally to adjust model complexity based on the observed data. The agents obtain feedback from other agents on the model error reduction associated with the communicated features. Although this increases the communication cost of exchanging features, it improves the quality/utility of what is being exchanged, leading to improved convergence rate. The approach is demonstrated in indoor hardware flight tests on a forest fire management scenario for which agents must learn the transition model of the fire spread depending on external factors such as wind and vegetation. It is shown that CF-Dec-iFDD has superior convergence rate compared to the alternative approaches.

ICRA Conference 2015 Conference Paper

Planning for decentralized control of multiple robots under uncertainty

• Christopher Amato
• George Konidaris 0001
• Gabriel Cruz
• Christopher A. Maynor
• Jonathan P. How
• Leslie Pack Kaelbling

This paper presents a probabilistic framework for synthesizing control policies for general multi-robot systems that is based on decentralized partially observable Markov decision processes (Dec-POMDPs). Dec-POMDPs are a general model of decision-making where a team of agents must cooperate to optimize a shared objective in the presence of uncertainty. Dec-POMDPs also consider communication limitations, so execution is decentralized. While Dec-POMDPs are typically intractable to solve for real-world problems, recent research on the use of macro-actions in Dec-POMDPs has significantly increased the size of problem that can be practically solved. We show that, in contrast to most existing methods that are specialized to a particular problem class, our approach can synthesize control policies that exploit any opportunities for coordination that are present in the problem, while balancing uncertainty, sensor information, and information about other agents. We use three variants of a warehouse task to show that a single planner of this type can generate cooperative behavior using task allocation, direct communication, and signaling, as appropriate. This demonstrates that our algorithmic framework can automatically optimize control and communication policies for complex multi-robot systems.

JMLR Journal 2015 Journal Article

RLPy: A Value-Function-Based Reinforcement Learning Framework for Education and Research

• Alborz Geramifard
• Christoph Dann
• Robert H. Klein
• William Dabney
• Jonathan P. How

RLPy is an object-oriented reinforcement learning software package with a focus on value-function-based methods using linear function approximation and discrete actions. The framework was designed for both educational and research purposes. It provides a rich library of fine-grained, easily exchangeable components for learning agents (e.g., policies or representations of value functions), facilitating recently increased specialization in reinforcement learning. RLPy is written in Python to allow fast prototyping, but is also suitable for large-scale experiments through its built-in support for optimized numerical libraries and parallelization. Code profiling, domain visualizations, and data analysis are integrated in a self-contained package available under the Modified BSD License at github.com/rlpy/rlpy. All of these properties allow users to compare various reinforcement learning algorithms with little effort. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2015. ( edit, beta )

IROS Conference 2015 Conference Paper

Robust incremental SLAM with consistency-checking

• Matthew C. Graham
• Jonathan P. How
• Donald E. Gustafson

Incorrect landmark and loop closure measurements can cause standard SLAM algorithms to fail catastrophically. Recently, several SLAM algorithms have been proposed that are robust to loop closure errors, but it is shown in this paper that they cannot provide robust solutions when landmark measurement errors occur. The root cause of this problem is that the robust SLAM algorithms only focus on generating solutions that are locally consistent (i. e. each measurement agrees with its corresponding estimates) rather than globally consistent (i. e. all of the measurements in the solution agree with each other). Moreover, these algorithms do not attempt to maximize the number of correct measurements included in the solution, meaning that often correct measurements are ignored and the solution quality suffers as a result. This paper proposes a new formulation of the robust SLAM problem that seeks a globally consistent map that also maximizes the number of measurements included in the solution. In addition, a novel incremental SLAM algorithm, called incremental SLAM with consistency-checking, is developed to solve the new robust SLAM problem. Finally, simulated and experimental results show that the new algorithm significantly outperforms state-of-the-art robust SLAM methods for datasets with incorrect landmark measurements and can match their performance for datasets with incorrect loop closures.

IJCAI Conference 2015 Conference Paper

Stick-Breaking Policy Learning in Dec-POMDPs

• Miao Liu
• Christopher Amato
• Xuejun Liao
• Lawrence Carin
• Jonathan P. How

Expectation maximization (EM) has recently been shown to be an efficient algorithm for learning finite-state controllers (FSCs) in large decentralized POMDPs (Dec-POMDPs). However, current methods use fixed-size FSCs and often converge to maxima that are far from the optimal value. This paper represents the local policy of each agent using variable-sized FSCs that are constructed using a stick-breaking prior, leading to a new framework called decentralized stick-breaking policy representation (Dec-SBPR). This approach learns the controller parameters with a variational Bayesian algorithm without having to assume that the Dec- POMDP model is available. The performance of Dec-SBPR is demonstrated on several benchmark problems, showing that the algorithm scales to large problems while outperforming other state-of-the-art methods.

UAI Conference 2014 Conference Paper

Approximate Decentralized Bayesian Inference

• Trevor Campbell
• Jonathan P. How

This paper presents an approximate method for performing Bayesian inference in models with conditional independence over a decentralized network of learning agents. The method first employs variational inference on each individual learning agent to generate a local approximate posterior, the agents transmit their local posteriors to other agents in the network, and finally each agent combines its set of received local posteriors. The key insight in this work is that, for many Bayesian models, approximate inference schemes destroy symmetry and dependencies in the model that are crucial to the correct application of Bayes’ rule when combining the local posteriors. The proposed method addresses this issue by including an additional optimization step in the combination procedure that accounts for these broken dependencies. Experiments on synthetic and real data demonstrate that the decentralized method provides advantages in computational performance and predictive test likelihood over previous batch and distributed methods.

IROS Conference 2014 Conference Paper

Camera control for learning nonlinear target dynamics via Bayesian nonparametric Dirichlet-process Gaussian-process (DP-GP) models

• Hongchuan Wei
• Wenjie Lu 0005
• Pingping Zhu
• Silvia Ferrari
• Robert H. Klein
• Shayegan Omidshafiei
• Jonathan P. How

This paper presents a camera control approach for learning unknown nonlinear target dynamics by approximating information value functions using particles that represent targets' position distributions. The target dynamics are described by a non-parametric mixture model that can learn a potentially infinite number of motion patterns. Assuming that each motion pattern can be represented as a velocity field, the target behaviors can be described by a non-parametric Dirichlet process-Gaussian process (DP-GP) mixture model. The DP-GP model has been successfully applied for clustering time-invariant spatial phenomena due to its flexibility to adapt to data complexity without overfitting. A new DP-GP information value function is presented that can be used by the sensor to explore and improve the DP-GP mixture model. The optimal camera control is computed to maximize this information value function online via a computationally efficient particle-based search method. The proposed approach is demonstrated through numerical simulations and hardware experiments in the RAVEN testbed at MIT.

IROS Conference 2014 Conference Paper

Health aware stochastic planning for persistent package delivery missions using quadrotors

• Ali-Akbar Agha-Mohammadi
• Nazim Kemal Ure
• Jonathan P. How
• John Vian

In persistent missions, taking system's health and capability degradation into account is an essential factor to predict and avoid failures. The state space in health-aware planning problems is often a mixture of continuous vehicle-level and discrete mission-level states. This in particular poses a challenge when the mission domain is partially observable and restricts the use of computationally expensive forward search methods. This paper presents a method that exploits a structure that exists in many health-aware planning problems and performs a two-layer planning scheme. The lower layer exploits the local linearization and Gaussian distribution assumption over vehicle-level states while the higher layer maintains a non-Gaussian distribution over discrete mission-level variables. This two-layer planning scheme allows us to limit the expensive online forward search to the mission-level states, and thus predict system's behavior over longer horizons in the future. We demonstrate the performance of the method on a long duration package delivery mission using a quadrotor in a partially-observable domain in the presence of constraints and health/capability degradation.

ICRA Conference 2014 Conference Paper

Human aware UAS path planning in urban environments using nonstationary MDPs

• Rakshit Allamaraju
• Hassan A. Kingravi
• Allan Axelrod
• Girish Chowdhary 0001
• Robert C. Grande
• Jonathan P. How
• Christopher Crick
• Weihua Sheng

A growing concern with deploying Unmanned Aerial Vehicles (UAVs) in urban environments is the potential violation of human privacy, and the backlash this could entail. Therefore, there is a need for UAV path planning algorithms that minimize the likelihood of invading human privacy. We formulate the problem of human-aware path planning as a nonstationary Markov Decision Process, and provide a novel model-based reinforcement learning solution that leverages Gaussian process clustering. Our algorithm is flexible enough to accommodate changes in human population densities by employing Bayesian nonparametrics, and is real-time computable. The approach is validated experimentally on a large-scale long duration experiment with both simulated and real UAVs.

UAI Conference 2014 Conference Paper

Quantifying Nonlocal Informativeness in High-Dimensional, Loopy Gaussian Graphical Models

• Daniel S. Levine 0002
• Jonathan P. How

We consider the problem of selecting informative observations in Gaussian graphical models containing both cycles and nuisances. More specifically, we consider the subproblem of quantifying conditional mutual information measures that are nonlocal on such graphs. The ability to efficiently quantify the information content of observations is crucial for resource-constrained data acquisition (adaptive sampling) and data processing (active learning) systems. While closedform expressions for Gaussian mutual information exist, standard linear algebraic techniques, with complexity cubic in the network size, are intractable for high-dimensional distributions. We investigate the use of embedded trees for computing nonlocal pairwise mutual information and demonstrate through numerical simulations that the presented approach achieves a significant reduction in computational cost over inversionbased methods.

ICRA Conference 2014 Conference Paper

Reinforcement learning with multi-fidelity simulators

• Mark Cutler
• Thomas J. Walsh 0001
• Jonathan P. How

We present a framework for reinforcement learning (RL) in a scenario where multiple simulators are available with decreasing amounts of fidelity to the real-world learning scenario. Our framework is designed to limit the number of samples used in each successively higher-fidelity/cost simulator by allowing the agent to choose to run trajectories at the lowest level that will still provide it with information. The approach transfers state-action Q-values from lower-fidelity models as heuristics for the “Knows What It Knows” family of RL algorithms, which is applicable over a wide range of possible dynamics and reward representations. Theoretical proofs of the framework's sample complexity are given and empirical results are demonstrated on a remote controlled car with multiple simulators. The approach allows RL algorithms to find near-optimal policies for the real world with fewer expensive real-world samples than previous transfer approaches or learning without simulators.

ICML Conference 2014 Conference Paper

Sample Efficient Reinforcement Learning with Gaussian Processes

• Robert C. Grande
• Thomas J. Walsh 0001
• Jonathan P. How

This paper derives sample complexity results for using Gaussian Processes (GPs) in both model-based and model-free reinforcement learning (RL). We show that GPs are KWIK learnable, proving for the first time that a model-based RL approach using GPs, GP-Rmax, is sample efficient (PAC-MDP). However, we then show that previous approaches to model-free RL using GPs take an exponential number of steps to find an optimal policy, and are therefore not sample efficient. The third and main contribution is the introduction of a model-free RL algorithm using GPs, DGPQ, which is sample efficient and, in contrast to model-based algorithms, capable of acting in real time, as demonstrated on a five-dimensional aircraft simulator.

UAI Conference 2013 Conference Paper

Batch-iFDD for Representation Expansion in Large MDPs

• Alborz Geramifard
• Thomas J. Walsh 0001
• Nicholas Roy
• Jonathan P. How

Matching pursuit (MP) methods are a promising class of feature construction algorithms for value function approximation. Yet existing MP methods require creating a pool of potential features, mandating expert knowledge or enumeration of a large feature pool, both of which hinder scalability. This paper introduces batch incremental feature dependency discovery (Batch-iFDD) as an MP method that inherits a provable convergence property. Additionally, Batch-iFDD does not require a large pool of features, leading to lower computational complexity. Empirical policy evaluation results across three domains with up to one million states highlight the scalability of Batch-iFDD over the previous state of the art MP algorithm.

ICRA Conference 2013 Conference Paper

Rapid transfer of controllers between UAVs using learning-based adaptive control

• Girish Chowdhary 0001
• Tongbin Wu
• Mark Cutler
• Jonathan P. How

Commonly used Proportional-Integral-Derivative based UAV flight controllers are often seen to provide adequate trajectory-tracking performance, but only after extensive tuning. The gains of these controllers are tuned to particular platforms, which makes transferring controllers from one UAV to other time-intensive. This paper formulates the problem of control-transfer from a source system to a transfer system and proposes a solution that leverages well-studied techniques in adaptive control. It is shown that concurrent learning adaptive controllers improve the trajectory tracking performance of a quadrotor with the baseline linear controller directly imported from another quadrotor whose inertial characteristics and throttle mapping are very different. Extensive flight-testing, using indoor quadrotor platforms operated in MIT's RAVEN environment, is used to validate the method.

ICRA Conference 2013 Conference Paper

Reinforcement learning with misspecified model classes

• Joshua Mason Joseph
• Alborz Geramifard
• John W. Roberts
• Jonathan P. How
• Nicholas Roy

Real-world robots commonly have to act in complex, poorly understood environments where the true world dynamics are unknown. To compensate for the unknown world dynamics, we often provide a class of models to a learner so it may select a model, typically using a minimum prediction error metric over a set of training data. Often in real-world domains the model class is unable to capture the true dynamics, due to either limited domain knowledge or a desire to use a small model. In these cases we call the model class misspecified, and an unfortunate consequence of misspecification is that even with unlimited data and computation there is no guarantee the model with minimum prediction error leads to the best performing policy. In this work, our approach improves upon the standard maximum likelihood model selection metric by explicitly selecting the model which achieves the highest expected reward, rather than the most likely model. We present an algorithm for which the highest performing model from the model class is guaranteed to be found given unlimited data and computation. Empirically, we demonstrate that our algorithm is often superior to the maximum likelihood learner in a batch learning setting for two common RL benchmark problems and a third real-world system, the hydrodynamic cart-pole, a domain whose complex dynamics cannot be known exactly.

ICRA Conference 2013 Conference Paper

Scalable reward learning from demonstration

• Bernard Michini
• Mark Cutler
• Jonathan P. How

Reward learning from demonstration is the task of inferring the intents or goals of an agent demonstrating a task. Inverse reinforcement learning methods utilize the Markov decision process (MDP) framework to learn rewards, but typically scale poorly since they rely on the calculation of optimal value functions. Several key modifications are made to a previously developed Bayesian nonparametric inverse reinforcement learning algorithm that avoid calculation of an optimal value function and no longer require discretization of the state or action spaces. Experimental results given demonstrate the ability of the resulting algorithm to scale to larger problems and learn in domains with continuous demonstrations.

ICRA Conference 2012 Conference Paper

Improving the efficiency of Bayesian inverse reinforcement learning

• Bernard Michini
• Jonathan P. How

Inverse reinforcement learning (IRL) is the task of learning the reward function of a Markov Decision Process (MDP) given knowledge of the transition function and a set of expert demonstrations. While many IRL algorithms exist, Bayesian IRL [1] provides a general and principled method of reward learning by casting the problem in the Bayesian inference framework. However, the algorithm as originally presented suffers from several inefficiencies that prohibit its use for even moderate problem sizes. This paper proposes modifications to the original Bayesian IRL algorithm to improve its efficiency and tractability in situations where the state space is large and the expert demonstrations span only a small portion of it. The key insight is that the inference task should be focused on states that are similar to those encountered by the expert, as opposed to making the naive assumption that the expert demonstrations contain enough information to accurately infer the reward function over the entire state space. A modified algorithm is presented and experimental results show substantially faster convergence while maintaining the solution quality of the original method.

ICRA Conference 2011 Conference Paper

A decentralized approach to multi-agent planning in the presence of constraints and uncertainty

• Aditya Undurti
• Jonathan P. How

We address the problem of planning in the presence of uncertainty and constraints for teams of unmanned vehicles. The problem is formulated as a Constrained Markov Decision Process (C-MDP). We allow for plans with a non-zero but bounded probability of violating constraints, a quantity that we define as risk and provide a solution technique that keeps the risk below a specified threshold while optimizing reward. We also use the decoupling between the dynamics of individual agents to assume transition independence and use this assumption to reduce the complexity of the problem. We provide representative simulation results to show that our technique achieves high reward while keeping risk bounded.

ICRA Conference 2011 Conference Paper

Decentralized path planning for multi-agent teams in complex environments using rapidly-exploring random trees

• Vishnu Desaraju
• Jonathan P. How

This paper presents a novel approach to address the challenge of planning paths for multi-agent systems operating in complex environments. The algorithm developed, Decentralized Multi-Agent Rapidly-exploring Random Tree (DMA-RRT), is an extension of the Closed-loop RRT (CL-RRT) algorithm to the multi-agent case, retaining its ability to plan quickly even with complex constraints. Moreover, a merit-based token passing coordination strategy is developed to dynamically update the planning order based on a measure of each agent's incentive to replan, derived from the CL-RRT. Agents with a greater incentive plan sooner, yielding a greater reduction of the global cost and greater improvement in the team's overall performance. An extended version of the algorithm, Cooperative DMA-RRT, allows agents to modify others' plans in order to select paths that reduce their combined cost and thus further improve global performance. The paths generated by both algorithms are proven to satisfy inter-agent constraints, such as collision avoidance, and a set of simulation and experimental results verify performance.

ICRA Conference 2011 Conference Paper

Design and flight testing of an autonomous variable-pitch quadrotor

• Buddy Michini
• Josh Redding
• Nazim Kemal Ure
• Mark Cutler
• Jonathan P. How

This video submission presents a design concept of an autonomous variable-pitch quadrotor with constant motor speed. The main aim of this work is to increase the maneuverability of the quadrotor vehicle concept while largely maintaining its mechanical simplicity. This added maneuverability will allow autonomous agile maneuvers like inverted hover and flip. A custom in lab built quadrotor with onboard attitude stabilization is developed and tested in the ACL's (Aerospace Controls Laboratory) RAVEN (Real-time indoor Autonomous Vehicle test ENvironment). Initial flight results show that the quadrotor is capable of waypoint tracking and hovering both upright and inverted.

ICML Conference 2011 Conference Paper

Online Discovery of Feature Dependencies

• Alborz Geramifard
• Finale Doshi
• Josh Redding
• Nicholas Roy
• Jonathan P. How

ICRA Conference 2010 Conference Paper

A voice-commandable robotic forklift working alongside humans in minimally-prepared outdoor environments

• Seth J. Teller
• Matthew R. Walter
• Matthew E. Antone
• Andrew Correa
• Randall Davis
• Luke Fletcher
• Emilio Frazzoli
• James R. Glass

One long-standing challenge in robotics is the realization of mobile autonomous robots able to operate safely in existing human workplaces in a way that their presence is accepted by the human occupants. We describe the development of a multi-ton robotic forklift intended to operate alongside human personnel, handling palletized materials within existing, busy, semi-structured outdoor storage facilities.

ICRA Conference 2010 Conference Paper

An online algorithm for constrained POMDPs

• Aditya Undurti
• Jonathan P. How

This work seeks to address the problem of planning in the presence of uncertainty and constraints. Such problems arise in many situations, including the basis of this work, which involves planning for a team of first responders (both humans and robots) operating in an urban environment. The problem is framed as a Partially-Observable Markov Decision Process (POMDP) with constraints, and it is shown that even in a relatively simple planning problem, modeling constraints as large penalties does not lead to good solutions. The main contribution of the work is a new online algorithm that explicitly ensures constraint feasibility while remaining computationally tractable. Its performance is demonstrated on an example problem and it is demonstrated that our online algorithm generates policies comparable to an offline constrained POMDP algorithm.

IROS Conference 2010 Conference Paper

Threat-aware path planning in uncertain urban environments

• Georges Aoude
• Brandon Luders
• Daniel S. Levine 0002
• Jonathan P. How

This paper considers the path planning problem for an autonomous vehicle in an urban environment populated with static obstacles and moving vehicles with uncertain intents. We propose a novel threat assessment module, consisting of an intention predictor and a threat assessor, which augments the host vehicle's path planner with a real-time threat value representing the risks posed by the estimated intentions of other vehicles. This new threat-aware planning approach is applied to the CL-RRT path planning framework, used by the MIT team in the 2007 DARPA Grand Challenge. The strengths of this approach are demonstrated through simulation and experiments performed in the RAVEN testbed facilities.

ICRA Conference 2009 Conference Paper

Vision-based guidance and control of a hovering vehicle in unknown, GPS-denied environments

• Spencer Ahrens
• Daniel S. Levine 0002
• Gregory Andrews
• Jonathan P. How

This paper describes the system architecture and core algorithms for a quadrotor helicopter that uses vision data to navigate an unknown, indoor, GPS-denied environment. Without external sensing, an estimation system that relies only on integrating inertial data will have rapidly drifting position estimates. Micro aerial vehicles (MAVs) are stringently weight-constrained, leaving little margin for additional sensors beyond the mission payload. The approach taken in this paper is to introduce an architecture that exploits a common mission payload, namely a video camera, as a dual-use sensor to aid in navigation. Several core algorithms, including a fast environment mapper and a novel heuristic for obstacle avoidance, are also presented. Finally, drift-free hover and obstacle avoidance flight tests in a controlled environment are presented and analyzed.

ICRA Conference 2008 Conference Paper

Autonomous aircraft flight control for constrained environments

• Jonathan P. How
• James S. McGrew
• Adrian A. Frank
• George H. Hines

The real-time indoor autonomous vehicle test environment (RAVEN) at MIT's Aerospace Controls Laboratory is home to a diverse fleet of aircraft, from a styrofoam and cellophane dragonfly to a set of quadrotor Draganflyer helicopters. The helicopters are used primarily for swarm and health management research. Alongside these machines is a set of more conventional aircraft designed to study autonomous aircraft flight control in constrained environments. The objectives of this work are to develop and validate flight control concepts for aggressive (aerobatic) maneuvers, and, in particular, to identify the sensor suites needed, and the likely limits of achievable performance. Our work is motivated by the future goals of flying micro (or nano) air vehicles in constrained (e. g. , urban or indoors) environments.

IROS Conference 2008 Conference Paper

Motion planning for urban driving using RRT

• Yoshiaki Kuwata
• Gaston A. Fiore
• Justin Teo
• Emilio Frazzoli
• Jonathan P. How

This paper provides a detailed analysis of the motion planning subsystem for the MIT DARPA Urban Challenge vehicle. The approach is based on the Rapidly-exploring Random Trees (RRT) algorithm. The purpose of this paper is to present the numerous extensions made to the standard RRT algorithm that enable the on-line use of RRT on robotic vehicles with complex, unstable dynamics and significant drift, while preserving safety in the face of uncertainty and limited sensing. The paper includes numerous simulation and race results that clearly demonstrate the effectiveness of the planning system.

ICRA Conference 2007 Conference Paper

The MIT Indoor Multi-Vehicle Flight Testbed

• Mario J. Valenti
• Brett Bethke
• Daniel Dale
• Adrian A. Frank
• James S. McGrew
• Spencer Ahrens
• Jonathan P. How
• John Vian

This paper and video present the components and flight tests of an indoor, multi-vehicle testbed that was developed to study long duration UAV missions in a controlled environment. This testbed is designed to use real hardware to examine research questions related to single-and multi-vehicle health management, such as vehicle failures, refueling, and maintenance. The testbed has both aerial and ground vehicles that operate autonomously in a large, indoor flight test area and can be used to execute many different mission scenarios. The success of this testbed is largely related to our choice of vehicles, sensors, and the system's command and control architecture. The video presents flight test results from single-and multi-vehicle experiments over the past year.

ICRA Conference 2000 Conference Paper

An Indoor Absolute Positioning System with No Line of Sight Restrictions and Building-Wide Coverage

• Eric Prigge
• Jonathan P. How

Accurate sensing of vehicle position and attitude is required in many mobile robot applications, but is a very challenging problem in real office building or warehouse environments. Many existing indoor positioning systems are limited in workspace and robustness because they require clear lines of sight, have insufficient coverage area, or do not provide absolute measurements. This work presents a new absolute position and attitude sensing system with no line of sight restrictions and an operating area that can span an entire office building or warehouse. The system uses beacons fixed at known locations throughout a building to create dipole magnetic fields. All of the beacons produce fields at all times, and a code division multiple access (CDMA) method is presented to distinguish between the fields produced by the individual beacons. This method provides advantages over existing magnetic field techniques in coverage volume, accuracy, and resistance to distortion. Initial experimental results demonstrate centimeter-level positioning accuracy.