Daniela Rus

ICML Conference 2025 Conference Paper

ABNet: Adaptive explicit-Barrier Net for Safe and Scalable Robot Learning

• Wei Xiao 0003
• Tsun-Hsuan Wang
• Chuang Gan 0001
• Daniela Rus

Safe learning is central to AI-enabled robots where a single failure may lead to catastrophic results. Existing safe learning methods are not scalable, inefficient and hard to train, and tend to generate unstable signals under noisy inputs that are challenging to be deployed for robots. To address these challenges, we propose Adaptive explicit-Barrier Net (ABNet) in which barriers explicitly show up in the closed-form model that guarantees safety. The ABNet has the potential to incrementally scale toward larger safe foundation models. Each head of ABNet could learn safe control policies from different features and focuses on specific part of the observation. In this way, we do not need to directly construct a large model for complex tasks, which significantly facilitates the training of the model while ensuring its stable output. Most importantly, we can still formally prove the safety guarantees of the ABNet. We demonstrate the efficiency and strength of ABNet in 2D robot obstacle avoidance, safe robot manipulation, and vision-based end-to-end autonomous driving, with results showing much better robustness and guarantees over existing models.

ICRA Conference 2025 Conference Paper

Adaptive Walker: User Intention and Terrain Aware Intelligent Walker with High-Resolution Tactile and IMU Sensor

• Yunho Choi
• Seokhyun Hwang
• JaeYoung Moon
• Hosu Lee
• Dohyeon Yeo
• Minwoo Seong
• Yiyue Luo
• SeungJun Kim 0001

In this paper, we present an adaptive walker system designed to address limitations in current intelligent walker technologies. While recent advancements have been made in this field, existing systems often struggle to seamlessly interpret user intent for speed control and lack adaptability across diverse scenarios and terrain. Our proposed solution incorporates high-resolution tactile sensors, deep learning algorithms, IMU sensors, and linear motors to dynamically adjust to the user's intentions and terrain changes. The system is capable of predicting the user's desired speed with an error margin of only 20. 99%, relying solely on tactile input from hand and arm contact points. Additionally, it maintains the walker's horizontal stability with an error of less than 1 degree by adjusting leg lengths in response to variations in ground angle. This adaptive walker enhances user safety and comfort, particularly for individuals with reduced strength or cognitive abilities, and offers reliable assistance on uneven terrain such as uphill and downhill paths.

ICRA Conference 2025 Conference Paper

AI-Enhanced Automatic Design of Efficient Underwater Gliders

• Peter Yichen Chen
• Pingchuan Ma 0002
• Niklas Hagemann
• John W. Romanishin
• Wei Wang 0078
• Daniela Rus
• Wojciech Matusik

The development of novel autonomous underwater gliders has been hindered by limited shape diversity, primarily due to the reliance on traditional design tools that depend heavily on manual trial and error. Building an automated design framework is challenging due to the complexities of representing glider shapes and the high computational costs associated with modeling complex solid-fluid interactions. In this work, we introduce an AI-enhanced automated computational framework designed to overcome these limitations by enabling the creation of underwater robots with non-trivial hull shapes. Our approach involves an algorithm that cooptimizes both shape and control signals, utilizing a reducedorder geometry representation and a differentiable neural-network-based fluid surrogate model. This end-to-end design workflow facilitates rapid iteration and evaluation of hydrodynamic performance, leading to the discovery of optimal and complex hull shapes across various control settings. We validate our method through wind tunnel experiments and swimming pool gliding tests, demonstrating that our computationally designed gliders surpass manually designed counterparts in terms of energy efficiency. By addressing challenges in efficient shape representation and neural fluid surrogate models, our work paves the way for the development of highly efficient underwater gliders, with implications for long-range ocean exploration and environmental monitoring.

NeurIPS Conference 2025 Conference Paper

Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples

• Shiva Sreeram
• Alaa Maalouf
• Pratyusha Sharma
• Daniela Rus

Recently, Sharma et al. (2024) suggested a method called LAyer- SElective-Rank reduction (LASER) which demonstrated that pruning high‑order components of carefully chosen LLM’s weight matrices can boost downstream accuracy—without any gradient‑based fine‑tuning. Yet LASER’s exhaustive, per‑matrix search (each requiring full‑dataset forward passes) makes it impractical for rapid deployment. We demonstrate that this overhead can be removed and find that: (i) Only a small, carefully chosen subset of matrices needs to be inspected—eliminating the layer‑by‑layer sweep, (ii) The gradient of each matrix’s singular values pinpoints which matrices merit reduction, (iii) Increasing the factorization search space by allowing matrices rows to cluster around multiple subspaces and then decomposing each cluster separately further reduces overfitting on the original training data and further lifts accuracy by up to 24. 6 percentage points, and finally, (iv) we discover that evaluating on just 100 samples rather than the full training data—both for computing the indicative gradients and for measuring the final accuracy—suffices to further reduce the search time; we explain that as adaptation to downstream tasks is dominated by prompting style, not dataset size. As a results, we show that combining these findings yields a fast and robust adaptation algorithm for downstream tasks. Overall, with a single gradient step on 100 examples and a quick scan of the top candidate layers and factorization techniques, we can adapt LLMs to new datasets—entirely without fine‑tuning.

ICRA Conference 2025 Conference Paper

Generating Out-of-Distribution Scenarios Using Language Models

• Erfan Aasi
• Phat Nguyen
• Shiva Sreeram
• Guy Rosman
• Sertac Karaman
• Daniela Rus

The deployment of autonomous vehicles controlled by machine learning techniques requires extensive testing in diverse real-world environments, robust handling of edge cases and out-of-distribution scenarios, and comprehensive safety validation to ensure that these systems can navigate safely and effectively under unpredictable conditions. Addressing Out-OfDistribution (OOD) driving scenarios is essential for enhancing safety, as OOD scenarios help validate the reliability of the models within the vehicle's autonomy stack. However, generating OOD scenarios is challenging due to their long-tailed distribution and rarity in urban driving datasets. Recently, Large Language Models (LLMs) have shown promise in autonomous driving, particularly for their zero-shot generalization and common-sense reasoning capabilities. In this paper, we leverage these LLM strengths to introduce a framework for generating diverse OOD driving scenarios. Our approach uses LLMs to construct a branching tree, where each branch represents a unique OOD scenario. These scenarios are then simulated in the CARLA simulator using an automated framework that aligns scene augmentation with the corresponding textual descriptions. We evaluate our framework through extensive simulations, and assess its performance via a diversity metric that measures the richness of the scenarios. Additionally, we introduce a new “OOD-ness” metric, which quantifies how much the generated scenarios deviate from typical urban driving conditions. Furthermore, we explore the capacity of modern Vision-Language Models (VLMs) to interpret and safely navigate through the simulated OOD scenarios. Our findings offer valuable insights into the reliability of language models in addressing OOD scenarios within the context of urban driving.

ICRA Conference 2025 Conference Paper

Generative-AI-Driven Jumping Robot Design Using Diffusion Models

• Byungchul Kim
• Tsun-Hsuan Wang
• Daniela Rus

Astract-Recent advances in foundation models are significantly expanding the capabilities of AI models. As part of this progress, this paper introduces a robot design framework that uses a diffusion model approach for generating 3D mesh structures. Specifically, we focus on generating directly fabri-cable robot structures that require no post-processing guided by human-imposed design constraints. Our approach can find the optimal design of the robot by optimizing or composing embedding vectors of the model. The efficacy of the framework is validated through an application to design, fabricate, and evaluate a jumping robot. Our solution is an optimized jumping robot with a 41% increase in jump height compared to the state-of-the-art design. Additionally, when the robot is augmented with an optimized foot, it can land reliably with a success ratio of 88% in contrast to the 4% success ratio of the base robot.

ICRA Conference 2025 Conference Paper

Hypergraph-Transformer (HGT) for Interaction Event Prediction in Laparoscopic and Robotic Surgery

• Lianhao Yin
• Yutong Ban
• Jennifer A. Eckhoff
• Ozanan R. Meireles
• Daniela Rus
• Guy Rosman

Understanding and anticipating events and actions is critical for intraoperative assistance and decision-making during minimally invasive surgery. We propose a predictive neural network that is capable of understanding and predicting critical interaction aspects of surgical workflow based on endoscopic, intracorporeal video data, while flexibly leveraging surgical knowledge graphs. The approach incorporates a hypergraph-transformer (HGT) structure that encodes expert knowledge into the network design and predicts the hidden embedding of the graph. We verify our approach on established surgical datasets and applications, including the prediction of action-triplets, and the achievement of the Critical View of Safety (CVS), which is a critical safety measure. Moreover, we address specific, safety-related forecasts of surgical processes, such as predicting the clipping of the cystic duct or artery without prior achievement of the CVS. Our results demonstrate improvement in prediction of interactive event when incorporating with our approach compared to unstructured alternatives.

ICRA Conference 2025 Conference Paper

Large-Expansion Bi-Layer Auxetics Create Compliant Cellular Motion

• Lillian Chin
• Gregory Xie
• Jeffrey I. Lipton
• Daniela Rus

There is significant interest in creating compliant modular robots that can change their volume. Inspired by how biological cells move, these systems can potentially combine the resilience of modular robotics with the increased environmental interactions of soft robotics. However, current versions have limited speed, expansion, and portability. In this paper, we address these concerns through AuxSwarm, a compliant system composed of auxetic-based robotic voxels. These voxels control their volume through a scissor-like bi-layer auxetic design, growing up to 1. 57 times their original size in 0. 2 seconds. This combination of speed and expansion is unique across modular soft robots, enabling dynamic locomotion capabilities. We characterize the voxels and demonstrate the versatility of this approach through case studies of 2D bending and 3D cube flipping. AuxSwarm provides a first step towards addressable voxel-based smart materials, while simultaneously addressing the robustness and actuation challenges faced by soft robots.

ICRA Conference 2025 Conference Paper

Learning Object Properties Using Robot Proprioception via Differentiable Robot-Object Interaction

• Peter Yichen Chen
• Chao Liu 0021
• Pingchuan Ma 0002
• John Eastman
• Daniela Rus
• Dylan Randle
• Yuri Ivanov
• Wojciech Matusik

Differentiable simulation has become a powerful tool for system identification. While prior work has focused on identifying robot properties using robot-specific data or object properties using object-specific data, our approach calibrates object properties by using information from the robot, without relying on data from the object itself. Specifically, we utilize robot joint encoder information, which is commonly available in standard robotic systems. Our key observation is that by analyzing the robot's reactions to manipulated objects, we can infer properties of those objects, such as inertia and softness. Leveraging this insight, we develop differentiable simulations of robot-object interactions to inversely identify the properties of the manipulated objects. Our approach relies solely on proprioception – the robot's internal sensing capabilities – and does not require external measurement tools or vision-based tracking systems. This general method is applicable to any articulated robot and requires only joint position information. We demonstrate the effectiveness of our method on a low-cost robotic platform, achieving accurate mass and elastic modulus estimations of manipulated objects with just a few seconds of computation on a laptop.

ICLR Conference 2025 Conference Paper

Oscillatory State-Space Models

• T. Konstantin Rusch
• Daniela Rus

We propose Linear Oscillatory State-Space models (LinOSS) for efficiently learning on long sequences. Inspired by cortical dynamics of biological neural networks, we base our proposed LinOSS model on a system of forced harmonic oscillators. A stable discretization, integrated over time using fast associative parallel scans, yields the proposed state-space model. We prove that LinOSS produces stable dynamics only requiring nonnegative diagonal state matrix. This is in stark contrast to many previous state-space models relying heavily on restrictive parameterizations. Moreover, we rigorously show that LinOSS is universal, i.e., it can approximate any continuous and causal operator mapping between time-varying functions, to desired accuracy. In addition, we show that an implicit-explicit discretization of LinOSS perfectly conserves the symmetry of time reversibility of the underlying dynamics. Together, these properties enable efficient modeling of long-range interactions, while ensuring stable and accurate long-horizon forecasting. Finally, our empirical results, spanning a wide range of time-series tasks from mid-range to very long-range classification and regression, as well as long-horizon forecasting, demonstrate that our proposed LinOSS model consistently outperforms state-of-the-art sequence models. Notably, LinOSS outperforms Mamba and LRU by nearly 2x on a sequence modeling task with sequences of length 50k.

ICRA Conference 2025 Conference Paper

Realizing Emergent Collective Behaviors Through Robotic Swarmalators

• Richard Beattie
• Steven Ceron
• Daniela Rus

Swarmalators move as a function of their pairwise phase interactions, and control their phase as a function of their relative position or motion to other agents. This enables dual sync and swarm behaviors that mimic those exhibited by diverse natural and artificial swarms; these behaviors have almost entirely been explored only through computational simulations. Here, we realize through a 15-robot collective many of the predicted swarmalator behaviors when agents are chiral and non-chiral, when there is frequency coupling, and when the natural frequency distribution is homogeneous and heterogeneous. This work presents an experimental platform that can realize many theoretically predicted collective behaviors, it sheds light on the differences between the simulations and experiments, and it will serve in future studies to realize swarmalator and active matter collective behaviors.

ICLR Conference 2025 Conference Paper

ReGen: Generative Robot Simulation via Inverse Design

• Phat Nguyen
• Tsun-Hsuan Wang
• Zhang-Wei Hong
• Erfan Aasi
• Andrew Silva
• Guy Rosman
• Sertac Karaman
• Daniela Rus

Simulation plays a key role in scaling robot learning and validating policies, but constructing simulations remains labor-intensive. In this paper, we introduce ReGen, a generative simulation framework that automates this process using inverse design. Given an agent's behavior (such as a motion trajectory or objective function) and its textual description, we infer the underlying scenarios and environments that could have caused the behavior. Our approach leverages large language models to construct and expand a graph that captures cause-and-effect relationships and relevant entities with properties in the environment, which is then processed to configure a robot simulation environment. Our approach supports (i) augmenting simulations based on ego-agent behaviors, (ii) controllable, counterfactual scenario generation, (iii) reasoning about agent cognition and mental states, and (iv) reasoning with distinct sensing modalities, such as braking due to faulty GPS signals. We demonstrate our method in autonomous driving and robot manipulation tasks, generating more diverse, complex simulated environments compared to existing simulations with high success rates, and enabling controllable generation for corner cases. This approach enhances the validation of robot policies and supports data or simulation augmentation, advancing scalable robot learning for improved generalization and robustness.

IROS Conference 2025 Conference Paper

Safe Motion Planning and Control Using Predictive and Adaptive Barrier Methods for Autonomous Surface Vessels

• Alejandro Gonzalez-Garcia
• Wei Xiao 0003
• Wei Wang 0078
• Alejandro Astudillo
• Wilm Decré
• Jan Swevers
• Carlo Ratti
• Daniela Rus

Safe motion planning is essential for autonomous vessel operations, especially in challenging spaces such as narrow inland waterways. However, conventional motion planning approaches are often computationally intensive or overly conservative. This paper proposes a safe motion planning strategy combining Model Predictive Control (MPC) and Control Barrier Functions (CBFs). We introduce a time-varying inflated ellipse obstacle representation, where the inflation radius is adjusted depending on the relative position and attitude between the vessel and the obstacle. The proposed adaptive inflation reduces the conservativeness of the controller compared to traditional fixed-ellipsoid obstacle formulations. The MPC solution provides an approximate motion plan, and high-order CBFs ensure the vessel’s safety using the varying inflation radius. Simulation and real-world experiments demonstrate that the proposed strategy enables the fully-actuated autonomous robot vessel to navigate through narrow spaces in real time and resolve potential deadlocks, all while ensuring safety.

ICLR Conference 2025 Conference Paper

SafeDiffuser: Safe Planning with Diffusion Probabilistic Models

• Wei Xiao 0003
• Tsun-Hsuan Wang
• Chuang Gan 0001
• Ramin M. Hasani
• Mathias Lechner
• Daniela Rus

Diffusion models have shown promise in data-driven planning. While these planners are commonly employed in applications where decisions are critical, they still lack established safety guarantees. In this paper, we address this limitation by introducing SafeDiffuser, a method to equip diffusion models with safety guarantees via control barrier functions. The key idea of our approach is to embed finite-time diffusion invariance, i.e., a form of specification consisting of safety constraints, into the denoising diffusion procedure. This way we enable data generation under safety constraints. We show that SafeDiffusers maintain the generative performance of diffusion models while also providing robustness in safe data generation. We evaluate our method on a series of tasks, including maze path generation, legged robot locomotion, and 3D space manipulation, and demonstrate the advantages of robustness over vanilla diffusion models.

AAAI Conference 2025 Conference Paper

The Master Key Filters Hypothesis: Deep Filters Are General

• Zahra Babaiee
• Peyman M. Kiasari
• Daniela Rus
• Radu Grosu

This paper challenges the prevailing view that convolutional neural network (CNN) filters become increasingly specialized in deeper layers. Motivated by recent observations of clusterable repeating patterns in depthwise separable CNNs (DS-CNNs) trained on ImageNet, we extend this investigation across various domains and datasets. Our analysis of DS-CNNs reveals that deep filters maintain generality, contradicting the expected transition to class-specific features. We demonstrate the generalizability of these filters through transfer learning experiments, showing that frozen filters from models trained on different datasets perform well and can be further improved when sourced from larger, better-performing models. Our findings indicate that spatial features learned by depthwise separable convolutions remain generic across all layers, domains, and architectures. This research provides new insights into the nature of generalization in neural networks, particularly in DS-CNNs, and has significant implications for transfer learning and model design.

NeurIPS Conference 2025 Conference Paper

The Quest for Universal Master Key Filters in DS-CNNs

• Zahra Babaiee
• Peyman M. Kiasari
• Daniela Rus
• Radu Grosu

A recent study has proposed the ``Master Key Filters Hypothesis" for convolutional neural network filters. This paper extends this hypothesis by radically constraining its scope to a single set of just 8 universal filters that depthwise separable convolutional networks inherently converge to. While conventional DS-CNNs employ thousands of distinct trained filters, our analysis reveals these filters are predominantly linear shifts (ax+b) of our discovered universal set. Through systematic unsupervised search, we extracted these fundamental patterns across different architectures and datasets. Remarkably, networks initialized with these 8 unique frozen filters achieve over 80\% ImageNet accuracy, and even outperform models with thousands of trainable parameters when applied to smaller datasets. The identified master key filters closely match Difference of Gaussians (DoGs), Gaussians, and their derivatives, structures that are not only fundamental to classical image processing but also strikingly similar to receptive fields in mammalian visual systems. Our findings provide compelling evidence that depthwise convolutional layers naturally gravitate toward this fundamental set of spatial operators regardless of task or architecture. This work offers new insights for understanding generalization and transfer learning through the universal language of these master key filters.

ICML Conference 2025 Conference Paper

Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models

• Zahra Babaiee
• Peyman M. Kiasari
• Daniela Rus
• Radu Grosu

Recent advancements in multimodal large language models have driven breakthroughs in visual question answering. Yet, a critical gap persists, ‘conceptualization’—the ability to recognize and reason about the same concept despite variations in visual form, a basic ability of human reasoning. To address this challenge, we introduce the Visual Graph Arena (VGA), a dataset featuring six graph-based tasks designed to evaluate and improve AI systems’ capacity for visual abstraction. VGA uses diverse graph layouts (e. g. , Kamada-Kawai vs. planar) to test reasoning independent of visual form. Experiments with state-of-the-art vision models and multimodal LLMs reveal a striking divide: humans achieved near-perfect accuracy across tasks, while models totally failed on isomorphism detection and showed limited success in path/cycle tasks. We further identify behavioral anomalies suggesting pseudo-intelligent pattern matching rather than genuine understanding. These findings underscore fundamental limitations in current AI models for visual understanding. By isolating the challenge of representation-invariant reasoning, the VGA provides a framework to drive progress toward human-like conceptualization in AI visual models. The Visual Graph Arena is available at: https: //vga. csail. mit. edu/.

ICRA Conference 2024 Conference Paper

Approximating Robot Configuration Spaces with few Convex Sets using Clique Covers of Visibility Graphs

• Peter Werner
• Alexandre Amice
• Tobia Marcucci
• Daniela Rus
• Russ Tedrake

Many computations in robotics can be dramatically accelerated if the robot configuration space is described as a collection of simple sets. For example, recently developed motion planners rely on a convex decomposition of the free space to design collision-free trajectories using fast convex optimization. In this work, we present an efficient method for approximately covering complex configuration spaces with a small number of polytopes. The approach constructs a visibility graph using sampling and generates a clique cover of this graph to find clusters of samples that have mutual line of sight. These clusters are then inflated into large, full-dimensional, polytopes. We evaluate our method on a variety of robotic systems and show that it consistently covers larger portions of free configuration space, with fewer polytopes, and in a fraction of the time compared to previous methods.

IROS Conference 2024 Conference Paper

Competitive Multi-Team Behavior in Dynamic Flight Scenarios

• Tim Seyde
• Mathias Lechner
• Joshua Rountree
• Daniela Rus

Efficiently learning strategic multi-agent behavior remains a challenge for robotic systems deployed in real-world scenarios, especially when considering underactuated or dynamically unstable systems. Such systems demand an integrated approach that informs long-term strategic planning with constraints imposed by reactive control, and vice versa, to effectively accomplish task objectives in competitive scenarios. In this paper, we introduce a hierarchical control model to address this: a high-level controller synthesizes strategic guidance from aggregated team experiences, while a low-level controller formulates corresponding task-specific continuous controls. We apply this concept to coordination of competitive multi-team behavior in dynamic flight scenarios with F-16 aircraft. This work introduces a hierarchical reinforcement learning approach for multi-agent coordination, leveraging decoupled distributional value representations at the high-level together with goal-conditioned policy learning at the low-level, providing a control structure that integrates long-horizon strategic planning with short-horizon dynamic control. We further provide a parallel simulator for efficient learning with multi-agent F-16 dynamics.

NeurIPS Conference 2024 Conference Paper

DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning

• Zijian Zhou
• Xiaoqiang Lin
• Xinyi Xu
• Alok Prakash
• Daniela Rus
• Bryan Kian Hsiang Low

In-context learning (ICL) allows transformer-based language models that are pre-trained on general text to quickly learn a specific task with a few "task demonstrations" without updating their parameters, significantly boosting their flexibility and generality. ICL possesses many distinct characteristics from conventional machine learning, thereby requiring new approaches to interpret this learning paradigm. Taking the viewpoint of recent works showing that transformers learn in context by formulating an internal optimizer, we propose an influence function-based attribution technique, DETAIL, that addresses the specific characteristics of ICL. We empirically verify the effectiveness of our approach for demonstration attribution while being computationally efficient. Leveraging the results, we then show how DETAIL can help improve model performance in real-world scenarios through demonstration reordering and curation. Finally, we experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.

ICRA Conference 2024 Conference Paper

Directly 3D Printed, Pneumatically Actuated Multi-Material Robotic Hand

• Hanna Matusik
• Chao Liu 0021
• Daniela Rus

Soft robotic manipulators with many degrees of freedom can carry out complex tasks safely around humans. However, manufacturing of soft robotic hands with several degrees of freedom requires a complex multi-step manual process, which significantly increases their cost. We present a design of a multi-material 15 DoF robotic hand with five fingers including an opposable thumb. Our design has 15 pneumatic actuators based on a series of hollow chambers that are driven by an external pressure system. The thumb utilizes rigid joints and the palm features internal rigid structure and soft skin. The design can be directly 3D printed using a multi-material additive manufacturing process without any assembly process and therefore our hand can be manufactured for less than 300 dollars. We test the hand in conjunction with a low-cost vision-based teleoperation system on different tasks.

ICRA Conference 2024 Conference Paper

Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models

• Tsun-Hsuan Wang
• Alaa Maalouf
• Wei Xiao 0003
• Yutong Ban
• Alexander Amini
• Guy Rosman
• Sertac Karaman
• Daniela Rus

As autonomous driving technology matures, end-to-end methodologies have emerged as a leading strategy, promising seamless integration from perception to control via deep learning. However, existing systems grapple with challenges such as unexpected open set environments and the complexity of black-box models. At the same time, the evolution of deep learning introduces larger, multimodal foundational models, offering multi-modal visual and textual understanding. In this paper, we harness these multimodal foundation models to enhance the robustness and adaptability of autonomous driving systems. We introduce a method to extract nuanced spatial features from transformers and the incorporation of latent space simulation for improved training and policy debugging. We use pixel/patch-aligned feature descriptors to expand foundational model capabilities to create an end-to-end multimodal driving model, demonstrating unparalleled results in diverse tests. Our solution combines language with visual perception and achieves significantly greater robustness on out-of-distribution situations.

ICRA Conference 2024 Conference Paper

Embedded air channels transform soft lattices into sensorized grippers

• Annan Zhang
• Lillian Chin
• Daniel L. Tong
• Daniela Rus

Sensing plays a pivotal role in robotic manipulation, dictating the accuracy and versatility with which objects are handled. Vision-based sensing methods often suffer from fabrication complexity and low durability, while approaches that rely on direct measurements on the gripper often have limited resolution and are difficult to scale. Here, we present a soft robotic gripper made out of two cubic lattices that are sensorized by embedding air channels within the structure. The lattices are 3D printed from a single build material, simplifying the fabrication process. The flexibility of this approach offers significant control over sensor and lattice design, while the pressure-based internal sensing provides measurements with minimal disruption to the grasping surface. With only 12 sensors, 6 per lattice, this gripper can estimate an object’s weight and location and offer new insights into grasp parameters like friction coefficients and grasp force.

IROS Conference 2024 Conference Paper

Exploring Latent Pathways: Enhancing the Interpretability of Autonomous Driving with a Variational Autoencoder

• Anass Bairouk
• Mirjana Maras
• Simon Herlin
• Alexander Amini
• Marc Blanchon
• Ramin M. Hasani
• Patrick Chareyre
• Daniela Rus

Autonomous driving presents a complex challenge, which is usually addressed with artificial intelligence models that are end-to-end or modular in nature. Within the landscape of modular approaches, a bio-inspired neural circuit policy model has emerged as an innovative control module, offering a compact and inherently interpretable system to infer a steering wheel command from abstract visual features. Here, we take a leap forward by integrating a variational autoencoder with the neural circuit policy controller, forming a solution that directly generates steering commands from input camera images. By substituting the traditional convolutional neural network approach to feature extraction with a variational autoencoder, we enhance the system’s interpretability, enabling a more transparent and understandable decision-making process. In addition to the architectural shift toward a variational autoencoder, this study introduces the automatic latent perturbation tool, a novel contribution designed to probe and elucidate the latent features within the variational autoencoder. The automatic latent perturbation tool automates the interpretability process, offering granular insights into how specific latent variables influence the overall model’s behavior. Through a series of numerical experiments, we demonstrate the interpretative power of the variational autoencoder-neural circuit policy model and the utility of the automatic latent perturbation tool in making the inner workings of autonomous driving systems more transparent.

ICML Conference 2024 Conference Paper

Large Scale Dataset Distillation with Domain Shift

• Noel Loo
• Alaa Maalouf
• Ramin M. Hasani
• Mathias Lechner
• Alexander Amini
• Daniela Rus

Dataset Distillation seeks to summarize a large dataset by generating a reduced set of synthetic samples. While there has been much success at distilling small datasets such as CIFAR-10 on smaller neural architectures, Dataset Distillation methods fail to scale to larger high-resolution datasets and architectures. In this work, we introduce D ataset D istillation with D omain S hift ( D3S ), a scalable distillation algorithm, made by reframing the dataset distillation problem as a domain shift one. In doing so, we derive a universal bound on the distillation loss, and provide a method for efficiently approximately optimizing it. We achieve state-of-the-art results on Tiny-ImageNet, ImageNet-1k, and ImageNet-21K over a variety of recently proposed baselines, including high cross-architecture generalization. Additionally, our ablation studies provide lessons on the importance of validation-time hyperparameters on distillation performance, motivating the need for standardization.

IROS Conference 2024 Conference Paper

Learning autonomous driving from aerial imagery

• Varun Murali
• Guy Rosman
• Sertac Karaman
• Daniela Rus

In this work, we consider the problem of learning end to end perception to control for ground vehicles solely from aerial imagery. Photogrammetric simulators allow the synthesis of novel views through the transformation of pre-generated assets into novel views. However, they have a large setup cost, require careful collection of data and often human effort to create usable simulators. We use a Neural Radiance Field (NeRF) as an intermediate representation to synthesize novel views from the point of view of a ground vehicle. These novel viewpoints can then be used for several downstream autonomous navigation applications. In this work, we demonstrate the utility of novel view synthesis though the application of training a policy for end to end learning from images and depth data. In a traditional real to sim to real framework, the collected data would be transformed into a visual simulator which could then be used to generate novel views. In contrast, using a NeRF allows a compact representation and the ability to optimize over the parameters of the visual simulator as more data is gathered in the environment. We demonstrate the efficacy of our method in a custom built mini-city environment through the deployment of imitation policies on robotic cars. We additionally consider the task of place localization and demonstrate that our method is able to relocalize the car in the real world.

ICRA Conference 2024 Conference Paper

Learning with Chemical versus Electrical Synapses Does it Make a Difference?

• Mónika Farsang
• Mathias Lechner
• David Lung
• Ramin M. Hasani
• Daniela Rus
• Radu Grosu

Bio-inspired neural networks have the potential to advance our understanding of neural computation and improve the state-of-the-art of AI systems. Bio-electrical synapses directly transmit neural signals, by enabling fast current flow between neurons. In contrast, bio-chemical synapses transmit neural signals indirectly, through neurotransmitters. Prior work showed that interpretable dynamics for complex robotic control, can be achieved by using chemical synapses, within a sparse, bio-inspired architecture, called Neural Circuit Policies (NCPs). However, a comparison of these two synaptic models, within the same architecture, remains an unexplored area. In this work we aim to determine the impact of using chemical synapses compared to electrical synapses, in both sparse and all-to-all connected networks. We conduct experiments with autonomous lane-keeping through a photorealistic autonomous driving simulator to evaluate their performance under diverse conditions and in the presence of noise. The experiments highlight the substantial influence of the architectural and synaptic-model choices, respectively. Our results show that employing chemical synapses yields noticeable improvements compared to electrical synapses, and that NCPs lead to better results in both synaptic models.

ICLR Conference 2024 Conference Paper

Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control

• Neehal Tumma
• Mathias Lechner
• Noel Loo
• Ramin M. Hasani
• Daniela Rus

Developing autonomous agents that can interact with changing environments is an open challenge in machine learning. Robustness is particularly important in these settings as agents are often fit offline on expert demonstrations but deployed online where they must generalize to the closed feedback loop within the environment. In this work, we explore the application of recurrent neural networks to tasks of this nature and understand how a parameterization of their recurrent connectivity influences robustness in closed-loop settings. Specifically, we represent the recurrent connectivity as a function of rank and sparsity and show both theoretically and empirically that modulating these two variables has desirable effects on network dynamics. The proposed low-rank, sparse connectivity induces an interpretable prior on the network that proves to be most amenable for a class of models known as closed-form continuous-time neural networks (CfCs). We find that CfCs with fewer parameters can outperform their full-rank, fully-connected counterparts in the online setting under distribution shift. This yields memory-efficient and robust agents while opening a new perspective on how we can modulate network dynamics through connectivity.

ICRA Conference 2024 Conference Paper

Liquids Identification and Manipulation via Digitally Fabricated Impedance Sensors

• Junyi Zhu 0001
• Young Joong Lee
• Yiyue Luo
• Tianyu Xu 0008
• Chao Liu 0021
• Daniela Rus
• Stefanie Müller 0001
• Wojciech Matusik

Despite recent exponential advancements in computer vision and reinforcement learning, it remains challenging for robots to interact with liquids. These challenges are particularly pronounced due to the limitations imposed by opaque containers, transparent liquids, fine-grained splashes, and visual obstructions arising from the robot’s own manipulation activities. Yet, there exists a substantial opportunity for robotics to excel in liquid identification and manipulation, given its potential role in chemical handling in laboratories and various manufacturing sectors such as pharmaceuticals or beverages. In this work, we present a novel approach for liquid class identification and state estimation leveraging electrical impedance sensing. We design and mount a digitally embroidered electrode array to a commercial robot gripper. Coupled with a customized impedance sensing board, we collect data on liquid manipulation with a swept frequency sensing mode and a frequency-specific impedance measuring mode. Our developed learning-based model achieves an accuracy of 93. 33% in classifying 9 different types of liquids (8 liquids + air), and 97. 65% in estimating the liquid state. We investigate the effectiveness of our system with a series of ablation studies. These findings highlight our work as a promising solution for enhancing robotic manipulation in liquid-related tasks.

ICML Conference 2024 Conference Paper

LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery

• Pingchuan Ma 0002
• Tsun-Hsuan Wang
• Minghao Guo
• Zhiqing Sun
• Joshua B. Tenenbaum
• Daniela Rus
• Chuang Gan 0001
• Wojciech Matusik

Large Language Models have recently gained significant attention in scientific discovery for their extensive knowledge and advanced reasoning capabilities. However, they encounter challenges in effectively simulating observational feedback and grounding it with language to propel advancements in physical scientific discovery. Conversely, human scientists undertake scientific discovery by formulating hypotheses, conducting experiments, and revising theories through observational analysis. Inspired by this, we propose to enhance the knowledge-driven, abstract reasoning abilities of LLMs with the computational strength of simulations. We introduce Scientific Generative Agent (SGA), a bilevel optimization framework: LLMs act as knowledgeable and versatile thinkers, proposing scientific hypotheses and reason about discrete components, such as physics equations or molecule structures; meanwhile, simulations function as experimental platforms, providing observational feedback and optimizing via differentiability for continuous parts, such as physical parameters. We conduct extensive experiments to demonstrate our framework’s efficacy in constitutive law discovery and molecular design, unveiling novel solutions that differ from conventional human expectations yet remain coherent upon analysis.

ICRA Conference 2024 Conference Paper

Modeling and Control of Intrinsically Elasticity Coupled Soft-Rigid Robots

• Zach J. Patterson
• Cosimo Della Santina
• Daniela Rus

While much work has been done recently in the realm of model-based control of soft robots and soft-rigid hybrids, most works examine robots that have an inherently serial structure. While these systems have been prevalent in the literature, there is an increasing trend toward designing soft-rigid hybrids with intrinsically coupled elasticity between various degrees of freedom. In this work, we seek to address the issues of modeling and controlling such structures, particularly when underactuated. We introduce several simple models for elastic coupling, typical of those seen in these systems. We then propose a controller that compensates for the elasticity, and we prove its stability with Lyapunov methods without relying on the elastic dominance assumption. This controller is applicable to the general class of underactuated soft robots. After evaluating the controller in simulated cases, we then develop a simple hardware platform to evaluate both the models and the controller. Finally, using the hardware, we demonstrate a novel use case for underactuated, elastically coupled systems in "sensorless" force control.

ICRA Conference 2024 Conference Paper

Overparametrization helps offline-to-online generalization of closed-loop control from pixels

• Mathias Lechner
• Ramin M. Hasani
• Alexander Amini
• Tsun-Hsuan Wang
• Thomas A. Henzinger
• Daniela Rus

There is an ever-growing zoo of modern neural network models that can efficiently learn end-to-end control from visual observations. These advanced deep models, ranging from convolutional to Vision Transformers, from small to gigantic networks, have been extensively tested on offline image classification tasks. In this paper, we study these vision models with respect to the open-loop training to closed-loop generalization abilities, i. e. , deployment realizes a causal feedback loop that is not present during training. This causality gap typically emerges in robotics applications such as autonomous driving, where a network is trained to imitate the control commands of a human. In this setting, two situations arise: 1) Closed-loop testing in-distribution, where the test environment shares properties with those of offline training data. 2) Closed-loop testing under distribution shifts and out-of-distribution. Contrary to recently reported results, we show that under proper training guidelines, all vision architectures perform indistinguishably well on in-distribution deployment, resolving the causality gap. In situation 2, We observe that scale is the strongest factor in improving closed-loop generalization regardless of the choice of the model architecture. Our results predict the trend that in the future we will see larger and larger models being used in offline-training-online-deployment imitation learning tasks in robotic applications.

ICRA Conference 2024 Conference Paper

Reciprocal and Non-Reciprocal Swarmalators with Programmable Locomotion and Formations for Robot Swarms

• Steven Ceron
• Wei Xiao 0003
• Daniela Rus

Natural and robotic swarms often exhibit nonreciprocal interactions; agents do not exhibit equal and opposite forces on each other. By studying the effects of reciprocal and non-reciprocal interactions we are better able to design emergent behaviors in robot collectives composed of agents that exert attractive and repulsive forces on each other. Moreover, by controlling agent-specific coupling forces on-demand, we can enable a collective to exhibit desired behaviors previously not possible. We use a general form of the swarming oscillator, swarmalator, model to study reciprocal and non-reciprocal interactions among agents that affect each other’s motions over long and short distances, we use non-reciprocal coupling to elicit collective locomotion toward or away from target sites, and we use the control barrier function method to optimize the non-reciprocal interactions for a desired spatial formation. This work addresses the interests of the active matter, swarm robotics, and control barrier functions communities and demonstrates various collective behaviors with strong potential to be realized in macro- and micro- length scale robot swarms.

ICRA Conference 2024 Conference Paper

Robust Model Predictive Control with Control Barrier Functions for Autonomous Surface Vessels

• Wei Wang 0078
• Wei Xiao 0003
• Alejandro Gonzalez-Garcia
• Jan Swevers
• Carlo Ratti
• Daniela Rus

In autonomous robot navigation, the trajectories from path planners are considered to be safe regions, and deviations could endanger vessels. Model Predictive Control (MPC) stands as a popular choice for trajectory tracking problems as it naturally addresses operational constraints, such as dynamics and control constraints. Nevertheless, achieving robustness in changing environments like oceans and rivers, which are constantly subject to significant external disturbances, remains an ongoing challenge for MPC. It must consistently keep the system within a predefined safe region (such as a reference trajectory) even in the presence of model inaccuracies and perturbations. To address this challenge, we present a robust model predictive control strategy utilizing Control Barrier Functions (CBFs), which increases the disturbance-rejection abilities. We verify our method on an autonomous surface vessel in simulation and natural waters, both with external disturbances. Specifically, compared with the traditional MPC method, our proposed MPC-CBF strategy reduces tracking errors by 17. 82% and 40. 26% in simulations and field experiments, respectively. Although the control effort slightly increases by 7. 78% and 4. 20%, respectively, these results clearly demonstrate the enhanced resilience of MPC-CBF to disturbances.

IROS Conference 2024 Conference Paper

Strong Compliant Grasps Using a Cable-Driven Soft Gripper

• Gregory Xie
• Lillian Chin
• Byungchul Kim
• Rachel M. Holladay
• Daniela Rus

The natural flexibility of soft robotic grippers allows for versatile and compliant grasping. However, this same flexibility can restrict the gripper’s strength. Striking a balance between compliance and strength is essential for effective soft grippers. In this work, we present Flexible Robust Observant Gripper (FROG), a soft gripper that is both compliant and strong. We describe the mechanical design of the gripper, characterize the soft flexures used in the design, and analyze the grasp forces generated by the gripper. Utilizing the structure of the gripper, we develop feedforward grasp controllers and a classifier to distinguish between grasp types. Grasping experiments show that FROG can effectively grasp a variety of objects, including very soft or delicate items. Holding force tests show that our gripper can conform to the grasped object and exert large grasp forces.

IROS Conference 2024 Conference Paper

Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Models

• Phat Nguyen
• Tsun-Hsuan Wang
• Zhang-Wei Hong
• Sertac Karaman
• Daniela Rus

Generating varied scenarios through simulation is crucial for training and evaluating safety-critical systems, such as autonomous vehicles. Yet, the task of modeling the trajectories of other vehicles to simulate diverse and meaningful close interactions remains prohibitively costly. Adopting language descriptions to generate driving behaviors emerges as a promising strategy, offering a scalable and intuitive method for human operators to simulate a wide range of driving interactions. However, the scarcity of large-scale annotated language-trajectory data makes this approach challenging. To address this gap, we propose Text-to-Drive (T2D) to synthesize diverse driving behaviors via Large Language Models (LLMs). We introduce a knowledge-driven approach that operates in two stages. In the first stage, we employ the embedded knowledge of LLMs to generate diverse language descriptions of driving behaviors for a scene. Then, we leverage LLM’s reasoning capabilities to synthesize these behaviors in simulation. At its core, T2D employs an LLM to construct a state chart that maps low-level states to high-level abstractions. This strategy aids in downstream tasks such as summarizing low-level observations, assessing policy alignment with behavior description, and shaping the auxiliary reward, all without needing human supervision. With our knowledge-driven approach, we demonstrate that T2D generates more diverse trajectories compared to other baselines and offers a natural language interface that allows for interactive incorporation of human preference. Please check our website for more examples: here

ICRA Conference 2024 Conference Paper

Towards Centimeter-Scale Underwater Mobile Robots: An Architecture for Capable µAUVs

• Pascal Spino
• Daniela Rus

Underwater robots are indispensable for aquatic exploration, yet their size and complexity often limit broader application. This research presents a pioneering micro autonomous underwater vehicle (µAUV) design. This robot is distinguished by its utilization of mass-produced drone components, novel jet propulsion mechanisms, and multifunctional spherical shell. Its architecture is modular, appendage-free, and largely seal-free. Preliminary tests highlight its motion capabilities and set new benchmarks for centimeter-scale µAUV advancements.

ICLR Conference 2024 Conference Paper

Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation

• Noel Loo
• Ramin M. Hasani
• Mathias Lechner
• Alexander Amini
• Daniela Rus

Modern deep learning requires large volumes of data, which could contain sensitive or private information that cannot be leaked. Recent work has shown for homogeneous neural networks a large portion of this training data could be reconstructed with only access to the trained network parameters. While the attack was shown to work empirically, there exists little formal understanding of its effective regime and which datapoints are susceptible to reconstruction. In this work, we first build a stronger version of the dataset reconstruction attack and show how it can provably recover the \emph{entire training set} in the infinite width regime. We then empirically study the characteristics of this attack on two-layer networks and reveal that its success heavily depends on deviations from the frozen infinite-width Neural Tangent Kernel limit. Next, we study the nature of easily-reconstructed images. We show that both theoretically and empirically, reconstructed images tend to ``outliers'' in the dataset, and that these reconstruction attacks can be used for \textit{dataset distillation}, that is, we can retrain on reconstructed images and obtain high predictive accuracy.

ICLR Conference 2024 Conference Paper

Unveiling the Unseen: Identifiable Clusters in Trained Depthwise Convolutional Kernels

• Zahra Babaiee
• Peyman M. Kiasari
• Daniela Rus
• Radu Grosu

Recent advances in depthwise-separable convolutional neural networks (DS-CNNs) have led to novel architectures, that surpass the performance of classical CNNs, by a considerable scalability and accuracy margin. This paper reveals another striking property of DS-CNN architectures: discernible and explainable patterns emerge in their trained depthwise convolutional kernels in all layers. Through an extensive analysis of millions of trained filters, with different sizes and from various models, we employed unsupervised clustering with autoencoders, to categorize these filters. Astonishingly, the patterns converged into a few main clusters, each resembling the difference of Gaussian (DoG) functions, and their first and second-order derivatives. Notably, we classify over 95\% and 90\% of the filters from state-of-the-art ConvNeXtV2 and ConvNeXt models, respectively. This finding is not merely a technological curiosity; it echoes the foundational models neuroscientists have long proposed for the vision systems of mammals. Our results thus deepen our understanding of the emergent properties of trained DS-CNNs and provide a bridge between artificial and biological visual processing systems. More broadly, they pave the way for more interpretable and biologically-inspired neural network designs in the future.

IROS Conference 2024 Conference Paper

Wirelessly Actuated Rotation-free Magnetic Motor

• Umur Ulas Harman
• Ahmed Hafez
• Cameron Duffield
• Zihan Zhao
• Luke Dixon
• Daniela Rus
• Shuhei Miyashita

This paper addresses the challenge of actuating millimetre-sized motors, which are wirelessly driven by external magnetic fields. Traditional approaches, relying on rotating magnetic fields, often inadvertently cause the entire robot – especially if it is small and lightweight – to rotate, instead of a specified shaft in the motor. To overcome this issue, our study introduces a novel mechanism that leverages symmetrically configured magnetic motors to cancel out the torques, thus preventing unwanted rotation of the robot. This is achieved by utilizing a magnetic field along a single axis to induce rotational movement. The design features two millimetre-sized rotating magnets that interact to achieve a 90 ◦ rotation, complemented by an external magnetic field that accomplishes the remaining 270 ◦, thus completing a full rotation. Furthermore, we demonstrate that applying a perpendicularly oriented magnetic field can inversely affect the motor’s rotation direction. A proof-of-concept experiment employing this mechanism successfully actuated a gripper in a water tank while it is free-floating, showcasing its potential for enhancing robotic applications at the sub-centimeter scale, where the small net torque of a miniature motor is essential.

IROS Conference 2023 Conference Paper

A Fabrication and Simulation Recipe for Untethering Soft-Rigid Robots with Cable-Driven Stiffness Modulation

• James M. Bern
• Zach J. Patterson
• Leonardo Zamora Yañez
• Kristoff K. Misquitta
• Daniela Rus

We explore the idea of robotic mechanisms that can shift between soft and rigid states, with the long-term goal of creating robots that marry the flexibility and robustness of soft robots with the strength and precision of rigid robots. We present a simple yet effective method to achieve large and rapid stiffness variations by compressing and relaxing a flexure using cables. Next, we provide a differentiable modeling framework that can be used for motion planning, which simultaneously reasons about the modulated stiffness joints, tendons, rigid joints, and basic hydrodynamics. We apply this stiffness tuning and simulation recipe to create SoRiTu, an untethered soft-rigid robotic sea turtle capable of various swimming maneuvers.

ICML Conference 2023 Conference Paper

AutoCoreset: An Automatic Practical Coreset Construction Framework

• Alaa Maalouf
• Murad Tukan
• Vladimir Braverman
• Daniela Rus

A coreset is a small weighted subset of an input set that approximates its loss function, for a given set of queries. Coresets became prevalent in machine learning as they have shown to be advantageous for many applications. Unfortunately, coresets are constructed in a problem-dependent manner, where for each problem, a new coreset construction algorithm is suggested, taking years to prove its correctness. Even the generic frameworks require additional (problem-dependent) computations or proofs to be done by the user. Besides, many problems do not have (provable) small coresets, limiting their applicability. To this end, we suggest an automatic practical framework for constructing coresets, which requires (only) the input data and the desired cost function from the user, without the need for any other task-related computation to be done by the user. To do so, we reduce the problem of approximating a loss function to an instance of vector summation approximation, where the vectors we aim to sum are loss vectors of a specific subset of the queries, such that we aim to approximate the image of the function on this subset. We show that while this set is limited, the coreset is quite general. An extensive experimental study on various machine learning applications is also conducted. Finally, we provide a “plug and play" style implementation, proposing a user-friendly system that can be easily used to apply coresets for many problems. We believe that these contributions enable future research and easier use and applications of coresets.

ICRA Conference 2023 Conference Paper

BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation

• Zhijian Liu
• Haotian Tang
• Alexander Amini
• Xinyu Yang 0002
• Huizi Mao
• Daniela Rus
• Song Han 0003

Multi-sensor fusion is essential for an accurate and reliable autonomous driving system. Recent approaches are based on point-level fusion: augmenting the LiDAR point cloud with camera features. However, the camera-to-LiDAR projection throws away the semantic density of camera features, hindering the effectiveness of such methods, especially for semantic-oriented tasks (such as 3D scene segmentation). In this paper, we propose BEVFusion, an efficient and generic multi-task multi-sensor fusion framework. It unifies multi-modal features in the shared bird's-eye view (BEV) representation space, which nicely preserves both geometric and semantic information. To achieve this, we diagnose and lift the key efficiency bottlenecks in the view transformation with optimized BEV pooling, reducing latency by more than $\mathbf{40}\times$. BEVFusion is fundamentally task-agnostic and seamlessly supports different 3D perception tasks with almost no architectural changes. It establishes the new state of the art on the nuScenes benchmark, achieving 1. 3% higher mAP and NDS on 3D object detection and 13. 6% higher mIoU on BEV map segmentation, with 1. 9× lower computation cost. Code to reproduce our results is available at https://github.com/mit-han-lab/bevfusion.

ICML Conference 2023 Conference Paper

Dataset Distillation with Convexified Implicit Gradients

• Noel Loo
• Ramin M. Hasani
• Mathias Lechner
• Daniela Rus

We propose a new dataset distillation algorithm using reparameterization and convexification of implicit gradients (RCIG), that substantially improves the state-of-the-art. To this end, we first formulate dataset distillation as a bi-level optimization problem. Then, we show how implicit gradients can be effectively used to compute meta-gradient updates. We further equip the algorithm with a convexified approximation that corresponds to learning on top of a frozen finite-width neural tangent kernel. Finally, we improve bias in implicit gradients by parameterizing the neural network to enable analytical computation of final-layer parameters given the body parameters. RCIG establishes the new state-of-the-art on a diverse series of dataset distillation tasks. Notably, with one image per class, on resized ImageNet, RCIG sees on average a 108% improvement over the previous state-of-the-art distillation algorithm. Similarly, we observed a 66% gain over SOTA on Tiny-ImageNet and 37% on CIFAR-100.

ICRA Conference 2023 Conference Paper

Deep Learning on Home Drone: Searching for the Optimal Architecture

• Alaa Maalouf
• Yotam Gurfinkel
• Barak Diker
• Oren Gal
• Daniela Rus
• Dan Feldman

We suggest the first system that runs real-time semantic segmentation via deep learning on the weak microcomputer Raspberry Pi Zero v2 (whose price was $15) attached to a toy drone. In particular, since the Raspberry Pi weighs less than 16 grams, and its size is half of a credit card, we could easily attach it to the common commercial DJI Tello toy-drone ( $\times 92. 5\times 41$ mm). The result is an autonomous drone (no laptop nor human in the loop) that can detect and classify objects in real-time from a video stream of an onboard monocular RGB camera (no GPS or LIDAR sensors). The companion videos demonstrate how this Tello drone scans the lab for people (e. g. for the use of firefighters or security forces) and for an empty parking slot outside the lab. Existing deep learning solutions are either much too slow for real-time computation on such IoT devices, or provide results of impractical quality. Our main challenge was to design a system that takes the best of all worlds among numerous combinations of networks, deep learning platforms/frameworks, compression techniques, and compression ratios. To this end, we provide an efficient searching algorithm that aims to find the optimal combination which results in the best tradeoff between the network running time and its accuracy/performance.

ICRA Conference 2023 Conference Paper

Deep Reinforcement Learning Based Tracking Control of an Autonomous Surface Vessel in Natural Waters

• Wei Wang 0078
• Xiaojing Cao
• Alejandro Gonzalez-Garcia
• Lianhao Yin
• Niklas Hagemann
• Yuanyuan Qiao 0002
• Carlo Ratti
• Daniela Rus

Accurate control of autonomous marine robots still poses challenges due to the complex dynamics of the environment. In this paper, we propose a Deep Reinforcement Learning (DRL) approach to train a controller for autonomous surface vessel (ASV) trajectory tracking and compare its performance with an advanced nonlinear model predictive controller (NMPC) in real environments. Taking into account environmental disturbances (e. g. , wind, waves, and currents), noisy measurements, and non-ideal actuators presented in the physical ASV, several effective reward functions for DRL tracking control policies are carefully designed. The control policies were trained in a simulation environment with diverse tracking trajectories and disturbances. The performance of the DRL controller has been verified and compared with the NMPC in both simulations with model-based environmental disturbances and in natural waters. Simulations show that the DRL controller has 53. 33% lower tracking error than that of NMPC. Experimental results further show that, compared to NMPC, the DRL controller has 35. 51% lower tracking error, indicating that DRL controllers offer better disturbance rejection in river environments than NMPC.

NeurIPS Conference 2023 Conference Paper

DiffuseBot: Breeding Soft Robots With Physics-Augmented Generative Diffusion Models

• Tsun-Hsuan Johnson Wang
• Juntian Zheng
• Pingchuan Ma
• Yilun Du
• Byungchul Kim
• Andrew Spielberg
• Josh Tenenbaum
• Chuang Gan

Nature evolves creatures with a high complexity of morphological and behavioral intelligence, meanwhile computational methods lag in approaching that diversity and efficacy. Co-optimization of artificial creatures' morphology and control in silico shows promise for applications in physical soft robotics and virtual character creation; such approaches, however, require developing new learning algorithms that can reason about function atop pure structure. In this paper, we present DiffuseBot, a physics-augmented diffusion model that generates soft robot morphologies capable of excelling in a wide spectrum of tasks. \name bridges the gap between virtually generated content and physical utility by (i) augmenting the diffusion process with a physical dynamical simulation which provides a certificate of performance, and (ii) introducing a co-design procedure that jointly optimizes physical design and control by leveraging information about physical sensitivities from differentiable simulation. We showcase a range of simulated and fabricated robots along with their capabilities. Check our website: https: //diffusebot. github. io/

NeurIPS Conference 2023 Conference Paper

Gigastep - One Billion Steps per Second Multi-agent Reinforcement Learning

• Mathias Lechner
• lianhao yin
• Tim Seyde
• Tsun-Hsuan Johnson Wang
• Wei Xiao
• Ramin Hasani
• Joshua Rountree
• Daniela Rus

Multi-agent reinforcement learning (MARL) research is faced with a trade-off: it either uses complex environments requiring large compute resources, which makes it inaccessible to researchers with limited resources, or relies on simpler dynamics for faster execution, which makes the transferability of the results to more realistic tasks challenging. Motivated by these challenges, we present Gigastep, a fully vectorizable, MARL environment implemented in JAX, capable of executing up to one billion environment steps per second on consumer-grade hardware. Its design allows for comprehensive MARL experimentation, including a complex, high-dimensional space defined by 3D dynamics, stochasticity, and partial observations. Gigastep supports both collaborative and adversarial tasks, continuous and discrete action spaces, and provides RGB image and feature vector observations, allowing the evaluation of a wide range of MARL algorithms. We validate Gigastep's usability through an extensive set of experiments, underscoring its role in widening participation and promoting inclusivity in the MARL research community.

ICRA Conference 2023 Conference Paper

Infrastructure-based End-to-End Learning and Prevention of Driver Failure

• Noam Buckman
• Shiva Sreeram
• Mathias Lechner
• Yutong Ban
• Ramin M. Hasani
• Sertac Karaman
• Daniela Rus

Intelligent intersection managers can improve safety by detecting dangerous drivers or failure modes in autonomous vehicles, warning oncoming vehicles as they approach an intersection. In this work, we present FailureNet, a recurrent neural network trained end-to-end on trajectories of both nominal and reckless drivers in a scaled miniature city. FailureNet observes the poses of vehicles as they approach an intersection and detects whether a failure is present in the autonomy stack, warning cross-traffic of potentially dangerous drivers. FailureNet can accurately identify control failures, upstream perception errors, and speeding drivers, distinguishing them from nominal driving. The network is trained and deployed with autonomous vehicles in the MiniCity. Compared to speed or frequency-based predictors, FailureNet's recurrent neural network structure provides improved predictive power, yielding upwards of 84% accuracy when deployed on hardware.

ICRA Conference 2023 Conference Paper

Learned Risk Metric Maps for Kinodynamic Systems

• Ross E. Allen
• Wei Xiao 0003
• Daniela Rus

We present Learned Risk Metric Maps (LRMM) for real-time estimation of coherent risk metrics of high-dimensional dynamical systems operating in unstructured, partially observed environments. LRMM models are simple to design and train-requiring only procedural generation of obstacle sets, state and control sampling, and supervised training of a function approximator-which makes them broadly applicable to arbitrary system dynamics and obstacle sets. In a parallel autonomy setting, we demonstrate the model's ability to rapidly infer collision probabilities of a fast-moving car-like robot driving recklessly in an obstructed environment; allowing the LRMM agent to intervene, take control of the vehicle, and avoid collisions. In this time-critical scenario, we show that LRMMs can evaluate risk metrics 20-100x times faster than alternative safety algorithms based on control barrier functions (CBFs) and Hamilton-Jacobi reachability (HJ-reach), leading to 5–15 % fewer obstacle collisions by the LRMM agent than CBFs and HJ-reach. This performance improvement comes in spite of the fact that the LRMM model only has access to local/partial observation of obstacles, whereas the CBF and HJ-reach agents are granted privileged/global information. We also show that our model can be equally well trained on a 12-dimensional quadrotor system operating in an obstructed indoor environment. The LRMM codebase is provided at https://github.com/mit-drl/pyrmm.

IROS Conference 2023 Conference Paper

LiDAR Missing Measurement Detection for Autonomous Driving in Rain

• Chen Zhang 0018
• Zefan Huang
• Marcelo H. Ang
• Daniela Rus

Autonomous driving in rain remains challenging. Rain causes sensor performance degradation that can affect sensor measurement quality. During the rain, lasers may suffer from energy loss due to raindrop absorption. As a result, some laser measurements reflected from obstacles may not be recognized by the LiDAR sensor, thus raising potential risks for autonomous vehicles. This work investigates a novel task that aims to detect those missing measurements. Our solution uses a two-stage learning method to generate an anomaly score for each missing measurement, representing the likelihood of being caused by rain. We evaluate our method with real-world data and demonstrate its effectiveness in identifying anomalous missing measurements through qualitative and quantitative experiments.

ICLR Conference 2023 Conference Paper

Liquid Structural State-Space Models

• Ramin M. Hasani
• Mathias Lechner
• Tsun-Hsuan Wang
• Makram Chahine
• Alexander Amini
• Daniela Rus

A proper parametrization of state transition matrices of linear state-space models (SSMs) followed by standard nonlinearities enables them to efficiently learn representations from sequential data, establishing the state-of-the-art on an extensive series of long-range sequence modeling benchmarks. In this paper, we show that we can improve further when the structured SSM, such as S4, is given by a linear liquid time-constant (LTC) state-space model. LTC neural networks are causal continuous-time neural networks with an input-dependent state transition module, which makes them learn to adapt to incoming inputs at inference. We show that by using a diagonal plus low-rank decomposition of the state transition matrix introduced in S4, and a few simplifications, the LTC-based structured state-space model, dubbed Liquid-S4, improves generalization across sequence modeling tasks with long-term dependencies such as image, text, audio, and medical time-series, with an average performance of 87.32\% on the Long-Range Arena benchmark. On the full raw Speech Command recognition dataset, Liquid-S4 achieves 96.78\% accuracy with a 30\% reduction in parameter counts compared to S4. The additional gain in performance is the direct result of the Liquid-S4's kernel structure that takes into account the similarities of the input sequence samples during training and inference.

IROS Conference 2023 Conference Paper

Local Non-Cooperative Games with Principled Player Selection for Scalable Motion Planning

• Makram Chahine
• Roya Firoozi
• Wei Xiao 0003
• Mac Schwager
• Daniela Rus

Game-theoretic motion planners are a powerful tool for the control of interactive multi-agent robot systems. Indeed, contrary to predict-then-plan paradigms, game-theoretic planners do not ignore the interactive nature of the problem, and simultaneously predict the behaviour of other agents while considering change in one's policy. This, however, comes at the expense of computational complexity, especially as the number of agents considered grows. In fact, planning with more than a handful of agents can quickly become intractable, disqualifying game-theoretic planners as possible candidates for large scale planning. In this paper, we propose a planning algorithm enabling the use of game-theoretic planners in robot systems with a large number of agents. Our planner is based on the reality of locality of information and thus deploys local games with a selected subset of agents in a receding horizon fashion to plan collision avoiding trajectories. We propose five different principled schemes for selecting game participants and compare their collision avoidance performance. We observe that the use of Control Barrier Functions for priority ranking is a potent solution to the player selection problem for motion planning.

IROS Conference 2023 Conference Paper

Machine Learning Best Practices for Soft Robot Proprioception

• Annan Zhang
• Tsun-Hsuan Wang
• Ryan L. Truby
• Lillian Chin
• Daniela Rus

Machine learning-based approaches for soft robot proprioception have recently gained popularity, in part due to the difficulties in modeling the relationship between sensor signals and robot shape. However, to date, there exists no systematic analysis of the required design choices to set up a machine learning pipeline for soft robot proprioception. Here, we present the first study examining how design choices on different levels of the machine learning pipeline affect the performance of a neural network for predicting the state of a soft robot. We address the most frequent questions researchers face, such as how to choose the appropriate sensor and actuator signals, process input and output data, deal with time series, and pick the best neural network architecture. By testing our hypotheses on data collected from two vastly different systems–an electrically actuated robotic platform and a pneumatically actuated soft trunk–we seek conclusions that may generalize beyond one specific type of soft robot and hope to provide insights for researchers to use machine learning for soft robot proprioception.

ICML Conference 2023 Conference Paper

On the Forward Invariance of Neural ODEs

• Wei Xiao 0003
• Tsun-Hsuan Wang
• Ramin M. Hasani
• Mathias Lechner
• Yutong Ban
• Chuang Gan 0001
• Daniela Rus

We propose a new method to ensure neural ordinary differential equations (ODEs) satisfy output specifications by using invariance set propagation. Our approach uses a class of control barrier functions to transform output specifications into constraints on the parameters and inputs of the learning system. This setup allows us to achieve output specification guarantees simply by changing the constrained parameters/inputs both during training and inference. Moreover, we demonstrate that our invariance set propagation through data-controlled neural ODEs not only maintains generalization performance but also creates an additional degree of robustness by enabling causal manipulation of the system’s parameters/inputs. We test our method on a series of representation learning tasks, including modeling physical dynamics and convexity portraits, as well as safe collision avoidance for autonomous vehicles.

NeurIPS Conference 2023 Conference Paper

On the Size and Approximation Error of Distilled Datasets

• Alaa Maalouf
• Murad Tukan
• Noel Loo
• Ramin Hasani
• Mathias Lechner
• Daniela Rus

Dataset Distillation is the task of synthesizing small datasets from large ones while still retaining comparable predictive accuracy to the original uncompressed dataset. Despite significant empirical progress in recent years, there is little understanding of the theoretical limitations/guarantees of dataset distillation, specifically, what excess risk is achieved by distillation compared to the original dataset, and how large are distilled datasets? In this work, we take a theoretical view on kernel ridge regression (KRR) based methods of dataset distillation such as Kernel Inducing Points. By transforming ridge regression in random Fourier features (RFF) space, we provide the first proof of the existence of small (size) distilled datasets and their corresponding excess risk for shift-invariant kernels. We prove that a small set of instances exists in the original input space such that its solution in the RFF space coincides with the solution of the original data. We further show that a KRR solution can be generated using this distilled set of instances which gives an approximation towards the KRR solution optimized on the full input data. The size of this set is linear in the dimension of the RFF space of the input set or alternatively near linear in the number of effective degrees of freedom, which is a function of the kernel, number of data points, and the regularization parameter $\lambda$. The error bound of this distilled set is also a function of $\lambda$. We verify our bounds analytically and empirically.

ICML Conference 2023 Conference Paper

Provable Data Subset Selection For Efficient Neural Networks Training

• Murad Tukan
• Samson Zhou
• Alaa Maalouf
• Daniela Rus
• Vladimir Braverman
• Dan Feldman

Radial basis function neural networks ( RBFNN ) are well-known for their capability to approximate any continuous function on a closed bounded set with arbitrary precision given enough hidden neurons. In this paper, we introduce the first algorithm to construct coresets for RBFNNs, i. e. , small weighted subsets that approximate the loss of the input data on any radial basis function network and thus approximate any function defined by an RBFNN on the larger input data. In particular, we construct coresets for radial basis and Laplacian loss functions. We then use our coresets to obtain a provable data subset selection algorithm for training deep neural networks. Since our coresets approximate every function, they also approximate the gradient of each weight in a neural network, which is a particular function on the input. We then perform empirical evaluations on function approximation and dataset subset selection on popular network architectures and data sets, demonstrating the efficacy and accuracy of our coreset construction.

AAAI Conference 2023 Conference Paper

Quantization-Aware Interval Bound Propagation for Training Certifiably Robust Quantized Neural Networks

• Mathias Lechner
• Đorđe Žikelić
• Krishnendu Chatterjee
• Thomas A. Henzinger
• Daniela Rus

We study the problem of training and certifying adversarially robust quantized neural networks (QNNs). Quantization is a technique for making neural networks more efficient by running them using low-bit integer arithmetic and is therefore commonly adopted in industry. Recent work has shown that floating-point neural networks that have been verified to be robust can become vulnerable to adversarial attacks after quantization, and certification of the quantized representation is necessary to guarantee robustness. In this work, we present quantization-aware interval bound propagation (QA-IBP), a novel method for training robust QNNs. Inspired by advances in robust learning of non-quantized networks, our training algorithm computes the gradient of an abstract representation of the actual network. Unlike existing approaches, our method can handle the discrete semantics of QNNs. Based on QA-IBP, we also develop a complete verification procedure for verifying the adversarial robustness of QNNs, which is guaranteed to terminate and produce a correct answer. Compared to existing approaches, the key advantage of our verification procedure is that it runs entirely on GPU or other accelerator devices. We demonstrate experimentally that our approach significantly outperforms existing methods and establish the new state-of-the-art for training and certifying the robustness of QNNs.

ICRA Conference 2023 Conference Paper

Risk-Aware Neural Navigation From BEV Input for Interactive Driving

• Suzanna Jiwani
• Xiao Li 0025
• Sertac Karaman
• Daniela Rus

Safety has been a key goal for autonomous driving since its inception, and we believe recognizing and responding to risk is a key component of safety. In this work, we aim to answer the question, “How can explainable risk representations be generated and used to produce risk-averse trajectories? ” To answer this question, previous work uses risk metrics to formulate an optimization problem. In contrast, our work is based on research showing the usefulness of grids as a representation to generate image-based risk maps through a trained neural network. We propose a method of determining risk from a bird's eye view (BEV) of an autonomous vehicle's surroundings. Our method consists of (1) a risk map generator, which is trained to recognize risk associated with nearby agents and the map, (2) differentiable value iteration using the risk map to learn a policy, and (3) a trajectory sampler, which samples from this policy to generate a trajectory. We evaluate our planner in a close-loop manner and find improvements in its overall ability to mimic human driving while maintaining comparable safety statistics. Self-ablation also reveals the potential for fine-tuning the behavior of the planner given a designer's needs.

IROS Conference 2023 Conference Paper

Roblets: Robotic Tablets That Self-Assemble and Self-Fold into a Robot

• Junyi Han
• Daniela Rus
• Shuhei Miyashita

Inspired by human proteins that are synthesized from only 20 types of amino acids, the development of self-assembly methods that allow robots to be built simply by randomly stirring the parts has been explored for many years. The key challenges include how to synthesize parts in pieces into a three-dimensional functional structure in a practical time, and subsequently, achieve a controlled robotic motion, all with minimal human intervention. This study proposes a method of self-assembling a 3D robot by first self-assembling random parts into a 2D structure and then self-folding it into a 3D shape. Once self-folded, the robot, whose compositional parts contain magnets, becomes capable of performing basic tasks such as block-pushing upon an application of an external magnetic field. Self-assembly from parts into a two-dimensional structure was performed by repeatedly colliding the parts with each other, and combining them with complementary-shaped parts, like matching jigsaw puzzle pieces. Self-folding was performed by shrinking a heat-responsive film attached across the hinge of each assembly part in hot water, causing the entire 2D structure to self-fold. The experiment demonstrated a series of 13 parts self-assembling into the shape of a 3D beetle, then walking and pushing an object in 13 minutes. The self-assembly process is programmed (mechanically) to generate the same geometry even if the number of parts is greater than the necessary number for the structure, thus is capable of generating multiple structures simultaneously.

IROS Conference 2023 Conference Paper

SMART-Degradation: A Dataset for LiDAR Degradation Evaluation in Rain

• Chen Zhang 0018
• Zefan Huang
• Beatrix Xue Lin Tung
• Marcelo H. Ang
• Daniela Rus

Sensor degradation is one of the major challenges for autonomous driving. During the rain, the interference from raindrops can negatively influence LiDAR measurements. For example, valid measurements could be reduced during the rain, and some measurements may become noisy. Unreliable measurements can lead to potential safety issues if autonomous driving systems are unaware of these changes. In this work, we will release a naturalistic driving dataset to advance the research in studying LiDAR degradation. Our dataset consists of 3D LiDAR scans collected by a data collection vehicle under various rainy conditions. Besides these raw scans, we also release LiDAR scan pairs (each pair consists of one scan from rainy weather and one scan from clear weather at the same location). These LiDAR pairs are developed to help researchers identify LiDAR degradation. Finally, we will release a toolbox integrated with mapping, localization, and scan synthesis functions used to create this dataset. This toolbox can facilitate dataset creation for studying degradation in other harsh weather conditions. More information can be found at https://smart-rain-dataset.github.io/.

IROS Conference 2023 Conference Paper

SMART-Rain: A Degradation Evaluation Dataset for Autonomous Driving in Rain

• Chen Zhang 0018
• Zefan Huang
• Hongliang Guo 0003
• Lei Qin
• Marcelo H. Ang
• Daniela Rus

Autonomous driving in the rain remains a challenge. One main problem is performance degradation caused by rain. This work introduces a new dataset to study this problem. Our dataset is collected from a full-scale vehicle equipped with a 3D LiDAR sensor and multiple forward-facing cameras under various rainy conditions. In addition, rainfall intensity is recorded in real-time from a rain sensor. The combination of sensor and rainfall intensity measurement is designed for studying algorithm performance under different levels of rainfall. In this work, in addition to presenting dataset creation details, we also introduce three degradation evaluation tasks with baseline results, including rainfall intensity estimation, LiDAR degradation estimation, and 2D object detection evaluation. This dataset, development kit, and baseline codes will be made available at https://smart-rain-dataset.github.io/

ICRA Conference 2023 Conference Paper

SmartRainNet: Uncertainty Estimation For Laser Measurement in Rain

• Chen Zhang 0018
• Zefan Huang
• Beatrix Xue Lin Tung
• Marcelo H. Ang
• Daniela Rus

Adverse weather has raised a big challenge for autonomous vehicles. Unreliable measurements due to sensor degradation could seriously affect the performance of autonomous driving tasks, such as perception and localization. In this work, we study sensor degradation in rainy weather and present a novel method that evaluates the uncertainty for each laser measurement from a 3D LiDAR. With uncertainty estimation, downstream tasks that rely on LiDAR input (e. g. , perception or localization) can increase their reliability by adjusting their reliance on laser measurements with varying fidelity. Alternatively, uncertainty estimation can be used for sensor performance evaluation. Our proposed method, SmartRainNet, uses an attention-based Mixture Density Network to model the dependence between neighboring laser measurements and then calculate the probability density for each laser measurement as an uncertainty score. We evaluate SmartRainNet on synthetic and naturalistic sensor degradation datasets and provide qualitative and quantitative results to demonstrate the effectiveness of our method in evaluating uncertainty. Finally, we demonstrate three practical applications of uncertainty estimation to address autonomous driving challenges in rainy weather.

ICLR Conference 2023 Conference Paper

SoftZoo: A Soft Robot Co-design Benchmark For Locomotion In Diverse Environments

• Tsun-Hsuan Wang
• Pingchuan Ma 0002
• Andrew Spielberg
• Zhou Xian
• Hao Zhang
• Joshua B. Tenenbaum
• Daniela Rus
• Chuang Gan 0001

While significant research progress has been made in robot learning for control, unique challenges arise when simultaneously co-optimizing morphology. Existing work has typically been tailored for particular environments or representations. In order to more fully understand inherent design and performance tradeoffs and accelerate the development of new breeds of soft robots, a comprehensive virtual platform — with well-established tasks, environments, and evaluation metrics — is needed. In this work, we introduce SoftZoo, a soft robot co-design platform for locomotion in diverse environments. SoftZoo supports an extensive, naturally-inspired material set, including the ability to simulate environments such as flat ground, desert, wetland, clay, ice, snow, shallow water, and ocean. Further, it provides a variety of tasks relevant for soft robotics, including fast locomotion, agile turning, and path following, as well as differentiable design representations for morphology and control. Combined, these elements form a feature-rich platform for analysis and development of soft robot co-design algorithms. We benchmark prevalent representations and co-design algorithms, and shed light on 1) the interplay between environment, morphology, and behavior (2) the importance of design space representations 3) the ambiguity in muscle formation and controller synthesis and 4) the value of differentiable physics. We envision that SoftZoo will serve as a standard platform and template an approach toward the development of novel representations and algorithms for co-designing soft robots’ behavioral and morphological intelligence. Demos are available on our project page.

ICLR Conference 2023 Conference Paper

Solving Continuous Control via Q-learning

• Tim Seyde
• Peter Werner
• Wilko Schwarting
• Igor Gilitschenski
• Martin A. Riedmiller
• Daniela Rus
• Markus Wulfmeier

While there has been substantial success for solving continuous control with actor-critic methods, simpler critic-only methods such as Q-learning find limited application in the associated high-dimensional action spaces. However, most actor-critic methods come at the cost of added complexity: heuristics for stabilisation, compute requirements and wider hyperparameter search spaces. We show that a simple modification of deep Q-learning largely alleviates these issues. By combining bang-bang action discretization with value decomposition, framing single-agent control as cooperative multi-agent reinforcement learning (MARL), this simple critic-only approach matches performance of state-of-the-art continuous actor-critic methods when learning from features or pixels. We extend classical bandit examples from cooperative MARL to provide intuition for how decoupled critics leverage state information to coordinate joint optimization, and demonstrate surprisingly strong performance across a variety of continuous control tasks.

IROS Conference 2023 Conference Paper

Towards Cooperative Flight Control Using Visual-Attention

• Lianhao Yin
• Makram Chahine
• Tsun-Hsuan Wang
• Tim Seyde
• Chao Liu 0021
• Mathias Lechner
• Ramin M. Hasani
• Daniela Rus

The cooperation of a human pilot with an autonomous agent during flight control realizes parallel autonomy. We propose an air-guardian system that facilitates cooperation between a pilot with eye tracking and a parallel end-to-end neural control system. Our vision-based air-guardian system combines a causal continuous-depth neural network model with a cooperation layer to enable parallel autonomy between a pilot and a control system based on perceived differences in their attention profiles. The attention profiles for neural networks are obtained by computing the networks' saliency maps (feature importance) through the VisualBackProp algorithm, while the attention profiles for humans are either obtained by eye tracking of human pilots or saliency maps of networks trained to imitate human pilots. When the attention profile of the pilot and guardian agents align, the pilot makes control decisions. Otherwise, the air-guardian makes interventions and takes over the control of the aircraft. We show that our attention-based air-guardian system can balance the trade-off between its level of involvement in the flight and the pilot's expertise and attention. The guardian system is particularly effective in situations where the pilot was distracted due to information overload. We demonstrate the effectiveness of our method for navigating flight scenarios in simulation with a fixed-wing aircraft and on hardware with a quadrotor platform.

ICRA Conference 2022 Conference Paper

A Deep Concept Graph Network for Interaction-Aware Trajectory Prediction

• Yutong Ban
• Xiao Li 0025
• Guy Rosman
• Igor Gilitschenski
• Ozanan R. Meireles
• Sertac Karaman
• Daniela Rus

Temporal patterns (how vehicles behave in our observed past) underline our reasoning of how people drive on the road, and can explain why we make certain predictions about interactions among road agents. In this paper we propose the ConceptNet trajectory predictor - a novel prediction framework that is able to incorporate agent interactions as explicit edges in a temporal knowledge graph. We demonstrate the sample efficiency and the overall accuracy of the proposed approach, and show that using the graphical structure to explicitly model interactions enables better detection of agent interactions and improved trajectory predictions on a large real-world driving dataset.

NeurIPS Conference 2022 Conference Paper

ActionSense: A Multimodal Dataset and Recording Framework for Human Activities Using Wearable Sensors in a Kitchen Environment

• Joseph DelPreto
• Chao Liu
• Yiyue Luo
• Michael Foshey
• Yunzhu Li
• Antonio Torralba
• Wojciech Matusik
• Daniela Rus

This paper introduces ActionSense, a multimodal dataset and recording framework with an emphasis on wearable sensing in a kitchen environment. It provides rich, synchronized data streams along with ground truth data to facilitate learning pipelines that could extract insights about how humans interact with the physical world during activities of daily living, and help lead to more capable and collaborative robot assistants. The wearable sensing suite captures motion, force, and attention information; it includes eye tracking with a first-person camera, forearm muscle activity sensors, a body-tracking system using 17 inertial sensors, finger-tracking gloves, and custom tactile sensors on the hands that use a matrix of conductive threads. This is coupled with activity labels and with externally-captured data from multiple RGB cameras, a depth camera, and microphones. The specific tasks recorded in ActionSense are designed to highlight lower-level physical skills and higher-level scene reasoning or action planning. They include simple object manipulations (e. g. , stacking plates), dexterous actions (e. g. , peeling or cutting vegetables), and complex action sequences (e. g. , setting a table or loading a dishwasher). The resulting dataset and underlying experiment framework are available at https: //action-sense. csail. mit. edu. Preliminary networks and analyses explore modality subsets and cross-modal correlations. ActionSense aims to support applications including learning from demonstrations, dexterous robot control, cross-modal predictions, and fine-grained action segmentation. It could also help inform the next generation of smart textiles that may one day unobtrusively send rich data streams to in-home collaborative or autonomous robot assistants.

IROS Conference 2022 Conference Paper

Automatic Co-Design of Aerial Robots Using a Graph Grammar

• Allan Zhao
• Tao Du 0001
• Jie Xu 0028
• Josie Hughes
• Juan Salazar
• Pingchuan Ma 0002
• Wei Wang 0078
• Daniela Rus

Unmanned aerial vehicles (UAVs) have broad applications including disaster response, transportation, photography, and mapping. A significant bottleneck in the development of UAVs is the limited availability of automatic tools for task-specific co-design of a UAV's shape and controller. The development of such tools is particularly challenging as UAVs can take many forms, including fixed-wing planes, radial copters, and hybrid topologies, with each class of topology showing different advantages. In this work, we present a computational design pipeline for UAVs based on a graph grammar that can search across a wide range of topologies. Graphs generated by the grammar encode different topologies and component selections, while continuous parameters encode the dimensions and properties of each component. We further augment the shape representation with deformation cages, which allow expressing a variety of wing shapes. Each UAV design is associated with an LQR controller with tunable continuous parameters. To search over this complex discrete and continuous design space, we develop a hybrid algorithm that combines discrete graph search strategies and gradient-based continuous optimization methods using a differentiable UAV simulator. We evaluate our pipeline on a set of simulated flight tasks requiring dynamic motions, showing that it discovers novel UAV designs that outperform canonical UAVs typically made by engineers.

AAMAS Conference 2022 Conference Paper

Autonomous Flight Arcade Challenge: Single- and Multi-Agent Learning Environments for Aerial Vehicles

• Paul Tylkin
• Tsun-Hsuan Wang
• Tim Seyde
• Kyle Palko
• Ross Allen
• Alexander Amini
• Daniela Rus

The Autonomous Flight Arcade (AFA) is a novel suite of singleand multi-agent learning environments for control of aerial vehicles. These environments incorporate realistic physics using the Unity game engine with diverse objectives and levels of decisionmaking sophistication. In addition to the environments themselves, we introduce an interface for interacting with them, including the ability to vary key parameters, thereby both changing the difficulty and the core challenges. We also introduce a pipeline for collecting human gameplay within the environments. We demonstrate the performance of artificial agents in these environments trained using deep reinforcement learning, and also motivate these environments as a benchmark for designing non-learned classical control policies and agents trained using imitation learning from human demonstrations. Finally, we motivate the use of AFA environments as a testbed for training artificial agents capable of cooperative human-AI decision making, including parallel autonomy.

ICRA Conference 2022 Conference Paper

Design of an Autonomous Latching System for Surface Vessels

• David Fernández-Gutiérrez
• Niklas Hagemann
• Wei Wang 0078
• Rens M. Doornbusch
• Joshua Jordan
• Jonathan Klein Schiphorst
• Pietro Leoni
• Fabio Duarte

Autonomous latching is essential for autonomous surface vessels (ASV) to reach full independence from human intervention. As part of the ASV Roboat project, a new solution for self-latching maneuvers has been developed and is presented here. We propose a system that has the key requirements of full integration with the navigation control system and zero-gap connection with the dock, the latter being essential for wireless charging of the ASV. Dedicated markers are used to identify docking targets, relying on computer vision algorithms to determine distance and bearing to the target. In its idle state, the locking solution uses mechanical power-off brakes, minimizing energy consumption while ensuring the boat stays in position indefinitely once docked. A prototype of the proposed mechanism has been built and installed in Roboat. Experimental tests showing the mechanism performance and capability to autonomously approach the docking station are discussed in this work.

NeurIPS Conference 2022 Conference Paper

Efficient Dataset Distillation using Random Feature Approximation

• Noel Loo
• Ramin Hasani
• Alexander Amini
• Daniela Rus

Dataset distillation compresses large datasets into smaller synthetic coresets which retain performance with the aim of reducing the storage and computational burden of processing the entire dataset. Today's best performing algorithm, \textit{Kernel Inducing Points} (KIP), which makes use of the correspondence between infinite-width neural networks and kernel-ridge regression, is prohibitively slow due to the exact computation of the neural tangent kernel matrix, scaling $O(|S|^2)$, with $|S|$ being the coreset size. To improve this, we propose a novel algorithm that uses a random feature approximation (RFA) of the Neural Network Gaussian Process (NNGP) kernel which reduces the kernel matrix computation to $O(|S|)$. Our algorithm provides at least a 100-fold speedup over KIP and can run on a single GPU. Our new method, termed an RFA Distillation (RFAD), performs competitively with KIP and other dataset condensation algorithms in accuracy over a range of large-scale datasets, both in kernel regression and finite-width network training. We demonstrate the effectiveness of our approach on tasks involving model interpretability and privacy preservation.

NeurIPS Conference 2022 Conference Paper

Evolution of Neural Tangent Kernels under Benign and Adversarial Training

• Noel Loo
• Ramin Hasani
• Alexander Amini
• Daniela Rus

Two key challenges facing modern deep learning is mitigating deep networks vulnerability to adversarial attacks, and understanding deep learning's generalization capabilities. Towards the first issue, many defense strategies have been developed, with the most common being Adversarial Training (AT). Towards the second challenge, one of the dominant theories that has emerged is the Neural Tangent Kernel (NTK) -- a characterization of neural network behavior in the infinite-width limit. In this limit, the kernel is frozen and the underlying feature map is fixed. In finite-widths however, there is evidence that feature learning happens at the earlier stages of the training (kernel learning) before a second phase where the kernel remains fixed (lazy training). While prior work has aimed at studying adversarial vulnerability through the lens of the frozen infinite-width NTK, there is no work which studies adversarial robustness of NTK during training. In this work, we perform an empirical study of the evolution of the NTK under standard and adversarial training, aiming to disambiguate the effect of adversarial training on kernel learning and lazy training. We find under adversarial training, the NTK rapidly converges to a different kernel (and feature map) than standard training. This new kernel provides adversarial robustness, even when non-robust training is performed on top of it. Furthermore, we find that adversarial training on top of a fixed kernel can yield a classifier with $76. 1\%$ robust accuracy under PGD attacks with $\varepsilon = 4/255$ on CIFAR-10.

ICRA Conference 2022 Conference Paper

Free-Space Ellipsoid Graphs for Multi-Agent Target Monitoring

• Aaron Ray
• Alyssa Pierson
• Daniela Rus

We apply a novel framework for decomposing and reasoning about free space in an environment to a multi-agent persistent monitoring problem. Our decomposition method represents free space as a collection of ellipsoids associated with a weighted connectivity graph. The same ellipsoids used for reasoning about connectivity and distance during high level planning can be used as state constraints in a Model Predictive Control algorithm to enforce collision-free motion. This structure allows for streamlined implementation in distributed multi-agent tasks in 2D and 3D environments. We illustrate its effectiveness for a team of tracking agents tasked with monitoring a group of target agents. Our algorithm uses the ellipsoid decomposition as a primitive for the coordination, path planning, and control of the tracking agents. Simulations with four tracking agents monitoring fifteen dynamic targets in obstacle-rich environments demonstrate the performance of our algorithm.

AAAI Conference 2022 Conference Paper

GoTube: Scalable Statistical Verification of Continuous-Depth Models

• Sophie A. Gruenbacher
• Mathias Lechner
• Ramin Hasani
• Daniela Rus
• Thomas A. Henzinger
• Scott A. Smolka
• Radu Grosu

We introduce a new statistical verification algorithm that formally quantifies the behavioral robustness of any timecontinuous process formulated as a continuous-depth model. Our algorithm solves a set of global optimization (Go) problems over a given time horizon to construct a tight enclosure (Tube) of the set of all process executions starting from a ball of initial states. We call our algorithm GoTube. Through its construction, GoTube ensures that the bounding tube is conservative up to a desired probability and up to a desired tightness. GoTube is implemented in JAX and optimized to scale to complex continuous-depth neural network models. Compared to advanced reachability analysis tools for timecontinuous neural networks, GoTube does not accumulate overapproximation errors between time steps and avoids the infamous wrapping effect inherent in symbolic techniques. We show that GoTube substantially outperforms state-of-theart verification tools in terms of the size of the initial ball, speed, time-horizon, task completion, and scalability on a large set of experiments. GoTube is stable and sets the stateof-the-art in terms of its ability to scale to time horizons well beyond what has been previously possible.

ICRA Conference 2022 Conference Paper

Graph Grammar-Based Automatic Design for Heterogeneous Fleets of Underwater Robots

• Allan Zhao
• Jie Xu 0028
• Juan Salazar
• Wei Wang 0078
• Pingchuan Ma 0002
• Daniela Rus
• Wojciech Matusik

Autonomous underwater vehicles (AUVs) are spe-cialized robots that are commonly used for seafloor surveying and ocean water sampling. Computational design approaches have emerged to reduce the effort required to design both individual AUVs as well as fleets. As the number and scale of underwater missions increases beyond the capabilities of a single vehicle, fleet level design will become more important. Depending on the mission, the optimal fleet may consist of multiple distinct types of AUVs designed to a variety of specifications. Moreover, the AUVs may differ in both continuous parameters (such as battery capacity) and discrete parameters (such as number and model of thrusters). In this work, we present a computational pipeline for designing these heterogeneous AUV fleets. Using a novel shape design space based on a graph grammar and deformation cages, we can express a variety of AUV architectures with different topologies, component selections, and dimensions. We search this space using a combination of discrete graph search and gradient-based continuous optimization, enabled by a differentiable AUV simulator. Finally, we formulate heterogeneous fleet design as a modified knapsack problem, and solve it using an efficient backtracking-based algorithm. We evaluate our pipeline on a simulated mission with nonuniform design requirements-surveying a section of seafloor with varying depth-and show that the best heterogeneous fleet outperforms the best fleet composed of a single vehicle type.

ICRA Conference 2022 Conference Paper

Latent Imagination Facilitates Zero-Shot Transfer in Autonomous Racing

• Axel Brunnbauer
• Luigi Berducci
• Andreas Brandstätter
• Mathias Lechner
• Ramin M. Hasani
• Daniela Rus
• Radu Grosu

World models learn behaviors in a latent imagination space to enhance the sample-efficiency of deep reinforcement learning (RL) algorithms. While learning world models for high-dimensional observations (e. g. , pixel inputs) has become practicable on standard RL benchmarks and some games, their effectiveness in real-world robotics applications has not been explored. In this paper, we investigate how such agents generalize to real-world autonomous vehicle control tasks, where advanced model-free deep RL algorithms fail. In particular, we set up a series of time-lap tasks for an F1TENTH racing robot, equipped with a high-dimensional LiDAR sensor, on a set of test tracks with a gradual increase in their complexity. In this continuous-control setting, we show that model-based agents capable of learning in imagination substantially outperform model-free agents with respect to performance, sample efficiency, successful task completion, and generalization. Moreover, we show that the generalization ability of model-based agents strongly depends on the choice of their observation model. We provide extensive empirical evidence for the effectiveness of world models provided with long enough memory horizons in sim2real tasks.

ICRA Conference 2022 Conference Paper

Learning Interactive Driving Policies via Data-driven Simulation

• Tsun-Hsuan Wang
• Alexander Amini
• Wilko Schwarting
• Igor Gilitschenski
• Sertac Karaman
• Daniela Rus

Data-driven simulators promise high data-efficiency for driving policy learning. When used for modelling interactions, this data-efficiency becomes a bottleneck: small underlying datasets often lack interesting and challenging edge cases for learning interactive driving. We address this challenge by proposing a data-driven simulation engine† that uses inpainted ado vehicles for learning robust driving policies. Thus, our approach can be used to learn policies that involve multi-agent interactions and allows for training via state-of-the-art policy learning methods. We evaluate the approach for learning standard interaction scenarios in driving. In extensive experiments, our work demonstrates that the resulting policies can be directly transferred to a full-scale autonomous vehicle without making use of any traditional sim-to-real transfer techniques such as domain randomization.

ICRA Conference 2022 Conference Paper

Self-Reconfiguring Robotic Gantries Powered by Modular Magnetic Lead Screws

• John W. Romanishin
• James M. Bern
• Daniela Rus

This paper outlines the design, specifications, and algorithms for a new modular self-reconfigurable robotic system; at its foundation is a novel modular magnetically geared linear actuator paired with a kinematic coupling connector. Motivating this work is the core idea that high performance actuators as well as inexpensive, precise and repeatable connectors are the key ingredients required for useful real-world self-reconfiguring machines. This work builds upon existing research in the areas of modular self-reconfigurable robots, magnetic lead screws, modular machine tools and kinematic couplings. Magnetic lead screws (MLS) have many desirable characteristics applicable to modular robots, including a tolerance for slight misalignments, high efficiency, zero backlash, robustness, inherent series elasticity, high force capability, and the ability to gracefully separate and reattach. Due to their high mechanical efficiency, MLS actuators are able to be combined in parallel to provide for increased forces and stiffness. Our system implements a MLS through two separable elements: brushless motor powered actuators called carts which pair with modular passive tracks which constrain the carts' movement to a line. This paper also explores the design for a connector which is able to precisely align modules through the use of a 4-way symmetric kinematic coupling.

ICRA Conference 2022 Conference Paper

Simulation and Fabrication of Soft Robots with Embedded Skeletons

• James M. Bern
• Fatemeh Zargarbashi
• Annan Zhang
• Josie Hughes
• Daniela Rus

Soft robots can be incredibly robust and safe but typically fail to match the strength and precision of rigid robots. This dichotomy between soft and rigid is recently starting to break down, with emerging research interest in hybrid soft-rigid robots. In this work, we draw inspiration from Nature, which achieves the best of both worlds by coupling soft and rigid tissues-like muscle and bone-to produce biological systems capable of both robustness and strength. We present foundational, general-purpose pipelines to simulate and fabricate cable-driven soft-rigid robots with embedded skeletons. We show that robots built using these methods can fluidly mimic biological systems while achieving greater force output and external load resistance than purely soft robots. Finally, we show how our simulation and fabrication pipelines can be leveraged to create more complex robots and do model-based control.

ICRA Conference 2022 Conference Paper

VISTA 2. 0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles

• Alexander Amini
• Tsun-Hsuan Wang
• Igor Gilitschenski
• Wilko Schwarting
• Zhijian Liu
• Song Han 0003
• Sertac Karaman
• Daniela Rus

Simulation has the potential to transform the development of robust algorithms for mobile agents deployed in safety-critical scenarios. However, the poor photorealism and lack of diverse sensor modalities of existing simulation engines remain key hurdles towards realizing this potential. Here, we present VISTA † † Full code release for the VISTA data-driven simulation engine is available here: vista. csail. mit.edu. , an open source, data-driven simulator that integrates multiple types of sensors for autonomous vehicles. Using high fidelity, real-world datasets, VISTA represents and simulates RGB cameras, 3D LiDAR, and event-based cameras, enabling the rapid generation of novel viewpoints in simulation and thereby enriching the data available for policy learning with corner cases that are difficult to capture in the physical world. Using VISTA, we demonstrate the ability to train and test perception-to-control policies across each of the sensor types and showcase the power of this approach via deployment on a full scale autonomous vehicle. The policies learned in VISTA exhibit sim-to-real transfer without modification and greater robustness than those trained exclusively on real-world data.

IROS Conference 2022 Conference Paper

Wirelessly Magnetically Actuated Motor for Tissue Regeneration Robotic Implant

• Cameron Duffield
• Abigail F. Smith
• Daniela Rus
• Dana D. Damian
• Shuhei Miyashita

In biomedical engineering, robotic implants provide new methods to restore and improve bodily function, and regenerate tissue. A significant challenge with the design of these devices is to safely actuate them for weeks or months, while they are residing in a patient's body. Magnetic, and other force-at-distance actuation methods, allow mechanisms to be controlled remotely and without contact or line of sight to the device. In this paper, we present a novel magnetic field driven wireless motor. The motor drives a robotic implant for the treatment of long gap esophageal atresia and short bowel syndrome. The motor is equipped with two oppositely oriented permanent magnets which experience forces in opposite directions when a magnetic field is applied tangential to the magnets' directions. The implant can produce a force of 2 N. It is demonstrated with an ex vivo porcine esophagus.

ICRA Conference 2021 Conference Paper

Adaptive Nonlinear Model Predictive Control for Autonomous Surface Vessels With Largely Varying Payload

• Wei Wang 0078
• Niklas Hagemann
• Carlo Ratti
• Daniela Rus

Autonomous surface vessels (ASVs) always carry payloads such as passengers and cargoes. The change in the payload can sometimes be several times the weight of the vessel. The payload can cause significant changes in the dynamics of the vessel, thereby degrading the performance of the controller. This paper proposes an adaptive nonlinear model predictive control (A-NMPC) strategy for ASV trajectory tracking, which allows real-time changes in dynamics caused by severe payload variation. First, a nonlinear dynamic model that updates with the vessel’s payload is established. Then a pressure sensing method is proposed to estimate the payload of the vessel. Further, a parametric cost function that considers changing dynamics, as well as input and state constraints, is formulated in the NMPC algorithm. The tracking ability of A-NMPC is systematically studied on three different sizes of vessels in the simulation where the payload of these vessels changes eight times their inherent weight. Numerical results show that when the payload changes greatly the vessels with A-NMPC can accurately track the reference trajectory while the vessels with conventional NMPC cannot. Finally, the tracking experiments with a quarter-scale vessel in a swimming pool further verify the effectiveness of the proposed A-NMPC strategy.

ICRA Conference 2021 Conference Paper

Adversarial Training is Not Ready for Robot Learning

• Mathias Lechner
• Ramin M. Hasani
• Radu Grosu
• Daniela Rus
• Thomas A. Henzinger

Adversarial training is an effective method to train deep learning models that are resilient to norm-bounded perturbations, with the cost of nominal performance drop. While adversarial training appears to enhance the robustness and safety of a deep model deployed in open-world decision-critical applications, counterintuitively, it induces undesired behaviors in robot learning settings. In this paper, we show theoretically and experimentally that neural controllers obtained via adversarial training are subjected to three types of defects, namely transient, systematic, and conditional errors. We first generalize adversarial training to a safety-domain optimization scheme allowing for more generic specifications. We then prove that such a learning process tends to cause certain error profiles. We support our theoretical results by a thorough experimental safety analysis in a robot-learning task. Our results suggest that adversarial training is not yet ready for robot learning.

ICRA Conference 2021 Conference Paper

Aggregating Long-Term Context for Learning Laparoscopic and Robot-Assisted Surgical Workflows

• Yutong Ban
• Guy Rosman
• Thomas M. Ward
• Daniel A. Hashimoto
• Taisei Kondo
• Hidekazu Iwaki
• Ozanan R. Meireles
• Daniela Rus

Analyzing surgical workflow is crucial for surgical assistance robots to understand surgeries. With the understanding of the complete surgical workflow, the robots are able to assist the surgeons in intra-operative events, such as by giving a warning when the surgeon is entering specific keys or high-risk phases. Deep learning techniques have recently been widely applied to recognizing surgical workflows. Many of the existing temporal neural network models are limited in their capability to handle long-term dependencies in the data, instead, relying upon the strong performance of the underlying per-frame visual models. We propose a new temporal network structure that leverages task-specific network representation to collect long-term sufficient statistics that are propagated by a sufficient statistics model (SSM). We implement our approach within an LSTM backbone for the task of surgical phase recognition and explore several choices for propagated statistics. We demonstrate superior results over existing and novel state-of-the-art segmentation techniques on two laparoscopic cholecystectomy datasets: the publicly available Cholec80 dataset and MGH100, a novel dataset with more challenging and clinically meaningful segment labels.

ICRA Conference 2021 Conference Paper

Autonomous Navigation in Dynamic Environments with Multi-Modal Perception Uncertainties

• Hongliang Guo 0003
• Zefan Huang
• Qi Heng Ho
• Marcelo H. Ang
• Daniela Rus

This paper addresses the safe path planning problem for autonomous mobility with multi-modal perception uncertainties. Specifically, we assume that different sensor inputs lead to different Gaussian process regulated perception uncertainties (named as multi-modal perception uncertainties). We implement a Bayesian inference algorithm, which merges the multi-modal GP-regulated uncertainties into a unified one and translates the unified uncertainty into a dynamic risk map. With the safe path planner taking the risk map as input, we are able to plan a safe path for the autonomous vehicle to follow. Experimental results on an autonomous golf cart testbed validate the applicability and efficiency of the proposed algorithm.

NeurIPS Conference 2021 Conference Paper

Causal Navigation by Continuous-time Neural Networks

• Charles Vorbach
• Ramin Hasani
• Alexander Amini
• Mathias Lechner
• Daniela Rus

Imitation learning enables high-fidelity, vision-based learning of policies within rich, photorealistic environments. However, such techniques often rely on traditional discrete-time neural models and face difficulties in generalizing to domain shifts by failing to account for the causal relationships between the agent and the environment. In this paper, we propose a theoretical and experimental framework for learning causal representations using continuous-time neural networks, specifically over their discrete-time counterparts. We evaluate our method in the context of visual-control learning of drones over a series of complex tasks, ranging from short- and long-term navigation, to chasing static and dynamic objects through photorealistic environments. Our results demonstrate that causal continuous-time deep models can perform robust navigation tasks, where advanced recurrent models fail. These models learn complex causal control representations directly from raw visual inputs and scale to solve a variety of tasks using imitation learning.

NeurIPS Conference 2021 Conference Paper

Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition

• Lucas Liebenwein
• Alaa Maalouf
• Dan Feldman
• Daniela Rus

We present a novel global compression framework for deep neural networks that automatically analyzes each layer to identify the optimal per-layer compression ratio, while simultaneously achieving the desired overall compression. Our algorithm hinges on the idea of compressing each convolutional (or fully-connected) layer by slicing its channels into multiple groups and decomposing each group via low-rank decomposition. At the core of our algorithm is the derivation of layer-wise error bounds from the Eckart–Young–Mirsky theorem. We then leverage these bounds to frame the compression problem as an optimization problem where we wish to minimize the maximum compression error across layers and propose an efficient algorithm towards a solution. Our experiments indicate that our method outperforms existing low-rank compression approaches across a wide range of networks and data sets. We believe that our results open up new avenues for future research into the global performance-size trade-offs of modern neural networks.

IROS Conference 2021 Conference Paper

Context and Orientation Aware Path Tracking

• Nicholas Michael Bünger
• Sahil Panjwani
• Malika Meghjani
• Zefan Huang
• Marcelo H. Ang
• Daniela Rus

Autonomous vehicles on city roads and especially in pedestrian environments require agility to navigate narrow passages and turn in tight spaces, leading to the need for a real-time, robust and adaptable controller. In this paper, we present orientation and context aware controllers for autonomous vehicles that can closely track the reference path wit alh respect to the current state of the vehicle, environmental properties, and the desired target orientation at the desired target location. Our proposed controllers are derived from the widely used pure pursuit controller. We validate our proposed controllers with respect to the baseline pure pursuit controller in simulation and on a full-size autonomous vehicle in a pedestrian environment. Our experimental results suggest significant improvements in adaptability and tracking performance compared to the pure pursuit controller.

ICRA Conference 2021 Conference Paper

Deep Imitation Learning for Autonomous Navigation in Dynamic Pedestrian Environments

• Lei Qin
• Zefan Huang
• Chen Zhang 0018
• Hongliang Guo 0003
• Marcelo H. Ang
• Daniela Rus

Navigation through dynamic pedestrian environments in a socially compliant manner is still a challenging task for autonomous vehicles. Classical methods usually lead to unnatural vehicle behaviours for pedestrian navigation due to the difficulty in modeling social conventions mathematically. This paper presents an end-to-end path planning system that achieves autonomous navigation in dynamic environments through imitation learning. The proposed system is based on a fully convolutional neural network that maps the raw sensory data into a confidence map for path extraction. Additionally, a classification network is introduced to reduce the unnecessary re-plannings and ensures that the vehicle goes back to the global path when re-planning is not needed. The imitation learning based path planner is implemented on an autonomous wheelchair and tested in a new real-world dynamic pedestrian environment. Experimental results show that the proposed system is able to generate paths for different driving tasks, such as pedestrian following, static and dynamic obstacles avoidance, etc. In comparison to the state-of-the-art method, our system is superior in terms of generating human-like trajectories.

ICLR Conference 2021 Conference Paper

Deep Learning meets Projective Clustering

• Alaa Maalouf
• Harry Lang
• Daniela Rus
• Dan Feldman

A common approach for compressing Natural Language Processing (NLP) networks is to encode the embedding layer as a matrix $A\in\mathbb{R}^{n\times d}$, compute its rank-$j$ approximation $A_j$ via SVD (Singular Value Decomposition), and then factor $A_j$ into a pair of matrices that correspond to smaller fully-connected layers to replace the original embedding layer. Geometrically, the rows of $A$ represent points in $\mathbb{R}^d$, and the rows of $A_j$ represent their projections onto the $j$-dimensional subspace that minimizes the sum of squared distances (``errors'') to the points. In practice, these rows of $A$ may be spread around $k>1$ subspaces, so factoring $A$ based on a single subspace may lead to large errors that turn into large drops in accuracy. Inspired by \emph{projective clustering} from computational geometry, we suggest replacing this subspace by a set of $k$ subspaces, each of dimension $j$, that minimizes the sum of squared distances over every point (row in $A$) to its \emph{closest} subspace. Based on this approach, we provide a novel architecture that replaces the original embedding layer by a set of $k$ small layers that operate in parallel and are then recombined with a single fully-connected layer. Extensive experimental results on the GLUE benchmark yield networks that are both more accurate and smaller compared to the standard matrix factorization (SVD). For example, we further compress DistilBERT by reducing the size of the embedding layer by $40\%$ while incurring only a $0.5\%$ average drop in accuracy over all nine GLUE tasks, compared to a $2.8\%$ drop using the existing SVD approach. On RoBERTa we achieve $43\%$ compression of the embedding layer with less than a $0.8\%$ average drop in accuracy as compared to a $3\%$ drop previously.

IROS Conference 2021 Conference Paper

Designing and Deploying a Mobile UVC Disinfection Robot

• Alyssa Pierson
• John W. Romanishin
• Hunter Hansen
• Leonardo Zamora Yañez
• Daniela Rus

This paper presents a mobile UVC disinfection robot designed to mitigate the threat of airborne and surface pathogens. Our system comprises a mobile robot base, a custom UVC lamp assembly, and algorithms for autonomous navigation and path planning. We present a model of UVC disinfection and dosage of UVC light delivered by the mobile robot. We also discuss challenges and prototyping decisions for rapid deployment of the robot during the COVID-19 pandemic. Experimental results summarize a long-term deployment at The Greater Boston Food Bank, where the robot delivers (nightly) UVC dosages of at least 10 mJ/cm 2 to a 4000 ft 2 area in under 30 minutes. These dosages are capable of neutralizing 99% of coronaviruses, including SARS-CoV-2, on surfaces and in airborne particles. Further simulations present how this mobile UVC disinfection robot may be extended to classic problems in robotic path planning and adaptive multi-robot coverage control.

ICRA Conference 2021 Conference Paper

Efficient and Robust LiDAR-Based End-to-End Navigation

• Zhijian Liu
• Alexander Amini
• Sibo Zhu
• Sertac Karaman
• Song Han 0003
• Daniela Rus

Deep learning has been used to demonstrate end-to-end neural network learning for autonomous vehicle control from raw sensory input. While LiDAR sensors provide reliably accurate information, existing end-to-end driving solutions are mainly based on cameras since processing 3D data requires a large memory footprint and computation cost. On the other hand, increasing the robustness of these systems is also critical; however, even estimating the model’s uncertainty is very challenging due to the cost of sampling-based methods. In this paper, we present an efficient and robust LiDAR-based end-to-end navigation framework. We first introduce Fast-LiDARNet that is based on sparse convolution kernel optimization and hardware-aware model design. We then propose Hybrid Evidential Fusion that directly estimates the uncertainty of the prediction from only a single forward pass and then fuses the control predictions intelligently. We evaluate our system on a full-scale vehicle and demonstrate lane-stable as well as navigation capabilities. In the presence of out-of-distribution events (e. g. , sensor failures), our system significantly improves robustness and reduces the number of takeovers in the real world.

IROS Conference 2021 Conference Paper

Group Multi-Object Tracking for Dynamic Risk Map and Safe Path Planning

• Lyuyu Shen
• Hongliang Guo 0003
• Yechao Bai
• Lei Qin
• Marcelo H. Ang
• Daniela Rus

This paper studies the group multi-object tracking (MOT) problem in dynamic pedestrian environments, with intended application to safe navigation for autonomous vehicles. We complete a full autonomous vehicle navigation pipeline from object detection, tracking, grouping, to risk map generation and safe path planning. Our main contribution is to instantiate a group multi-object tracking algorithm, which provides the crucial grouped activity information, i. e. group position, group velocity, group size, to the risk map generator, and therewith produce a stable and robust risk map for the downstream safe path planner. Experimental results with real world data show the socially acceptable, robust and stable performance of the proposed algorithm over its individual MOT counterpart.

ICRA Conference 2021 Conference Paper

Interactive Planning for Autonomous Urban Driving in Adversarial Scenarios

• Yuanfu Luo
• Malika Meghjani
• Qi Heng Ho
• David Hsu
• Daniela Rus

Autonomous urban driving among human-driven cars requires a holistic understanding of road rules, driver intents and driving styles. This is challenging as a short-term, single instance, driver intent of lane change may not correspond to their driving styles for a longer duration. This paper presents an interactive behavior planner which accounts for road context, short-term driver intent, and long-term driving style to infer beliefs over the latent states of surrounding vehicles. We use a specialized Partially Observable Markov Decision Process to provide risk-averse decisions. Specifically, we consider adversarial driving scenarios caused by irrational drivers to validate the robustness of our proposed interactive behavior planner in simulation as well as on a full-size self-driving car. Our experimental results show that our algorithm enables safer and more travel time-efficient autonomous driving compared to baselines even in adversarial scenarios.

NeurIPS Conference 2021 Conference Paper

Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies

• Tim Seyde
• Igor Gilitschenski
• Wilko Schwarting
• Bartolomeo Stellato
• Martin Riedmiller
• Markus Wulfmeier
• Daniela Rus

Reinforcement learning (RL) for continuous control typically employs distributions whose support covers the entire action space. In this work, we investigate the colloquially known phenomenon that trained agents often prefer actions at the boundaries of that space. We draw theoretical connections to the emergence of bang-bang behavior in optimal control, and provide extensive empirical evaluation across a variety of recent RL algorithms. We replace the normal Gaussian by a Bernoulli distribution that solely considers the extremes along each action dimension - a bang-bang controller. Surprisingly, this achieves state-of-the-art performance on several continuous control benchmarks - in contrast to robotic hardware, where energy and maintenance cost affect controller choices. Since exploration, learning, and the final solution are entangled in RL, we provide additional imitation learning experiments to reduce the impact of exploration on our analysis. Finally, we show that our observations generalize to environments that aim to model real-world challenges and evaluate factors to mitigate the emergence of bang-bang solutions. Our findings emphasise challenges for benchmarking continuous control algorithms, particularly in light of potential real-world applications.

IROS Conference 2021 Conference Paper

LiDAR Degradation Quantification for Autonomous Driving in Rain

• Chen Zhang 0018
• Zefan Huang
• Marcelo H. Ang
• Daniela Rus

Autonomous driving in rainy conditions remains a big challenge. One of the issues is sensor degradation. LiDAR is commonly used in autonomous driving systems to perceive and understand surrounding environments. However, LiDAR performance can be degraded by rain, thereby influencing other system performance (e. g. , perception or localization). Therefore, knowing how much degradation exists in current LiDAR measurements is necessary. Most existing methods can only measure LiDAR degradation in controlled environments (e. g. , a chamber with simulated rain); how to quantify LiDAR degradation in dynamic environments while the autonomous vehicle is moving is still a difficult problem. In this work, we propose a novel approach to address this problem using an anomaly detection method. Our method has been evaluated on simulated and real-world data. Experimental results demonstrate the effectiveness of our method to capture LiDAR degradation and yield reasonable degradation estimations. Our experimental data and codes are accessible from http://rain.smart.mit.edu/smartrain/.

AAAI Conference 2021 Conference Paper

Liquid Time-constant Networks

• Ramin Hasani
• Mathias Lechner
• Alexander Amini
• Daniela Rus
• Radu Grosu

We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system’s dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems modulated via nonlinear interlinked gates. The resulting models represent dynamical systems with varying (i. e. , liquid) time-constants coupled to their hidden state, with outputs being computed by numerical differential equation solvers. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations, and give rise to improved performance on time-series prediction tasks. To demonstrate these properties, we first take a theoretical approach to find bounds over their dynamics, and compute their expressive power by the trajectory length measure in a latent trajectory space. We then conduct a series of time-series prediction experiments to manifest the approximation capability of Liquid Time-Constant Networks (LTCs) compared to classical and modern RNNs.

ICRA Conference 2021 Conference Paper

LVI-SAM: Tightly-coupled Lidar-Visual-Inertial Odometry via Smoothing and Mapping

• Tixiao Shan
• Brendan J. Englot
• Carlo Ratti
• Daniela Rus

We propose a framework for tightly-coupled lidar-visual-inertial odometry via smoothing and mapping, LVI-SAM, that achieves real-time state estimation and map-building with high accuracy and robustness. LVI-SAM is built atop a factor graph and is composed of two sub-systems: a visual-inertial system (VIS) and a lidar-inertial system (LIS). The two sub-systems are designed in a tightly-coupled manner, in which the VIS leverages LIS estimation to facilitate initialization. The accuracy of the VIS is improved by extracting depth information for visual features using lidar measurements. In turn, the LIS utilizes VIS estimation for initial guesses to support scan-matching. Loop closures are first identified by the VIS and further refined by the LIS. LVI-SAM can also function when one of the two sub-systems fails, which increases its robustness in both texture-less and feature-less environments. LVI-SAM is extensively evaluated on datasets gathered from several platforms over a variety of scales and environments. Our implementation is available at https://git.io/lvi-sam.

ICRA Conference 2021 Conference Paper

Multi-Objective Graph Heuristic Search for Terrestrial Robot Design

• Jie Xu 0028
• Andrew Spielberg
• Allan Zhao
• Daniela Rus
• Wojciech Matusik

We present methods for co-designing rigid robots over control and morphology (including discrete topology) over multiple objectives. Previous work has addressed problems in single-objective robot co-design or multi-objective control. However, the joint multi-objective co-design problem is extremely important for generating capable, versatile, algorithmically designed robots. In this work, we present Multi-Objective Graph Heuristic Search, which extends a single-objective graph heuristic search from previous work to enable a highly efficient multi-objective search in a combinatorial design topology space. Core to this approach, we introduce a new universal, multiobjective heuristic function based on graph neural networks that is able to effectively leverage learned information between different task trade-offs. We demonstrate our approach on six combinations of seven terrestrial locomotion and design tasks, including one three-objective example. We compare the captured Pareto fronts across different methods and demonstrate that our multi-objective graph heuristic search quantitatively and qualitatively outperforms other techniques.

IROS Conference 2021 Conference Paper

Multi-robot Task Assignment for Aerial Tracking with Viewpoint Constraints

• Aaron Ray
• Alyssa Pierson
• Hai Zhu 0002
• Javier Alonso-Mora
• Daniela Rus

We address the problem of assigning a team of drones to autonomously capture a set desired shots of a dynamic target in the presence of obstacles. We present a two-stage planning pipeline that generates offline an assignment of drone to shots and locally optimizes online the viewpoint. Given desired shot parameters, the high-level planner uses a visibility heuristic to predict good times for capturing each shot and uses an Integer Linear Program to compute drone assignments. An online Model Predictive Control algorithm uses the assignments as reference to capture the shots. The algorithm is validated in hardware with a pair of drones and a remote controlled car.

TCS Journal 2021 Journal Article

On coresets for support vector machines

• Murad Tukan
• Cenk Baykal
• Dan Feldman
• Daniela Rus

ICML Conference 2021 Conference Paper

On-Off Center-Surround Receptive Fields for Accurate and Robust Image Classification

• Zahra Babaiee
• Ramin M. Hasani
• Mathias Lechner
• Daniela Rus
• Radu Grosu

Robustness to variations in lighting conditions is a key objective for any deep vision system. To this end, our paper extends the receptive field of convolutional neural networks with two residual components, ubiquitous in the visual processing system of vertebrates: On-center and off-center pathways, with an excitatory center and inhibitory surround; OOCS for short. The On-center pathway is excited by the presence of a light stimulus in its center, but not in its surround, whereas the Off-center pathway is excited by the absence of a light stimulus in its center, but not in its surround. We design OOCS pathways via a difference of Gaussians, with their variance computed analytically from the size of the receptive fields. OOCS pathways complement each other in their response to light stimuli, ensuring this way a strong edge-detection capability, and as a result an accurate and robust inference under challenging lighting conditions. We provide extensive empirical evidence showing that networks supplied with OOCS pathways gain accuracy and illumination-robustness from the novel edge representation, compared to other baselines.

IROS Conference 2021 Conference Paper

Robotic Jigsaw: A Non-Holonomic Cutting Robot and Path Planning Algorithm

• Haisen Zhao
• Yash Talwekar
• Wenqing Lan
• Chetan Sharma
• Daniela Rus
• Adriana Schulz
• Jeffrey I. Lipton

Bladed tools such as jigsaws are common tools for wood workers on job-sites and in workshops, but do not currently have sufficient autonomous hardware or path planning algorithms to enable automation. Here we present a system of an autonomous robot and a path planning algorithm for automating jigsaw operations. The robot can drill holes, insert the jigsaw, and cut plywood. Our algorithm converts complex shapes into paths for the jigsaw, drill holes, and traversal movements for the robot. The algorithm decomposes input shapes into cuttable sections and determines possible locations for drilling entry holes for inserting the blade. We cast the drill hole problem as a set coverage problem with a trade-off between number of holes and cutting distance. We characterize the algorithm on a series of shapes and determined the algorithm found valid solutions. We executed an example on the robot to demonstrate the end-to-end system.

ICRA Conference 2021 Conference Paper

Robust Place Recognition using an Imaging Lidar

• Tixiao Shan
• Brendan J. Englot
• Fabio Duarte
• Carlo Ratti
• Daniela Rus

We propose a methodology for robust, real-time place recognition using an imaging lidar, which yields image-quality high-resolution 3D point clouds. Utilizing the intensity readings of an imaging lidar, we project the point cloud and obtain an intensity image. ORB feature descriptors are extracted from the image and encoded into a bag-of-words vector. The vector, used to identify the point cloud, is inserted into a database that is maintained by DBoW for fast place recognition queries. The returned candidate is further validated by matching visual feature descriptors. To reject matching outliers, we apply PnP, which minimizes the reprojection error of visual features’ positions in Euclidean space with their correspondences in 2D image space, using RANSAC. Combining the advantages from both camera and lidar-based place recognition approaches, our method is truly rotation-invariant, and can tackle reverse revisiting and upside down revisiting. The proposed method is evaluated on datasets gathered from a variety of platforms over different scales and environments. Our implementation and datasets are available at https://git.io/image-lidar.

IROS Conference 2021 Conference Paper

Semi-Cooperative Control for Autonomous Emergency Vehicles

• Noam Buckman
• Wilko Schwarting
• Sertac Karaman
• Daniela Rus

Autonomous control of an emergency vehicle will save lives through faster transport and shorter response. Towards this goal, it must overcome the challenge of inter- acting with existing human drivers on the road. We present a game-theoretic approach for semi-cooperative control of an autonomous emergency vehicle that can interact efficiently with humans on the road. We model the interactions between autonomous and human driven cars with Social Value Orientation, a metric from social psychology, that allows the controller to leverage their influence on the trajectories of neighboring human drivers. In addition, by using a modified version of iterative best response, we direct the algorithm to converge to Nash equilibria that are cooperative. We demonstrate the efficacy of our algorithm in simulations of drivers in traffic, with a variety of traffic densities and driver personalities. In simulations of prosocial human drivers, our algorithm provides an 8% improvement in distance-traveled compared to egoistic human drivers.

NeurIPS Conference 2021 Conference Paper

Sparse Flows: Pruning Continuous-depth Models

• Lucas Liebenwein
• Ramin Hasani
• Alexander Amini
• Daniela Rus

Continuous deep learning architectures enable learning of flexible probabilistic models for predictive modeling as neural ordinary differential equations (ODEs), and for generative modeling as continuous normalizing flows. In this work, we design a framework to decipher the internal dynamics of these continuous depth models by pruning their network architectures. Our empirical results suggest that pruning improves generalization for neural ODEs in generative modeling. We empirically show that the improvement is because pruning helps avoid mode-collapse and flatten the loss surface. Moreover, pruning finds efficient neural ODE representations with up to 98% less parameters compared to the original network, without loss of accuracy. We hope our results will invigorate further research into the performance-size trade-offs of modern continuous-depth models.

ICML Conference 2021 Conference Paper

The Logical Options Framework

• Brandon Araki
• Xiao Li 0025
• Kiran Vodrahalli
• Jonathan A. DeCastro
• Micah J. Fry
• Daniela Rus

Learning composable policies for environments with complex rules and tasks is a challenging problem. We introduce a hierarchical reinforcement learning framework called the Logical Options Framework (LOF) that learns policies that are satisfying, optimal, and composable. LOF efficiently learns policies that satisfy tasks by representing the task as an automaton and integrating it into learning and planning. We provide and prove conditions under which LOF will learn satisfying, optimal policies. And lastly, we show how LOF’s learned policies can be composed to satisfy unseen tasks with only 10-50 retraining steps on our benchmarks. We evaluate LOF on four tasks in discrete and continuous domains, including a 3D pick-and-place environment.

ICML Conference 2020 Conference Paper

A Natural Lottery Ticket Winner: Reinforcement Learning with Ordinary Neural Circuits

• Ramin M. Hasani
• Mathias Lechner
• Alexander Amini
• Daniela Rus
• Radu Grosu

We propose a neural information processing system obtained by re-purposing the function of a biological neural circuit model to govern simulated and real-world control tasks. Inspired by the structure of the nervous system of the soil-worm, C. elegans, we introduce ordinary neural circuits (ONCs), defined as the model of biological neural circuits reparameterized for the control of alternative tasks. We first demonstrate that ONCs realize networks with higher maximum flow compared to arbitrary wired networks. We then learn instances of ONCs to control a series of robotic tasks, including the autonomous parking of a real-world rover robot. For reconfiguration of the purpose of the neural circuit, we adopt a search-based optimization algorithm. Ordinary neural circuits perform on par and, in some cases, significantly surpass the performance of contemporary deep learning models. ONC networks are compact, 77% sparser than their counterpart neural controllers, and their neural dynamics are fully interpretable at the cell-level.

AAAI Conference 2020 Conference Paper

Deep Bayesian Nonparametric Learning of Rules and Plans from Demonstrations with a Learned Automaton Prior

• Brandon Araki
• Kiran Vodrahalli
• Thomas Leech
• Cristian-Ioan Vasile
• Mark Donahue
• Daniela Rus

We introduce a method to learn imitative policies from expert demonstrations that are interpretable and manipulable. We achieve interpretability by modeling the interactions between high-level actions as an automaton with connections to formal logic. We achieve manipulability by integrating this automaton into planning, so that changes to the automaton have predictable effects on the learned behavior. These qualities allow a human user to first understand what the model has learned, and then either correct the learned behavior or zeroshot generalize to new, similar tasks. We build upon previous work by no longer requiring additional supervised information which is hard to collect in practice. We achieve this by using a deep Bayesian nonparametric hierarchical model. We test our model on several domains and also show results for a real-world implementation on a mobile robotic arm platform.

NeurIPS Conference 2020 Conference Paper

Deep Evidential Regression

• Alexander Amini
• Wilko Schwarting
• Ava Soleimany
• Daniela Rus

Deterministic neural networks (NNs) are increasingly being deployed in safety critical domains, where calibrated, robust, and efficient measures of uncertainty are crucial. In this paper, we propose a novel method for training non-Bayesian NNs to estimate a continuous target as well as its associated evidence in order to learn both aleatoric and epistemic uncertainty. We accomplish this by placing evidential priors over the original Gaussian likelihood function and training the NN to infer the hyperparameters of the evidential distribution. We additionally impose priors during training such that the model is regularized when its predicted evidence is not aligned with the correct output. Our method does not rely on sampling during inference or on out-of-distribution (OOD) examples for training, thus enabling efficient and scalable uncertainty learning. We demonstrate learning well-calibrated measures of uncertainty on various benchmarks, scaling to complex computer vision tasks, as well as robustness to adversarial and OOD test samples.

ICLR Conference 2020 Conference Paper

Deep Orientation Uncertainty Learning based on a Bingham Loss

• Igor Gilitschenski
• Roshni Sahoo
• Wilko Schwarting
• Alexander Amini
• Sertac Karaman
• Daniela Rus

Reasoning about uncertain orientations is one of the core problems in many perception tasks such as object pose estimation or motion estimation. In these scenarios, poor illumination conditions, sensor limitations, or appearance invariance may result in highly uncertain estimates. In this work, we propose a novel learning-based representation for orientation uncertainty. By characterizing uncertainty over unit quaternions with the Bingham distribution, we formulate a loss that naturally captures the antipodal symmetry of the representation. We discuss the interpretability of the learned distribution parameters and demonstrate the feasibility of our approach on several challenging real-world pose estimation tasks involving uncertain orientations.

IROS Conference 2020 Conference Paper

Distributed Motion Control for Multiple Connected Surface Vessels

• Wei Wang 0078
• Zijian Wang 0003
• Luis A. Mateos
• Kuan Wei Huang
• Mac Schwager
• Carlo Ratti
• Daniela Rus

We propose a scalable cooperative control approach which coordinates a group of rigidly connected autonomous surface vessels to track desired trajectories in a planar water environment as a single floating modular structure. Our approach leverages the implicit information of the structure’s motion for force and torque allocation without explicit communication among the robots. In our system, a leader robot steers the entire group by adjusting its force and torque according to the structure’s deviation from the desired trajectory, while follower robots run distributed consensus-based controllers to match their inputs to amplify the leader’s intent using only onboard sensors as feedback. To cope with the nonlinear system dynamics in the water, the leader robot employs a nonlinear model predictive controller (NMPC), where we experimentally estimated the dynamics model of the floating modular structure in order to achieve superior performance for leader-following control. Our method has a wide range of potential applications in transporting humans and goods in many of today’s existing waterways. We conducted trajectory and orientation tracking experiments in hardware with three custom-built autonomous modular robotic boats, called Roboat, which are capable of holonomic motions and onboard state estimation. Simulation results with up to 65 robots also prove the scalability of our proposed approach.

ICRA Conference 2020 Conference Paper

Generating Visibility-Aware Trajectories for Cooperative and Proactive Motion Planning

• Noam Buckman
• Alyssa Pierson
• Sertac Karaman
• Daniela Rus

The safety of an autonomous vehicle not only depends on its own perception of the world around it, but also on the perception and recognition from other vehicles. If an ego vehicle considers the uncertainty other vehicles have about itself, then by reducing the estimated uncertainty it can increase its safety. In this paper, we focus on how an ego vehicle plans its trajectories through the blind spots of other vehicles. We create visibility-aware planning, where the ego vehicle chooses its trajectories such that it reduces the perceived uncertainty other vehicles may have about the state of the ego vehicle. We present simulations of traffic and highway environments, where an ego vehicle must pass another vehicle, make a lane change, or traverse a partially-occluded intersection. Emergent behavior shows that when using visibility-aware planning, the ego vehicle spends less time in a blind spot, and may slow down before entering the blind spot so as to increase the likelihood other vehicles perceive the ego vehicle.

ICRA Conference 2020 Conference Paper

Gershgorin Loss Stabilizes the Recurrent Neural Network Compartment of an End-to-end Robot Learning Scheme

• Mathias Lechner
• Ramin M. Hasani
• Daniela Rus
• Radu Grosu

Traditional robotic control suits require profound task-specific knowledge for designing, building and testing control software. The rise of Deep Learning has enabled end-to-end solutions to be learned entirely from data, requiring minimal knowledge about the application area. We design a learning scheme to train end-to-end linear dynamical systems (LDS)s by gradient descent in imitation learning robotic domains. We introduce a new regularization loss component together with a learning algorithm that improves the stability of the learned autonomous system, by forcing the eigenvalues of the internal state updates of an LDS to be negative reals. We evaluate our approach on a series of real-life and simulated robotic experiments, in comparison to linear and nonlinear Recurrent Neural Network (RNN) architectures. Our results show that our stabilizing method significantly improves test performance of LDS, enabling such linear models to match the performance of contemporary nonlinear RNN architectures. A video of the obstacle avoidance performance of our method on a mobile robot, in unseen environments, compared to other methods can be viewed at https://youtu.be/mhEsCoNao5E.

ICRA Conference 2020 Conference Paper

Helping Robots Learn: A Human-Robot Master-Apprentice Model Using Demonstrations via Virtual Reality Teleoperation

• Joseph DelPreto
• Jeffrey I. Lipton
• Lindsay M. Sanneman
• Aidan J. Fay
• Christopher K. Fourie
• Changhyun Choi
• Daniela Rus

As artificial intelligence becomes an increasingly prevalent method of enhancing robotic capabilities, it is important to consider effective ways to train these learning pipelines and to leverage human expertise. Working towards these goals, a master-apprentice model is presented and is evaluated during a grasping task for effectiveness and human perception. The apprenticeship model augments self-supervised learning with learning by demonstration, efficiently using the human's time and expertise while facilitating future scalability to supervision of multiple robots; the human provides demonstrations via virtual reality when the robot cannot complete the task autonomously. Experimental results indicate that the robot learns a grasping task with the apprenticeship model faster than with a solely self-supervised approach and with fewer human interventions than a solely demonstration-based approach; 100% grasping success is obtained after 150 grasps with 19 demonstrations. Preliminary user studies evaluating workload, usability, and effectiveness of the system yield promising results for system scalability and deployability. They also suggest a tendency for users to overestimate the robot's skill and to generalize its capabilities, especially as learning improves.

IROS Conference 2020 Conference Paper

LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping

• Tixiao Shan
• Brendan J. Englot
• Drew Meyers
• Wei Wang 0078
• Carlo Ratti
• Daniela Rus

We propose a framework for tightly-coupled lidar inertial odometry via smoothing and mapping, LIO-SAM, that achieves highly accurate, real-time mobile robot trajectory estimation and map-building. LIO-SAM formulates lidar-inertial odometry atop a factor graph, allowing a multitude of relative and absolute measurements, including loop closures, to be incorporated from different sources as factors into the system. The estimated motion from inertial measurement unit (IMU) pre-integration de-skews point clouds and produces an initial guess for lidar odometry optimization. The obtained lidar odometry solution is used to estimate the bias of the IMU. To ensure high performance in real-time, we marginalize old lidar scans for pose optimization, rather than matching lidar scans to a global map. Scan-matching at a local scale instead of a global scale significantly improves the real-time performance of the system, as does the selective introduction of keyframes, and an efficient sliding window approach that registers a new keyframe to a fixed-size set of prior "sub-keyframes. " The proposed method is extensively evaluated on datasets gathered from three platforms over various scales and environments.

ICRA Conference 2020 Conference Paper

Multiplexed Manipulation: Versatile Multimodal Grasping via a Hybrid Soft Gripper

• Lillian Chin
• Felipe Barscevicius
• Jeffrey I. Lipton
• Daniela Rus

The success of hybrid suction + parallel-jaw grippers in the Amazon Robotics/Picking Challenge have demonstrated the effectiveness of multimodal grasping approaches. However, existing multimodal grippers combine grasping modes in isolation and do not incorporate the benefits of compliance found in soft robotic manipulators. In this paper, we present a gripper that integrates three modes of grasping: suction, parallel jaw, and soft fingers. Using complaint handed shearing auxetics actuators as the foundation, this gripper is able to multiplex manipulation by creating unique grasping primitives through permutations of these grasping techniques. This gripper is able to grasp 88% of tested objects, 14% of which could only be grasped using a combination of grasping modes. The gripper is also able to perform in-hand object re-orientation of flat objects without the need for pre-grasp manipulation.

IROS Conference 2020 Conference Paper

Online Localization with Imprecise Floor Space Maps using Stochastic Gradient Descent

• Zhikai Li
• Marcelo H. Ang
• Daniela Rus

Many indoor spaces have constantly changing layouts and may not be mapped by an autonomous vehicle, yet maps such as floor plans or evacuation maps of these places are common. We propose a method for an autonomous robot to localize itself on such maps with inconsistent scale using Stochastic Gradient Descent (SGD) with scan matching using a 2D LiDAR. We also introduce a new scale state in 2D localization to manage the possible inconsistent scale of the input map. Experiments are conducted in an indoor corridor using three different input maps - a point cloud, a floor plan, and a hand-drawn map. The SGD localization algorithm is bench-marked to Adaptive Monte Carlo Localization (AMCL). In a point cloud mapped environment, our algorithm achieves 0. 264m and 5. 26° average position and heading error respectively. On the hand-drawn map, our SGD localization algorithm remains robust while AMCL fails. The role of the scale state in our SGD localization algorithm is demonstrated in poorly scaled maps.

ICML Conference 2020 Conference Paper

Prediction-Guided Multi-Objective Reinforcement Learning for Continuous Robot Control

• Jie Xu 0028
• Yunsheng Tian
• Pingchuan Ma 0002
• Daniela Rus
• Shinjiro Sueda
• Wojciech Matusik

Many real-world control problems involve conflicting objectives where we desire a dense and high-quality set of control policies that are optimal for different objective preferences (called Pareto-optimal). While extensive research in multi-objective reinforcement learning (MORL) has been conducted to tackle such problems, multi-objective optimization for complex continuous robot control is still under-explored. In this work, we propose an efficient evolutionary learning algorithm to find the Pareto set approximation for continuous robot control problems, by extending a state-of-the-art RL algorithm and presenting a novel prediction model to guide the learning process. In addition to efficiently discovering the individual policies on the Pareto front, we construct a continuous set of Pareto-optimal solutions by Pareto analysis and interpolation. Furthermore, we design seven multi-objective RL environments with continuous action space, which is the first benchmark platform to evaluate MORL algorithms on various robot control problems. We test the previous methods on the proposed benchmark problems, and the experiments show that our approach is able to find a much denser and higher-quality set of Pareto policies than the existing algorithms.

ICLR Conference 2020 Conference Paper

Provable Filter Pruning for Efficient Neural Networks

• Lucas Liebenwein
• Cenk Baykal
• Harry Lang
• Dan Feldman
• Daniela Rus

We present a provable, sampling-based approach for generating compact Convolutional Neural Networks (CNNs) by identifying and removing redundant filters from an over-parameterized network. Our algorithm uses a small batch of input data points to assign a saliency score to each filter and constructs an importance sampling distribution where filters that highly affect the output are sampled with correspondingly high probability. In contrast to existing filter pruning approaches, our method is simultaneously data-informed, exhibits provable guarantees on the size and performance of the pruned network, and is widely applicable to varying network architectures and data sets. Our analytical bounds bridge the notions of compressibility and importance of network structures, which gives rise to a fully-automated procedure for identifying and preserving filters in layers that are essential to the network's performance. Our experimental evaluations on popular architectures and data sets show that our algorithm consistently generates sparser and more efficient models than those constructed by existing filter pruning approaches.

IROS Conference 2020 Conference Paper

Roboat II: A Novel Autonomous Surface Vessel for Urban Environments

• Wei Wang 0078
• Tixiao Shan
• Pietro Leoni
• David Fernández-Gutiérrez
• Drew Meyers
• Carlo Ratti
• Daniela Rus

This paper presents a novel autonomous surface vessel (ASV), called Roboat II for urban transportation. Roboat II is capable of accurate simultaneous localization and mapping (SLAM), receding horizon tracking control and estimation, and path planning. Roboat II is designed to maximize the internal space for transport, and can carry payloads several times of its own weight. Moreover, it is capable of holonomic motions to facilitate transporting, docking, and inter-connectivity between boats. The proposed SLAM system receives sensor data from a 3D LiDAR, an IMU, and a GPS, and utilizes a factor graph to tackle the multi-sensor fusion problem. To cope with the complex dynamics in the water, Roboat II employs an online nonlinear model predictive controller (NMPC), where we experimentally estimated the dynamical model of the vessel in order to achieve superior performance for tracking control. The states of Roboat II are simultaneously estimated using a nonlinear moving horizon estimation (NMHE) algorithm. Experiments demonstrate that Roboat II is able to successfully perform online mapping and localization, plan its path and robustly track the planned trajectory in the confined river, implying that this autonomous vessel holds the promise on potential applications in transporting humans and goods in many of the waterways nowadays.

IROS Conference 2020 Conference Paper

Safe Path Planning with Multi-Model Risk Level Sets

• Zefan Huang
• Wilko Schwarting
• Alyssa Pierson
• Hongliang Guo 0003
• Marcelo H. Ang
• Daniela Rus

This paper investigates the safe path planning problem for an autonomous vehicle operating in unstructured, cluttered environments. While some objects may be accurately with canonical perception algorithms, other objects and clutter may be harder to track. We present an approach that combines two methods of risk assessment: for objects with reliable tracking, we use a Gaussian Process (GP) regulated risk map to describe the risk map information; for unknown objects that we fail to accurately track, we compute a Dynamic Risk Density (DRD) from the overall occupancy and velocity field from LiDAR scan snapshots. Several methods are proposed for combining the GP risk map and DRD, and the resultant hybrid risk map is used for the proposed safe path planning algorithm. Experimental results on an autonomous buggy show that the hybrid risk map is able to yield a safe path planner to navigate the autonomous testbed within the cluttered environments.

ICRA Conference 2020 Conference Paper

Sensorization of a Continuum Body Gripper for High Force and Delicate Object Grasping

• Josie Hughes
• Shuguang Li 0005
• Daniela Rus

The goal of achieving `universal grasping' where many objects can be handled with minimal control input is the focus of much research due to potential high impact applications ranging from grocery packing to recycling. However, many of the grippers developed suffer from limited sensing capabilities which can prevent handing of both heavy bulky items and also lightweight delicate objects which require fine control when grasping. Sensorizing such grippers is often challenging due to the highly deformable surfaces. We propose a novel sensing approach which uses highly flexible latex bladders. By measuring changes in the air pressure of the bladders, normal force and longitudinal strain can be measured. These sensors have been integrated into a `Magic Ball' origami gripper to provide both tactile and proprioceptive sensing. The sensors show reasonable sensitivity and repeatability, are durable and low-cost, and can be easily integrated into the gripper without affecting performance. When the sensors are used for classification, they enabled identification of 10 objects with over 90% accuracy, and also allow failure to be detected through slippage detection. A control algorithm has been developed which uses the sensor feedback to extend the capabilities of the gripper to include both delicate and strong grasping. It is shown that this closed loop controller enables delicate grasping of potato chips; 80% of those tested were grasped without damage.

IROS Conference 2020 Conference Paper

Uncertainty Aware Texture Classification and Mapping Using Soft Tactile Sensors

• Alexander Amini
• Jeffrey I. Lipton
• Daniela Rus

Spatial mapping of surface roughness is a critical enabling technology for automating adaptive sanding operations. We leverage GelSight sensors to convert the problem of surface roughness measurement into a vision classification problem. By combining GelSight sensors with Optitrack positioning systems we attempt to develop an accurate spatial mapping of surface roughness that can compare to human touch, the current state of the art for large scale manufacturing. To perform the classification, we propose the use of Bayesian neural networks in conjunction with uncertainty-aware prediction. We compare the sensor and network with a human baseline for both absolute and relative texture classification. To establish a baseline, we collected performance data from humans on their ability to classify materials into 60, 120, and 180 grit sanded pine boards. Our results showed that the probabilistic network performs at the level of human touch for absolute and relative classifications. Using the Bayesian approach enables establishing a confidence bound on our prediction. We were able to integrate the sensor with Optitrack to provide a spatial map of sanding grit applied to pine boards. From this result, we can conclude that GelSight with Bayesian neural networks can learn accurate representations for sanding, and could be a significant enabling technology for closed loop robotic sanding operations.

ICRA Conference 2020 Conference Paper

Weighted Buffered Voronoi Cells for Distributed Semi-Cooperative Behavior

• Alyssa Pierson
• Wilko Schwarting
• Sertac Karaman
• Daniela Rus

This paper introduces the Weighted Buffered Voronoi tessellation, which allows us to define distributed, semicooperative multi-agent navigation policies with guarantees on collision avoidance. We generate the Voronoi cells with dynamic weights that bias the boundary towards the agent with the lower relative weight while always maintaining a buffered distance between two agents. By incorporating agent weights, we can encode selfish or prioritized behavior among agents, where a more selfish agent will have a larger relative cell over less selfish agents. We consider this semi-cooperative since agents do not cooperate in symmetric ways. Furthermore, when all agents start in a collision-free configuration and plan their control actions within their cells, we prove that no agents will collide. Simulations demonstrate the performance of our algorithm for agents navigating to goal locations in a position-swapping game. We observe that agents with more egoistic weights consistently travel shorter paths to their goal than more altruistic agents.

IROS Conference 2019 Conference Paper

A Convolutional Network for Joint Deraining and Dehazing from A Single Image for Autonomous Driving in Rain

• Hao Sun
• Marcelo H. Ang
• Daniela Rus

In this paper, we focus on a rain removal task from a single image of the urban street scene for autonomous driving in rain. We develop a Convolutional Neural Network which takes a rainy image as input, and directly recovers a clean image in the presence of rain streaks, atmospheric veiling effect (haze, fog, mist) caused by distant rain streak accumulation. We propose a synthetic dataset containing images of urban street scenes with different rain intensities, orientations and haziness levels for training and evaluation. We evaluate our method quantitatively and qualitatively on the synthetic data. Experiments show that our model outperforms state-of-the-art methods. We also test our method qualitatively on the real-world data. Our model is fast and it takes 0. 05s for an image of $1024 \times 512$. Our model can be seamlessly integrated with existing image-based high-level perception algorithms for autonomous driving in rain. Experiment results show that our deraining method improves semantic segmentation and object detection largely for autonomous driving in rain.

ICRA Conference 2019 Conference Paper

A Simple Electric Soft Robotic Gripper with High-Deformation Haptic Feedback

• Lillian Chin
• Michelle C. Yuen
• Jeffrey I. Lipton
• Luis H. Trueba
• Rebecca Kramer-Bottiglio
• Daniela Rus

Compliant robotic grippers are more robust to uncertainties in grasping and manipulation tasks, especially when paired with tactile and proprioceptive feedback. Although considerable progress has been made towards achieving proprioceptive soft robotic grippers, current efforts require complex driving hardware or fabrication techniques. In this paper, we present a simple scalable soft robotic gripper integrated with high-deformation strain and pressure sensors. The gripper is composed of structurally-compliant handed shearing auxetic structures actuated by electric motors. Coupling deformable sensors with the compliant grippers enables gripper proprioception and object classification. With this sensorized system, we are able to identify objects' size to within 33% of actual radius and sort objects as hard/soft with 78% accuracy.

ICRA Conference 2019 Conference Paper

A Vacuum-driven Origami "Magic-ball" Soft Gripper

• Shuguang Li 0005
• John J. Stampfli
• Helen J. Xu
• Elian Malkin
• Evelin Villegas Diaz
• Daniela Rus
• Robert J. Wood

Soft robotics has yielded numerous examples of soft grippers that utilize compliance to achieve impressive grasping performances with great simplicity, adaptability, and robustness. Designing soft grippers with substantial grasping strength while remaining compliant and gentle is one of the most important challenges in this field. In this paper, we present a light-weight, vacuum-driven soft robotic gripper made of an origami “magic-ball” and a flexible thin membrane. We also describe the design and fabrication method to rapidly manufacture the gripper with different combinations of low-cost materials for diverse applications. Grasping experiments demonstrate that our gripper can lift a large variety of objects, including delicate foods, heavy bottles, and other miscellaneous items. The grasp force on 3D-printed objects is also characterized through mechanical load tests. The results reveal that our soft gripper can produce significant grasp force on various shapes using negative pneumatic pressure (vacuum). This new gripper holds the potential for many practical applications that require safe, strong, and simple grasping.

ICRA Conference 2019 Conference Paper

Autonomous Latching System for Robotic Boats

• Luis A. Mateos
• Wei Wang 0078
• Banti Gheneti
• Fabio Duarte
• Carlo Ratti
• Daniela Rus

Autonomous robotic boats are devised to transport people and goods similar to self-driving cars. One of the attractive features specially applied in water environment is to dynamically link and join multiple boats into one unit in order to form floating infrastructure such as bridges, markets or concert stages, as well as autonomously self-detach to perform individual tasks. In this paper we present a novel latching system that enables robotic boats to create dynamic united floating infrastructure while overcoming water disturbances. The proposed latching mechanism is based on the spherical joint (ball and socket) that allows rotation and free movements in two planes at the same time. In this configuration, the latching system is capable to securely and efficiently assemble/disassemble floating structures. The vision-based robot controller guides the self-driving robotic boats to latch with high accuracy in the millimeter range. Moreover, in case the robotic boat fails to latch due to harsh weather, the autonomous latching system is capable to recompute and reposition to latch successfully. We present experimental results from latching and docking in indoor environments. Also, we present results in outdoor environments from latching a couple of robotic boats in open water with calm and turbulent currents.

ICRA Conference 2019 Conference Paper

Central Pattern Generators Control of Momentum Driven Compliant Structures

• Stéphane Bonardi
• John W. Romanishin
• Daniela Rus
• Takashi Kubota

We introduce the concept of Momentum Driven Structures (MDS) made of inertially actuated units linked together by compliant elements as a potential solution for rough environments exploration. We propose a control method for MDS based on the bio-inspired concept of Central Pattern Generator (CPG) and study in simulation the impact of compliance distribution on locomotion performance using population based optimization techniques. Our results suggest that compliant structures outperform their rigid counterparts in terms of distance traveled. In addition, we show that co-evolved structures perform only marginally better than their control-only optimized equivalent, highlighting the fact that compliance modulation may not be a significant asset in such experiments, considering the related hardware complexity it introduces.

ICRA Conference 2019 Conference Paper

ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Robotics

• Yuanming Hu
• Jiancheng Liu
• Andrew Spielberg
• Joshua B. Tenenbaum
• William T. Freeman
• Jiajun Wu 0001
• Daniela Rus
• Wojciech Matusik

Physical simulators have been widely used in robot planning and control. Among them, differentiable simulators are particularly favored, as they can be incorporated into gradient-based optimization algorithms that are efficient in solving inverse problems such as optimal control and motion planning. Therefore, rigid body simulators and recently their differentiable variants are studied extensively. Simulating deformable objects is, however, more challenging compared to rigid body dynamics. The underlying physical laws of deformable objects are more complex, and the resulting systems have orders of magnitude more degrees of freedom and there-fore they are significantly more computationally expensive to simulate. Computing gradients with respect to physical design or controller parameters is typically even more computationally challenging. In this paper, we propose a real-time, differentiable hybrid Lagrangian-Eulerian physical simulator for deformable objects, ChainQueen, based on the Moving Least Squares Material Point Method (MLS-MPM). MLS-MPM can simulate deformable objects with collisions and can be seamlessly incorporated into soft robotic systems. We demonstrate that our simulator achieves high precision in both forward simulation and backward gradient computation. We have successfully employed it in a diverse set of inference, control and co-design tasks for soft robotics.

IROS Conference 2019 Conference Paper

Context and Intention Aware Planning for Urban Driving

• Malika Meghjani
• Yuanfu Luo
• Qi Heng Ho
• Panpan Cai
• Shashwat Verma
• Daniela Rus
• David Hsu

We present a novel autonomous driving system which uses the road contextual information and intentions of other road users for urban driving. Unlike highways, urban environments require the drivers to follow traffic signs and signals while using their best judgment for anomalous situations. In such scenarios, a self-driving car needs to understand and take into account the uncertainties in the environment to plan and decide its action accordingly. Our planner models the intentions of the surrounding vehicles leveraging a neural network, and integrates the road contextual information to reduce its environment uncertainties and also speed up the decision making process. We validate our planner in simulation and in a real urban environment. Our experimental results show that integrating intention inference and road contextual information for prediction, planning and decision making help improve safety and efficiency of our autonomous driving system.

ICRA Conference 2019 Conference Paper

Coordinated Control of a Reconfigurable Multi-Vessel Platform: Robust Control Approach

• Shinkyu Park
• Erkan Kayacan
• Carlo Ratti
• Daniela Rus

We propose a feedback control system for a reconfigurable multi-vessel platform. The platform consists of N propeller-driven vessels each of which is capable of latching to another vessel to form a rigid body of connected vessels. The main technical challenges are that i) depending on configurations of the platform the dynamic model would be different, and ii) the number of control variables in control system design increases as does the total number of vessels in the platform. To address these challenges, we develop a coordinated robust control scheme. Through experiments we assess trajectory tracking and disturbance attenuation performance of the control scheme in various configurations of the platform. Experiment results yield that average position and orientation tracking error are approximately 0. 09m and 3°, and the maximum tracking error-to-disturbance ratio is 1. 12.

IROS Conference 2019 Conference Paper

Decentralized Control for 3D M-Blocks for Path Following, Line Formation, and Light Gradient Aggregation

• John W. Romanishin
• John Mamish
• Daniela Rus

This paper presents a decentralized control frame-work for lattice-based Modular Self-Reconfigurable Robots (MSRR) which utilizes a novel magnetic fiducial system to facilitate neighbor identification and to enable algorithms which promise scalable functionality for systems with many modules. In this system individual modules autonomously follow simple behaviors while periodically accepting input from a centralized controller. This system is demonstrated with three initial behaviors: (1) Path following: modules follow a three dimensional path based on magnetic fiducial tags embedded in their neighbors, (2) Line formation: modules transform from a 3D structure into a line following a partially decentralized control algorithm, and (3) Light gradient aggregation: the formation of a group of modules guided by a global stimulus (i. e. visible light). This paper provides details of the neighbor identification system, introduces the three behaviors and presents the results of physical experiments performed with a system of twelve 3D M-Block robotic modules.

IROS Conference 2019 Conference Paper

Decentralized Pose Control of Modular Reconfigurable Robots Operating in Liquid Environments

• João V. Amorim Marques
• Anil Özdemir
• Matthew J. Doyle
• Daniela Rus
• Roderich Groß

Modular reconfigurable robots are touted for their flexibility, as their bodies can assume a wide range of shapes. A particular challenge is to make them move efficiently in 3D without compromising the scalability of the system. This paper proposes decentralized and fully reactive controllers for pose control of 3D modular reconfigurable robots. The robots operate in liquid environments, and move by routing fluid through themselves. Each module uses only two bits of sensory information per face. Additionally, the modules can use up to five bits of information that are exchanged via shared power lines. We prove that robots of convex shape are guaranteed to reach a goal object with a preferred orientation. Using computer simulations of Modular Hydraulic Propulsion robots, all controllers are assessed for different environments, system sizes and noise, and their performances compared against a centralized controller. Given the simplicity of the solutions, modules could be realized at scales below a millimeter-cube, where robots of high spatial resolution could perform accurate movements in 3D liquid environments.

IROS Conference 2019 Conference Paper

Dynamic Control of Soft Robots with Internal Constraints in the Presence of Obstacles

• Cosimo Della Santina
• Antonio Bicchi
• Daniela Rus

The development of effective reduced order models for soft robots is paving the way toward the development of a new generation of model based techniques, which leverage classic rigid robot control. However, several soft robot features differentiate the soft-bodied case from the rigid-bodied one. First, soft robots are built to work in the environment, so the presence of obstacles in their path should always be explicitly accounted by their control systems. Second, due to the complex kinematics, the actuation of soft robots is mapped to the state space nonlinearly resulting in spaces with different sizes. Moreover, soft robots often include internal constraints and thus actuation is typically limited in the range of action and it is often unidirectional. This paper proposes a control pipeline to tackle the challenge of controlling soft robots with internal constraints in environments with obstacles. We show how the constraints on actuation can be propagated and integrated with geometrical constraints, taking into account physical limits imposed by the presence of obstacles. We present a hierarchical control architecture capable of handling these constraints, with which we are able to regulate the position in space of the tip of a soft robot with the discussed characteristics.

ICRA Conference 2019 Conference Paper

Dynamic Risk Density for Autonomous Navigation in Cluttered Environments without Object Detection

• Alyssa Pierson
• Cristian Ioan Vasile
• Anshula Gandhi
• Wilko Schwarting
• Sertac Karaman
• Daniela Rus

In this paper, we examine the problem of navigating cluttered environments without explicit object detection and tracking. We introduce the dynamic risk density to map the congestion density and spatial flow of the environment to a cost function for the agent to determine risk when navigating that environment. We build upon our prior work, wherein the agent maps the density and motion of objects to an occupancy risk, then navigate the environment over a specified risk level set. Here, the agent does not need to identify objects to compute the occupancy risk, and instead computes this cost function using the occupancy density and velocity fields around them. Simulations show how this dynamic risk density encodes movement information for the ego agent and closely models the object-based congestion cost. We implement our dynamic risk density on an autonomous wheelchair and show how it can be used for navigating unstructured, crowded and cluttered environments.

IROS Conference 2019 Conference Paper

Infrastructure-free NLoS Obstacle Detection for Autonomous Cars

• Felix Naser
• Igor Gilitschenski
• Alexander Amini
• Christina Liao
• Guy Rosman
• Sertac Karaman
• Daniela Rus

Current perception systems mostly require direct line of sight to anticipate and ultimately prevent potential collisions at intersections with other road users. We present a fully integrated autonomous system capable of detecting shadows or weak illumination changes on the ground caused by a dynamic obstacle in NLoS scenarios. This additional virtual sensor “ShadowCam” extends the signal range utilized so far by computer-vision ADASs. We show that (1) our algorithm maintains the mean classification accuracy of around 70% even when it doesn’t rely on infrastructure – such as AprilTags - as an image registration method. We validate (2) in real-world experiments that our autonomous car driving in night time conditions detects a hidden approaching car earlier with our virtual sensor than with the front facing 2-D LiDAR.

IROS Conference 2019 Conference Paper

Learning-based Nonlinear Model Predictive Control of Reconfigurable Autonomous Robotic Boats: Roboats

• Erkan Kayacan
• Shinkyu Park
• Carlo Ratti
• Daniela Rus

This paper presents a Learning-based Nonlinear Model Predictive Control (LB-NMPC) algorithm for reconfigurable autonomous vessels to facilitate high-accurate path tracking. Each vessel is designed to latch to a pre-defined point of another vessel that allows the vessels to form a rigid body. The number of possible configurations of such vessels exponentially grows as the total number of vessels increases, which imposes a technical challenge in modeling and identification. In this work, we propose a framework consisting of a real-time parameter estimator and a feedback control strategy, which is capable of ensuring high-accurate path tracking for any feasible configuration of vessels. Novelty of our method is in that the parameter is estimated on-line and adjusts control parameters (e. g. , cost function and dynamic model) simultaneously to improve path-tracking performance. Through experiments on different configurations of connected-vessels, we demonstrate stability of our proposed approach and its effectiveness in high-accuracy in path tracking.

NeurIPS Conference 2019 Conference Paper

Learning-In-The-Loop Optimization: End-To-End Control And Co-Design Of Soft Robots Through Learned Deep Latent Representations

• Andrew Spielberg
• Allan Zhao
• Yuanming Hu
• Tao Du
• Wojciech Matusik
• Daniela Rus

Soft robots have continuum solid bodies that can deform in an infinite number of ways. Controlling soft robots is very challenging as there are no closed form solutions. We present a learning-in-the-loop co-optimization algorithm in which a latent state representation is learned as the robot figures out how to solve the task. Our solution marries hybrid particle-grid-based simulation with deep, variational convolutional autoencoder architectures that can capture salient features of robot dynamics with high efficacy. We demonstrate our dynamics-aware feature learning algorithm on both 2D and 3D soft robots, and show that it is more robust and faster converging than the dynamics-oblivious baseline. We validate the behavior of our algorithm with visualizations of the learned representation.

IROS Conference 2019 Conference Paper

Modular Volumetric Actuators Using Motorized Auxetics

• Jeffrey I. Lipton
• Lillian Chin
• Jacob Miske
• Daniela Rus

Volume change has become a critical actuation method in robotics. However, the need for fluid flow or thermal processes to generate volume changes limits the durability, speed, and efficiency of these actuators. In this paper, we develop a new electromechanical actuator that volumetrically expands. By combining auxetic materials with a servo, we produce a simple isotropically expanding actuator that can be modularly composed. We discuss the symmetry considerations in selecting an appropriate auxetic framework for our actuator, eventually choosing a double-layered polyhedral auxetic design. Characterization shows that a single actuator can expand in radius to 119% of the original size and generate 90N of force, while maintaining a small package and a speedy expansion / contraction cycle. Finally, we demonstrate the modularity of our actuators by linking three actuators to create a vertical tube-crawling robot. The small package and fast cycle time of our system highlight how viable these electromechanical volumetric actuators can be as an important actuator modality.

IROS Conference 2019 Conference Paper

Online System Identification Algorithm without Persistent Excitation for Robotic Systems: Application to Reconfigurable Autonomous Vessels

• Erkan Kayacan
• Shinkyu Park
• Carlo Ratti
• Daniela Rus

This paper investigates an online system identification problem of estimating unknown parameters in nonlinear system dynamics in the absence of persistently excitation. To estimate parameters, we develop an algorithm that updates parameter estimates using sensor data and a basis that is built on a finite number of recorded sensor data. Based on our proposed approach we show that the algorithm achieves exponential convergence in both state and parameter estimation errors without the persistent excitation condition. We demonstrate the effectiveness of the proposed approach using both simulations and experiments on a reconfiguration autonomous multi-vessel platform: Simulation results illustrate that the parameter estimated by the developed algorithm converge to their ground truths. Experiment results validate the performance of the developed algorithm in estimating platform’s system parameters across different multi-vessel configurations.

ICRA Conference 2019 Conference Paper

Optimizing Vehicle Distributions and Fleet Sizes for Shared Mobility-on-Demand

• Alex Wallar
• Javier Alonso-Mora
• Daniela Rus

Mobility-on-demand (MoD) systems are revolutionizing urban transit with the introduction of ride-sharing. Such systems have the potential to reduce vehicle congestion and improve accessibility of a city's transportation infrastructure. Recently developed algorithms can compute routes for vehicles in real-time for a city-scale volume of requests while allowing vehicles to carry multiple passengers at the same time. However, these algorithms focus on optimizing the performance for a given fleet of vehicles and do not tell us how many vehicles are needed to service all the requests. In this paper, we present an offline method to optimize the vehicle distributions and fleet sizes on historical demand data for MoD systems that allow passengers to share vehicles. We present an algorithm to determine how many vehicles are needed, where they should be initialized, and how they should be routed to service all the travel demand for a given period of time. Evaluation using 23, 529, 740 historical taxi requests from one month in Manhattan shows that on average 2864 four passenger vehicles are needed to service all of the taxi demand in a day with an average added travel delay of 2. 8 mins.

IROS Conference 2019 Conference Paper

Roboat: An Autonomous Surface Vehicle for Urban Waterways

• Wei Wang 0078
• Banti Gheneti
• Luis A. Mateos
• Fabio Duarte
• Carlo Ratti
• Daniela Rus

Unmanned surface vehicles (USVs) are typically designed for open area marine applications. In this paper, we present a new autonomy system (Roboat) for urban waterways which requires robust localization, perception, planning, and control. A novel localization system, based on the extended Kalman filter (EKF), is proposed for USVs, which utilizes LiDAR, camera, and IMU to provide a decimeter-level precision in dynamic GPS-attenuated urban waterways. Area and shape filters are proposed to crop water reflections and street obstacles from a pointcloud. Euclidean clustering and multi-object contour tracking are then introduced to detect and track the static and moving objects reliably in urban waters. An efficient path planner is tailored to calculate optimal trajectories to avoid these static and dynamic obstacles. Lastly, a nonlinear model predictive control (NMPC) scheme with full state integration is formulated for the four-control-input robot to accurately track the trajectory from the planner in rough water. Extensive experiments show that the robot is able to autonomously navigate in both the indoor waterway and the cluttered outdoor waterway in the presence of static and dynamic obstacles, implying that Roboat could have a great impact on the future of transportation in many coastal and riverside cities.

IROS Conference 2019 Conference Paper

Safe Path Planning with Gaussian Process Regulated Risk Map

• Hongliang Guo 0003
• Zehui Meng
• Zefan Huang
• Wei Kang Leong
• Ziyue Chen
• Malika Meghjani
• Marcelo H. Ang
• Daniela Rus

Government data identifies driver behaviour errors as a factor in 94% of car crashes, and autonomous vehicles (AVs), which avoids risky driver behaviours completely, are expected to reduce the number of road crashes significantly. Thus, one of the central focuses of developing AVs is to ensure safety during navigation. However, in reality, AV safety has been far below its expectation, and so far, no government has allowed for complete autonomous driving without human supervision. This paper proposes a dynamic safe path planning algorithm for AVs with Gaussian process regulated risk map. By reasonably assuming that the output of the object detection and tracking module follows a multi-variate Gaussian distribution, we put forward a safe path planning paradigm with Gaussian process regulated risk map, ensuring safety with high confidence. Both simulation results and in-vehicle tests demonstrate the effectiveness of the proposed algorithm.

IROS Conference 2019 Conference Paper

Sharing is Caring: Socially-Compliant Autonomous Intersection Negotiation

• Noam Buckman
• Alyssa Pierson
• Wilko Schwarting
• Sertac Karaman
• Daniela Rus

Current methods for autonomous management use strict first-come, first-serve (FCFS) ordering to manage incoming autonomous vehicles at an intersection. In this work, we present a coordination policy that swaps agent ordering to increase the system-wide performance while ensuring that the swaps are socially compliant. By considering an agent’s Social Value Orientation (SVO), a social psychology metric for their willingness to help another vehicle, the central coordinator can reduce system delays while ensuring each individual vehicle increases their own utility. The FCFS-SVO algorithm is both computationally tractable and accounts for a variety of real-world agent types, such as human drivers and a variety of social orientations. Simulation results show that average vehicle delays decrease with swapping by enabling cooperation between agents. In addition, we show that the proportion of human drivers, as well as, the distribution of prosocial and egoistic vehicles in the system can have a prominent effect on the performance of the system.

ICRA Conference 2019 Conference Paper

Sharing the Load: Human-Robot Team Lifting Using Muscle Activity

• Joseph DelPreto
• Daniela Rus

Seamless communication of desired motions and goals is essential for enabling effective physical human-robot collaboration. In such cases, muscle activity measured via surface electromyography (EMG) can provide insight into a person's intentions while minimally distracting from the task. The presented system uses two muscle signals to create a control framework for team lifting tasks in which a human and robot lift an object together. A continuous setpoint algorithm uses biceps activity to estimate changes in the user's hand height, and also allows the user to explicitly adjust the robot by stiffening or relaxing their arm. In addition to this pipeline, a neural network trained only on previous users classifies biceps and triceps activity to detect up or down gestures on a rolling basis; this enables finer control over the robot and expands the feasible workspace. The resulting system is evaluated by 10 untrained subjects performing a variety of team lifting and assembly tasks with rigid and flexible objects.

ICRA Conference 2019 Conference Paper

Variational End-to-End Navigation and Localization

• Alexander Amini
• Guy Rosman
• Sertac Karaman
• Daniela Rus

Deep learning has revolutionized the ability to learn “end-to-end” autonomous vehicle control directly from raw sensory data. While there have been recent extensions to handle forms of navigation instruction, these works are unable to capture the full distribution of possible actions that could be taken and to reason about localization of the robot within the environment. In this paper, we extend end-to-end driving networks with the ability to perform point-to-point navigation as well as probabilistic localization using only noisy GPS data. We define a novel variational network capable of learning from raw camera data of the environment as well as higher level roadmaps to predict (1) a full probability distribution over the possible control commands; and (2) a deterministic control command capable of navigating on the route specified within the map. Additionally, we formulate how our model can be used to localize the robot according to correspondences between the map and the observed visual road topology, inspired by the rough localization that human drivers can perform. We test our algorithms on real-world driving data that the vehicle has never driven through before, and integrate our point-topoint navigation algorithms onboard a full-scale autonomous vehicle for real-time performance. Our localization algorithm is also evaluated over a new set of roads and intersections to demonstrates rough pose localization even in situations without any GPS prior.

ICRA Conference 2018 Conference Paper

A General Pipeline for 3D Detection of Vehicles

• Xinxin Du
• Marcelo H. Ang
• Sertac Karaman
• Daniela Rus

Autonomous driving requires 3D perception of vehicles and other objects in the in environment. Much of the current methods support 2D vehicle detection. This paper proposes a flexible pipeline to adopt any 2D detection network and fuse it with a 3D point cloud to generate 3D information with minimum changes of the 2D detection networks. To identify the 3D box, an effective model fitting algorithm is developed based on generalised car models and score maps. A two-stage convolutional neural network (CNN) is proposed to refine the detected 3D box. This pipeline is tested on the KITTI dataset using two different 2D detection networks. The 3D detection results based on these two networks are similar, demonstrating the flexibility of the proposed pipeline. The results rank second among the 3D detection algorithms, indicating its competencies in 3D detection.

ICRA Conference 2018 Conference Paper

Autonomous Vehicle Navigation in Rural Environments Without Detailed Prior Maps

• Teddy Ort
• Liam Paull
• Daniela Rus

State-of-the-art autonomous driving systems rely heavily on detailed and highly accurate prior maps. However, outside of small urban areas, it is very challenging to build, store, and transmit detailed maps since the spatial scales are so large. Furthermore, maintaining detailed maps of large rural areas can be impracticable due to the rapid rate at which these environments can change. This is a significant limitation for the widespread applicability of autonomous driving technology, which has the potential for an incredibly positive societal impact. In this paper, we address the problem of autonomous navigation in rural environments through a novel mapless driving framework that combines sparse topological maps for global navigation with a sensor-based perception system for local navigation. First, a local navigation goal within the sensor view of the vehicle is chosen as a waypoint leading towards the global goal. Next, the local perception system generates a feasible trajectory in the vehicle frame to reach the waypoint while abiding by the rules of the road for the segment being traversed. These trajectories are updated to remain in the local frame using the vehicle's odometry and the associated uncertainty based on the least-squares residual and a recursive filtering approach, which allows the vehicle to navigate road networks reliably, and at high speed, without detailed prior maps. We demonstrate the performance of the system on a full-scale autonomous vehicle navigating in a challenging rural environment and benchmark the system on a large amount of collected data.

ICRA Conference 2018 Conference Paper

Conditional Compatibility Branch and Bound for Feature Cloud Matching

• Xiaotong Shen
• Marcelo H. Ang
• Daniela Rus

In this paper, we consider the problem of data association in feature cloud matching. While Joint Compatibility (JC) test is a widely adopted technique for searching the global optimal data association, it becomes less restrictive as more features are well matched. The early well-matched features contribute little to total matching cost while the gating threshold increases in the chi-square test, which allows the acceptance of bad feature pairings in the last step. In this paper, we propose the Conditional Compatibility (CC) test, which is not only more restrictive than JC test, but also probabilistically sound. The proposed test of a new feature pairing is based on the conditional probability distribution of feature locations given the early pairings. CC test can be added into any JC test based search algorithm, such as Joint Compatibility Branch and Bound (JCBB), Incremental Posterior Joint Compatibility (IPJC) and FastJCBB, without increasing much computational complexity. The more restrictive criterion of accepting a feature pairing, not only helps to reject bad associations, but also bounds the search space, which substantially improves the search efficiency. The real matching experiments justify that our algorithm produces better feature cloud matching results in a more efficient manner.

ICRA Conference 2018 Conference Paper

Design. Modeling, and Nonlinear Model Predictive Tracking Control of a Novel Autonomous Surface Vehicle

• Wei Wang 0078
• Luis A. Mateos
• Shinkyu Park
• Pietro Leoni
• Banti Gheneti
• Fabio Duarte
• Carlo Ratti
• Daniela Rus

In this paper, we present the design, modeling, and real-time nonlinear model predictive control (NMPC) of an autonomous robotic boat. The robot is easy to manufacture, highly maneuverable, and capable of accurate trajectory tracking in both indoor and outdoor environments. In particular, a cross type four-thruster configuration is proposed for the robotic boat to produce efficient holonomic motions. The robot prototype is rapidly 3D-printed and then sealed by adhering several layers of fiberglass. To achieve accurate tracking control, we formulate an NMPC strategy for the four-control-input boat with control input constraints, where the nonlinear dynamic model includes a Coriolis and centripetal matrix, the hydrodynamic added mass, and damping. By integrating “GPS” modules and an inertial measurement unit (IMU) into the robot, we demonstrate accurate trajectory tracking of the robotic boat along preplanned paths in both a swimming pool and a natural river. Furthermore, the code generation strategy employed in our paper yields a two order of magnitude improvement in the run time of the NMPC algorithm compared to similar systems. The robot is designed to form the basis for surface swarm robotics testbeds, on which collective algorithms for surface transportation and self-assembly of dynamic floating infrastructures can be assessed.

ICRA Conference 2018 Conference Paper

Joint Multi-Policy Behavior Estimation and Receding-Horizon Trajectory Planning for Automated Urban Driving

• Bingyu Zhou
• Wilko Schwarting
• Daniela Rus
• Javier Alonso-Mora

When driving in urban environments, an autonomous vehicle must account for the interaction with other traffic participants. It must reason about their future behavior, how its actions affect their future behavior, and potentially consider multiple motion hypothesis. In this paper we introduce a method for joint behavior estimation and trajectory planning that models interaction and multi-policy decision-making. The method leverages Partially Observable Markov Decision Processes to estimate the behavior of other traffic participants given the planned trajectory for the ego-vehicle, and Receding-Horizon Control for generating safe trajectories for the ego-vehicle. To achieve safe navigation we introduce chance constraints over multiple motion policies in the receding-horizon planner. These constraints account for uncertainty over the behavior of other traffic participants. The method is capable of running in real-time and we show its performance and good scalability in simulated multi-vehicle intersection scenarios.

ICRA Conference 2018 Conference Paper

Learning Steering Bounds for Parallel Autonomous Systems

• Alexander Amini
• Liam Paull
• Thomas Balch
• Sertac Karaman
• Daniela Rus

Deep learning has been successfully applied to “end-to-end” learning of the autonomous driving task, where a deep neural network learns to predict steering control commands from camera data input. However, the learned representations do not support higher-level decision making required for autonomous navigation, nor the uncertainty estimates required for parallel autonomy, where vehicle control is shared between human and robot. This paper tackles the problem of learning a representation to predict a continuous control probability distribution, and thus steering control options and bounds for those options, which can be used for autonomous navigation. Each mode of the distribution encodes a possible macro-action that the system could execute at that instant, and the covariances of the modes place bounds on safe steering control values. Our approach has the added advantage of being trained on unlabeled data collected from inexpensive cameras. The deep neural network based algorithm generates a probability distribution over the space of steering angles, from which we leverage Variational Bayesian methods to extract a mixture model and compute the different possible actions in the environment. A bound, which the autonomous vehicle must respect in our parallel autonomy setting, is then computed for each of these actions. We evaluate our approach on a challenging dataset containing a wide variety of driving conditions, and show that our algorithm is capable of parameterizing Gaussian Mixture Models for possible actions, and extract steering bounds with a mean error of only 2 degrees. Additionally, we demonstrate our system working on a full scale autonomous vehicle and evaluate its ability to successful handle various different parallel autonomy situations.

ICRA Conference 2018 Conference Paper

Multi-Vehicle Motion Planning for Social Optimal Mobility-on-Demand

• Jesper Karlsson
• Cristian Ioan Vasile
• Jana Tumova
• Sertac Karaman
• Daniela Rus

In this paper we consider a fleet of self-driving cars operating in a road network governed by rules of the road, such as the Vienna Convention on Road Traffic, providing rides to customers to serve their demands with desired deadlines. We focus on the associated motion planning problem that trades-off the demands' delays and level of violation of the rules of the road to achieve social optimum among the vehicles. Due to operating in the same environment, the interaction between the cars must be taken into account, and can induce further delays. We propose an integrated route and motion planning approach that achieves scalability with respect to the number of cars by resolving potential collision situations locally within so-called bubble spaces enclosing the conflict. The algorithms leverage the road geometries, and perform joint planning only for lead vehicles in the conflict and use queue scheduling for the remaining cars. Furthermore, a framework for storing previously resolved conflict situations is proposed, which can be use for quick querying of joint motion plans. We show the mobility-on-demand setup and effectiveness of the proposed approach in simulated case studies involving up to 10 self-driving vehicles.

ICRA Conference 2018 Conference Paper

Navigating Congested Environments with Risk Level Sets

• Alyssa Pierson
• Wilko Schwarting
• Sertac Karaman
• Daniela Rus

In this paper, we address the problem of navigating in a cluttered environment by introducing a congestion cost that maps the density and motion of objects to an occupancy risk. We propose that an agent can choose a “risk level set” from this cost function and construct a planning space from this set. In choosing different levels of risk, the agent adjusts its interactions with the other agents. From the assumption that agents are self-preserving, we show that any agent planning within their risk level set will avoid collisions with other agents. We then present an application of planning with risk level sets in the framework of an autonomous vehicle driving along a highway. Using the risk level sets, the agent can determine safe zones when planning a sequence of lane changes. Through simulations in Matlab, we demonstrate how the choice of risk threshold manifests as aggressive or conservative behavior.

ICRA Conference 2018 Conference Paper

Programmable Medicine: Autonomous, Ingestible, Deployable Hydrogel Patch and Plug for Stomach Ulcer Therapy

• Alexis du Plessis d'Argentre
• Samuel Perry
• Yoshitaka Iwata
• Haruna Iwasaki
• Eiji Iwase
• Assunta Fabozzo
• Iain Will
• Daniela Rus

Gastric ulcer is a chronic and complex (and often complete) erosion of the stomach wall that happens as a complication of a previous chronic, inflammatory process. It represents a catastrophic situation in which the patient is critical and its conditions need to be treated fast. This study presents a remotely navigatable and deployable ingestible patch and plug for gastric ulcer treatment. The patch/plug structure is made of agarose hydrogel that can change rigidity through hydration and dehydration. When dehydrated, it is rigid and can maintain a folded configuration so it can be ingested as a “pill”. This can be guided to the targeted location by a magnetic field, and be deployed instantly by hydration, namely by supplying water from the mouth. Due to the deployable origami design, it exhibits an expansion of 10 times its initial surface area, making the device suitable for the use of dressing a surface as a patch, and filling a hole as a plug.

ICRA Conference 2018 Conference Paper

Robot Assisted Carpentry for Mass Customization

• Jeffrey I. Lipton
• Adriana Schulz
• Andrew Spielberg
• Luite Trueba
• Wojciech Matusik
• Daniela Rus

Despite the ubiquity of carpentered items, the customization of carpentered items remains labor intensive. The generation of laymen editable templates for carpentry is difficult. Current design tools rely heavily on CNC fabrication, limiting applicability. We develop a template based system for carpentry and a robotic fabrication system using mobile robots and standard carpentry tools. Our end-to-end design and fabrication tool democratizes design and fabrication of carpentered items. Our method combines expert knowledge for template design, allows laymen users to customize and verify specific designs, and uses robotics system to fabricate parts. We validate our system using multiple designs to make customizable, verifiable templates and fabrication plans and show an end-to-end example that was designed, manufactured, and assembled using our tools.

IROS Conference 2018 Conference Paper

Robust LIDAR Localization for Autonomous Driving in Rain

• Chen Zhang 0018
• Marcelo H. Ang
• Daniela Rus

This paper introduces a map-based localization method aiming to increase robustness in rainy conditions. This method utilizes two types of features: ground reflectivity features and vertical features extracted from 3D LIDAR scans and builds vehicle pose belief with two filters: a histogram filter and a particle filter. The posterior distributions from the two filters are integrated to estimate vehicle poses. This method exploits advantages of both features and filters, compensating respective weakness to deal with complex urban environments. Testing was performed in the fair and rainy weather. Road test results prove robustness and reliability of the proposed method.

ICRA Conference 2018 Conference Paper

Task-Specific Sensor Planning for Robotic Assembly Tasks

• Guy Rosman
• Changhyun Choi
• Mehmet Remzi Dogar
• John W. Fisher III
• Daniela Rus

When performing multi-robot tasks, sensory feedback is crucial in reducing uncertainty for correct execution. Yet the utilization of sensors should be planned as an integral part of the task planning, taken into account several factors such as the tolerance of different inferred properties of the scene and interaction with different agents. In this paper we handle this complex problem in a principled, yet efficient way. We use surrogate predictors based on open-loop simulation to estimate and bound the probability of success for specific tasks. We reason about such task-specific uncertainty approximants and their effectiveness. We show how they can be incorporated into a multi-robot planner, and demonstrate results with a team of robots performing assembly tasks.

IROS Conference 2018 Conference Paper

Variational Autoencoder for End-to-End Control of Autonomous Driving with Novelty Detection and Training De-biasing

• Alexander Amini
• Wilko Schwarting
• Guy Rosman
• Brandon Araki
• Sertac Karaman
• Daniela Rus

This paper introduces a new method for end-to-end training of deep neural networks (DNNs) and evaluates it in the context of autonomous driving. DNN training has been shown to result in high accuracy for perception to action learning given sufficient training data. However, the trained models may fail without warning in situations with insufficient or biased training data. In this paper, we propose and evaluate a novel architecture for self-supervised learning of latent variables to detect the insufficiently trained situations. Our method also addresses training data imbalance, by learning a set of underlying latent variables that characterize the training data and evaluate potential biases. We show how these latent distributions can be leveraged to adapt and accelerate the training pipeline by training on only a fraction of the total dataset. We evaluate our approach on a challenging dataset for driving. The data is collected from a full-scale autonomous vehicle. Our method provides qualitative explanation for the latent variables learned in the model. Finally, we show how our model can be additionally trained as an end-to-end controller, directly outputting a steering control command for an autonomous vehicle.

ICRA Conference 2018 Conference Paper

Vehicle Detection, Tracking and Behavior Analysis in Urban Driving Environments Using Road Context

• Shashwat Verma
• You Hong Eng
• Hai Xun Kong
• Hans Andersen
• Malika Meghjani
• Wei Kang Leong
• Xiaotong Shen
• Chen Zhang 0018

We present a real-time vehicle detection and tracking system to accomplish the complex task of driving behavior analysis in urban environments. We propose a robust fusion system that combines a monocular camera and a 2D Lidar. This system takes advantage of three key components: robust vehicle detection using deep learning techniques, high precision range estimation from Lidar, and road context from the prior map knowledge. The camera and Lidar sensor fusion, data association and track management are all performed in the global map coordinate system by taking into account the sensors' characteristics. Lastly, behavior reasoning is performed by examining the tracked vehicle states in the lane coordinate system in which the road context is encoded. We validated our approach by tracking a leading vehicle while it performed usual urban driving behaviors such as lane keeping, stop-and-go at intersections, lane changing, overtaking and turning. The leading vehicle was tracked consistently throughout the 2. 3 km route and its behavior was classified reliably.

IROS Conference 2018 Conference Paper

Vehicle Rebalancing for Mobility-on-Demand Systems with Ride-Sharing

• Alex Wallar
• Menno Van Der Zee
• Javier Alonso-Mora
• Daniela Rus

Recent developments in Mobility-on-Demand (MoD) systems have demonstrated the potential of road vehicles as an efficient mode of urban transportation Newly developed algorithms can compute vehicle routes in real-time for batches of requests and allow for multiple requests to share vehicles. These algorithms have primarily focused on optimally producing vehicle schedules to pick up and drop off requests. The redistribution of idle vehicles to areas of high demand, known as rebalancing, on the contrary has received little attention in the context of ride-sharing. In this paper, we present a method to rebalance idle vehicles in a ride-sharing enabled MoD fleet. This method consists of an algorithm to optimally partition the fleet operating area into rebalancing regions, an algorithm to determine a real-time demand estimate for every region using incoming requests, and an algorithm to optimize the assignment of idle vehicles to these rebalancing regions using an integer linear program. Evaluation with historical taxi data from Manhattan shows that we can service 99. 8% of taxi requests in Manhattan using 3000 vehicles with an average waiting time of 57. 4 seconds and an average in-car delay of 13. 7 seconds. Moreover, we can achieve a higher service rate using 2000 vehicles than prior work achieved with 3000. Furthermore, with a fleet of 3000 vehicles, we reduce the average travel delay by 86%, the average waiting time by 37%, and the amount of ignored requests by 95% compared to earlier work at the expense of an increased distance travelled by the fleet.

ICRA Conference 2017 Conference Paper

A portable, 3D-printing enabled multi-vehicle platform for robotics research and education

• Jingjin Yu
• Shuai D. Han
• Wei N. Tang
• Daniela Rus

microMVP is an affordable, portable, and open source micro-scale mobile robot platform designed for robotics research and education. As a complete and unique multi-vehicle platform enabled by 3D printing and the maker culture, microMVP can be easily reproduced and requires little maintenance: a set of six micro vehicles, each measuring 8 × 5 × 6 cubic centimeters and weighing under 100 grams, and the accompanying tracking platform can be fully assembled in under two hours, all from readily available components. In this paper, we describe microMVP's hardware and software architecture, and the design thoughts that go into the making of the platform. The capabilities of microMVP APIs are then demonstrated with several single- and multi-robot path and motion planning algorithms. microMVP supports all common operation systems.

ICRA Conference 2017 Conference Paper

Autonomous locomotion of a miniature, untethered origami robot using hall effect sensor-based magnetic localization

• Steven Guitron
• Anubhav Guha
• Shuguang Li 0005
• Daniela Rus

Autonomous control of magnetically-actuated miniature robots enables greater versatility and complexity in function but has so far been a challenge to implement. In this paper, we present closed-loop position feedback control of a miniature origami robot utilizing its integrated magnet and an array of Hall effect sensors, enabling the robot's actuation, detection, and locomotion to be initiated from outside its body. An array of 33 Hall effect sensors arranged in repeated triangles cover a range of 60 mm by 75 mm, enabling position detection of the robot with average error of 0. 995±0. 520 mm. The robot's speed response to applied magnetic field was characterized, and a controller was designed to actuate the robot dependably. We demonstrate autonomous movement of the robot along preplanned paths and the viability of magnetic detection and actuation.

IROS Conference 2017 Conference Paper

Car detection for autonomous vehicle: LIDAR and vision fusion approach through deep learning framework

• Xinxin Du
• Marcelo H. Ang
• Daniela Rus

Technologies in autonomous vehicles have seen dramatic advances in recent years; however, it still lacks of robust perception systems for car detection. With the recent development in deep learning research, in this paper, we propose a LIDAR and vision fusion system for car detection through the deep learning framework. It consists of three major parts. The first part generates seed proposals for potential car locations in the image by taking LIDAR point cloud into account. The second part refines the location of the proposal boxes by exploring multi-layer information in the proposal network and the last part carries out the final detection task through a detection network which shares part of the layers with the proposal network. The evaluation shows that the proposed framework is able to generate high quality proposal boxes more efficiently (77. 6% average recall) and detect the car at the state of the art accuracy (89. 4% average precision). With further optimization of the framework structure, it has great potentials to be implemented onto the autonomous vehicle.

ICML Conference 2017 Conference Paper

Coresets for Vector Summarization with Applications to Network Graphs

• Dan Feldman
• Sedat Ozer
• Daniela Rus

We provide a deterministic data summarization algorithm that approximates the mean $\bar{p}=\frac{1}{n}\sum_{p\in P} p$ of a set $P$ of $n$ vectors in $\mathbb{R}^d$, by a weighted mean $\tilde{p}$ of a subset of $O(1/\epsilon)$ vectors, i. e. , independent of both $n$ and $d$. We prove that the squared Euclidean distance between $\bar{p}$ and $\tilde{p}$ is at most $\epsilon$ multiplied by the variance of $P$. We use this algorithm to maintain an approximated sum of vectors from an unbounded stream, using memory that is independent of $d$, and logarithmic in the $n$ vectors seen so far. Our main application is to extract and represent in a compact way friend groups and activity summaries of users from underlying data exchanges. For example, in the case of mobile networks, we can use GPS traces to identify meetings; in the case of social networks, we can use information exchange to identify friend groups. Our algorithm provably identifies the Heavy Hitter entries in a proximity (adjacency) matrix. The Heavy Hitters can be used to extract and represent in a compact way friend groups and activity summaries of users from underlying data exchanges. We evaluate the algorithm on several large data sets.

ICRA Conference 2017 Conference Paper

Correcting robot mistakes in real time using EEG signals

• Andres F. Salazar-Gomez
• Joseph DelPreto
• Stephanie Gil
• Frank H. Guenther
• Daniela Rus

Communication with a robot using brain activity from a human collaborator could provide a direct and fast feedback loop that is easy and natural for the human, thereby enabling a wide variety of intuitive interaction tasks. This paper explores the application of EEG-measured error-related potentials (ErrPs) to closed-loop robotic control. ErrP signals are particularly useful for robotics tasks because they are naturally occurring within the brain in response to an unexpected error. We decode ErrP signals from a human operator in real time to control a Rethink Robotics Baxter robot during a binary object selection task. We also show that utilizing a secondary interactive error-related potential signal generated during this closed-loop robot task can greatly improve classification performance, suggesting new ways in which robots can acquire human feedback. The design and implementation of the complete system is described, and results are presented for realtime closed-loop and open-loop experiments as well as offline analysis of both primary and secondary ErrP signals. These experiments are performed using general population subjects that have not been trained or screened. This work thereby demonstrates the potential for EEG-based feedback methods to facilitate seamless robotic control, and moves closer towards the goal of real-time intuitive interaction.

ICRA Conference 2017 Conference Paper

Distributed aggregation for modular robots in the pivoting cube model

• Sebastian Claici
• John W. Romanishin
• Jeffrey I. Lipton
• Stéphane Bonardi
• Kyle Gilpin
• Daniela Rus

We present a distributed control strategy for the aggregation of multiple modular robots into one connected structure optimized for use with 3D modular pivoting cube robots such as the 3D M-Blocks [1]. We use the intensity from a light source as input to a decentralized control algorithm that drives the robots together. We describe the algorithm, give provable guarantees on convergence, and discuss experiments carried out in simulation and with a hardware platform of ten 3D M-Blocks modules. In this paper we contribute provably correct algorithms for the aggregation of generic modular robots; we show how these algorithms can be applied on real hardware by evaluating them on the 3D M-Blocks platform.

ICRA Conference 2017 Conference Paper

Duckietown: An open, inexpensive and flexible platform for autonomy education and research

• Liam Paull
• Jacopo Tani
• Heejin Ahn
• Javier Alonso-Mora
• Luca Carlone
• Michal Cáp
• Yu Fan Chen
• Changhyun Choi

Duckietown is an open, inexpensive and flexible platform for autonomy education and research. The platform comprises small autonomous vehicles (“Duckiebots”) built from off-the-shelf components, and cities (“Duckietowns”) complete with roads, signage, traffic lights, obstacles, and citizens (duckies) in need of transportation. The Duckietown platform offers a wide range of functionalities at a low cost. Duckiebots sense the world with only one monocular camera and perform all processing onboard with a Raspberry Pi 2, yet are able to: follow lanes while avoiding obstacles, pedestrians (duckies) and other Duckiebots, localize within a global map, navigate a city, and coordinate with other Duckiebots to avoid collisions. Duckietown is a useful tool since educators and researchers can save money and time by not having to develop all of the necessary supporting infrastructure and capabilities. All materials are available as open source, and the hope is that others in the community will adopt the platform for education and research.

ICRA Conference 2017 Conference Paper

Enabling independent navigation for visually impaired people through a wearable vision-based feedback system

• Hsueh-Cheng Wang
• Robert K. Katzschmann
• Santani Teng
• Brandon Araki
• Laura Giarré
• Daniela Rus

This work introduces a wearable system to provide situational awareness for blind and visually impaired people. The system includes a camera, an embedded computer and a haptic device to provide feedback when an obstacle is detected. The system uses techniques from computer vision and motion planning to (1) identify walkable space; (2) plan step-by-step a safe motion trajectory in the space, and (3) recognize and locate certain types of objects, for example the location of an empty chair. These descriptions are communicated to the person wearing the device through vibrations. We present results from user studies with low- and high-level tasks, including walking through a maze without collisions, locating a chair, and walking through a crowded environment while avoiding people.

ICRA Conference 2017 Conference Paper

Functional co-optimization of articulated robots

• Andrew Spielberg
• Brandon Araki
• Cynthia R. Sung
• Russ Tedrake
• Daniela Rus

We present parametric trajectory optimization, a method for simultaneously computing physical parameters, actuation requirements, and robot motions for more efficient robot designs. In this scheme, robot dimensions, masses, and other physical parameters are solved for concurrently with traditional motion planning variables, including dynamically consistent robot states, actuation inputs, and contact forces. Our method requires minimal user domain knowledge, requiring only a coarse guess of the target robot configuration sequence and a parameterized robot topology as input. We demonstrate our results on four simulated robots, one of which we physically fabricated in order to demonstrate physical consistency. We demonstrate that by optimizing robot body parameters alongside robot trajectories, motion planning problems which would otherwise be infeasible can be made feasible, and actuation requirements can be significantly reduced.

IROS Conference 2017 Conference Paper

Hybrid control and learning with coresets for autonomous vehicles

• Guy Rosman
• Liam Paull
• Daniela Rus

Modern autonomous systems such as driverless vehicles need to safely operate in a wide range of conditions. A potential solution is to employ a hybrid systems approach, where safety is guaranteed in each individual mode within the system. This offsets complexity and responsibility from the individual controllers onto the complexity of determining discrete mode transitions. In this work we propose an efficient framework based on recursive neural networks and coreset data summarization to learn the transitions between an arbitrary number of controller modes that can have arbitrary complexity. Our approach allows us to efficiently gather annotation data from the large-scale datasets that are required to train such hybrid nonlinear systems to be safe under all operating conditions, favoring underexplored parts of the data. We demonstrate the construction of the embedding, and efficient detection of switching points for autonomous and non-autonomous car data. We further show how our approach enables efficient sampling of training data, to further improve either our embedding or the controllers.

ICRA Conference 2017 Conference Paper

Machine learning and coresets for automated real-time video segmentation of laparoscopic and robot-assisted surgery

• Mikhail Volkov 0002
• Daniel A. Hashimoto
• Guy Rosman
• Ozanan R. Meireles
• Daniela Rus

Context-aware segmentation of laparoscopic and robot assisted surgical video has been shown to improve performance and perioperative workflow efficiency, and can be used for education and time-critical consultation. Modern pressures on productivity preclude manual video analysis, and hospital policies and legacy infrastructure are often prohibitive of recording and storing large amounts of data. In this paper we present a system that automatically generates a video segmentation of laparoscopic and robot-assisted procedures according to their underlying surgical phases using minimal computational resources, and low amounts of training data. Our system uses an SVM and HMM in combination with an augmented feature space that captures the variability of these video streams without requiring analysis of the nonrigid and variable environment. By using the data reduction capabilities of online k-segment coreset algorithms we can efficiently produce results of approximately equal quality, in realtime. We evaluate our system in cross-validation experiments and propose a blueprint for piloting such a system in a real operating room environment with minimal risk factors.

ICRA Conference 2017 Conference Paper

Minimum-violation scLTL motion planning for mobility-on-demand

• Cristian Ioan Vasile
• Jana Tumova
• Sertac Karaman
• Calin Belta
• Daniela Rus

This work focuses on integrated routing and motion planning for an autonomous vehicle in a road network. We consider a problem in which customer demands need to be met within desired deadlines, and the rules of the road need to be satisfied. The vehicle might not, however, be able to satisfy these two goals at the same time. We propose a systematic way to compromise between delaying the satisfaction of the given demand and violating the road rules. We utilize scLTL formulas to specify desired behavior and develop a receding horizon approach including a periodically interacting routing algorithm and a RRT*-based motion planner. The proposed solution yields a provably minimum-violation trajectory. An illustrative case study is included.

ICRA Conference 2017 Conference Paper

Multi-robot path planning for a swarm of robots that can both fly and drive

• Brandon Araki
• John Strang
• Sarah Pohorecky
• Celine Qiu
• Tobias Naegeli
• Daniela Rus

The multi-robot path planning problem has been extensively studied for the cases of flying and driving vehicles. However, path planning for the case of vehicles that can both fly and drive has not yet been considered. Driving robots, while stable and energy efficient, are limited to mostly flat terrain. Quadcopters, on the other hand, are agile and highly mobile but have low energy efficiency and limited battery life. Combining a quadcopter with a driving mechanism presents a path planning challenge by enabling the selection of paths based off of both time and energy consumption. In this paper, we introduce a framework for multi-robot path planning for a swarm of flying-and-driving vehicles. By putting a lightweight driving platform on a quadcopter, we create a robust vehicle with an energy efficient driving mode and an agile flight mode. We extend two algorithms, priority planning with Safe Interval Path Planning and a multi-commodity network flow ILP, to accommodate multimodal locomotion, and we show that these algorithms can indeed plan collision-free paths for flying-and-driving vehicles on 3D graphs. Finally, we demonstrate that our system is able to plan paths and control the motions of 8 of our vehicles in a miniature town.

ICRA Conference 2017 Conference Paper

Parallel autonomy in automated vehicles: Safe motion generation with minimal intervention

• Wilko Schwarting
• Javier Alonso-Mora
• Liam Paull
• Sertac Karaman
• Daniela Rus

Current state-of-the-art vehicle safety systems, such as assistive braking or automatic lane following, are still only able to help in relatively simple driving situations. We introduce a Parallel Autonomy shared-control framework that produces safe trajectories based on human inputs even in much more complex driving scenarios, such as those commonly encountered in an urban setting. We minimize the deviation from the human inputs while ensuring safety via a set of collision avoidance constraints. We develop a receding horizon planner formulated as a Non-linear Model Predictive Control (NMPC) including analytic descriptions of road boundaries, and the configurations and future uncertainties of other traffic participants, and directly supplying them to the optimizer without linearization. The NMPC operates over both steering and acceleration simultaneously. Furthermore, the proposed receding horizon planner also applies to fully autonomous vehicles. We validate the proposed approach through simulations in a wide variety of complex driving scenarios such as left-turns across traffic, passing on busy streets, and under dynamic constraints in sharp turns on a race track.

ICRA Conference 2017 Conference Paper

Persistent surveillance of events with unknown, time-varying statistics

• Cenk Baykal
• Guy Rosman
• Sebastian Claici
• Daniela Rus

We consider the problem of monitoring stochastic, time-varying events occurring at discrete locations. Our problem formulation extends prior work in persistent surveillance by considering the objective of maximizing event detections in unknown, dynamic environments where the rates of events are time-inhomogeneous and may be subject to abrupt changes. We propose a novel monitoring algorithm that effectively strikes a balance between exploration and exploitation as well as a balance between remembering and discarding information to handle temporal variations in unknown environments. We present an analysis proving the long-run average optimality of the policies generated by our algorithm under the assumption that the total temporal variations are sub-linear. We present simulation results demonstrating the effectiveness of our algorithm in several monitoring scenarios inspired by real-world applications, and its robustness to both continuous-random and abrupt changes in the statistics of the observed processes.

ICRA Conference 2017 Conference Paper

Planning cuts for mobile robots with bladed tools

• Jeffrey I. Lipton
• Zachary Manchester
• Daniela Rus

Linear bladed cutting tools, such as jigsaws and reciprocating saws are vital manufacturing tools for humans. They enable people to cut structures that are much larger than themselves. Robots currently lack a generic path planner for linear bladed cutting tools. We developed a model for bladed tools based on Reeds-Shepp cars, and used the model to make a generic path planning algorithm for closed curves. We built an autonomous mobile robot which can implement the algorithm to cut arbitrarily large shapes in a 2D plane. We tested the robots performance and demonstrated the algorithm on several test cases.

IROS Conference 2017 Conference Paper

Predictive routing for autonomous mobility-on-demand systems with ride-sharing

• Javier Alonso-Mora
• Alex Wallar
• Daniela Rus

Ride-sharing, or carpooling, systems with autonomous vehicles will provide efficient and reliable urban mobility on demand. In this work we present a method for dynamic vehicle routing that leverages historical data to improve the performance of a network of self-driving taxis. In particular, we describe a constrained optimization method capable of assigning requests to autonomous vehicles in an informed way, to minimize the expected cost of serving both current and future travel requests. We allow several passengers with independent trips to share a vehicle and allow vehicles to pick additional passengers as they progress through their route. Based on historical data, we compute a probability distribution over future demand. Then, samples from the learned probability distribution are incorporated into a decoupled vehicle routing and passenger assignment method to take into account the predicted future demand. This method consists of three steps, namely pruning of feasible trips, assignment of trips to vehicles and rebalancing of idle vehicles. We show the benefits and trade-offs of this predictive approach in an experimental evaluation with over three million rides extracted from a dataset of taxi trips in New York City. Our method produces routes and assignments that, in expectation, reduce the travel and waiting times for passengers, with respect to a purely reactive approach. Besides the mobility on demand application, the method we present is general and could also be applied to other multi-task multi-vehicle assignment and routing problems.

ICRA Conference 2017 Conference Paper

Self-folded soft robotic structures with controllable joints

• Cynthia R. Sung
• Rhea Lin
• Shuhei Miyashita
• Sehyuk Yim
• Sangbae Kim
• Daniela Rus

This paper describes additive self-folding, an origami-inspired rapid fabrication approach for creating actuatable compliant structures. Recent work in 3-D printing and other rapid fabrication processes have mostly focused on rigid objects or objects that can achieve small deformations. In contrast, soft robots often require elastic materials and large amounts of movement. Additive self-folding is a process that involves cutting slices of a 3-D object in a long strip and then pleat folding them into a likeness of the original model. The zigzag pattern for folding enables large bending movements that can be actuated and controlled. Gaps between slices in the folded model can be designed to provide larger deformations or higher shape accuracy. We advance existing planar fabrication and self-folding techniques to automate the fabrication process, enabling highly compliant structures with complex 3-D geometries to be designed and fabricated within a few hours. We describe this process in this paper and provide algorithms for converting 3-D meshes into additive self-folding designs. The designs can be rapidly instrumented for global control using magnetic fields or tendon-driven for local bending. We also describe how the resulting structures can be modeled and their responses to tendon-driven control predicted. We test our design and fabrication methods on three models (a bunny, a tuna fish, and a starfish) and demonstrate the method's potential for actuation by actuating the tuna fish and starfish models using tendons and magnetic control.

IROS Conference 2016 Conference Paper

Cyclic hydraulic actuation for soft robotic devices

• Robert K. Katzschmann
• Austin de Maille
• David L. Dorhout
• Daniela Rus

Undulating structures are one of the most diverse and successful forms of locomotion in nature, both on ground and in water. This paper presents a comparative study for actuation by undulation in water. We focus on actuating a 1DOF systems with several mechanisms. A hydraulic pump attached to a soft body allows for water movement between two inner cavities, ultimately leading to a flexing actuation in a side-to-side manner. The effectiveness of six different, self-contained designs based on centrifugal pump, flexible impeller pump, external gear pump and rotating valves are compared. These hydraulic actuation systems combined with soft test bodies were then measured at a lower and higher oscillation frequency. The deflection characteristics of the soft body, the acoustic noise of the pump and the overall efficiency of the system are recorded. A brushless, centrifugal pump combined with a novel rotating valve performed at both test frequencies as the most efficient pump, producing sufficiently large cyclic body deflections along with the least acoustic noise among all pumps tested. An external gear pump design produced the largest body deflection, but consumes an order of magnitude more power and produced high noise levels. Further refinement remains on determining the suitable oscillation frequencies and inner cavity designs for optimal efficiency and movement.

NeurIPS Conference 2016 Conference Paper

Dimensionality Reduction of Massive Sparse Datasets Using Coresets

• Dan Feldman
• Mikhail Volkov
• Daniela Rus

In this paper we present a practical solution with performance guarantees to the problem of dimensionality reduction for very large scale sparse matrices. We show applications of our approach to computing the Principle Component Analysis (PCA) of any $n\times d$ matrix, using one pass over the stream of its rows. Our solution uses coresets: a scaled subset of the $n$ rows that approximates their sum of squared distances to \emph{every} $k$-dimensional \emph{affine} subspace. An open theoretical problem has been to compute such a coreset that is independent of both $n$ and $d$. An open practical problem has been to compute a non-trivial approximation to the PCA of very large but sparse databases such as the Wikipedia document-term matrix in a reasonable time. We answer both of these questions affirmatively. Our main technical result is a new framework for deterministic coreset constructions based on a reduction to the problem of counting items in a stream.

ICRA Conference 2016 Conference Paper

Distributed multi-robot formation control among obstacles: A geometric and optimization approach with consensus

• Javier Alonso-Mora
• Eduardo Montijano
• Mac Schwager
• Daniela Rus

This paper presents a distributed method for navigating a team of robots in formation in 2D and 3D environments with static and dynamic obstacles. The robots are assumed to have a reduced communication and visibility radius and share information with their neighbors. Via distributed consensus the robots compute (a) the convex hull of the robot positions and (b) the largest convex region within free space. The robots then compute, via sequential convex programming, the locally optimal parameters for the formation within this convex neighborhood of the robots. Reconfiguration is allowed, when required, by considering a set of target formations. The robots navigate towards the target collision-free formation with individual local planners that account for their dynamics. The approach is efficient and scalable with the number of robots and performs well in simulations with up to sixteen quadrotors.

IROS Conference 2016 Conference Paper

Fast Joint Compatibility Branch and Bound for feature cloud matching

• Xiaotong Shen
• Emilio Frazzoli
• Daniela Rus
• Marcelo H. Ang

In this work, we address the problem of robust data association for feature cloud matching. For matching two feature clouds observed at two different poses, we discover that the covariance matrix of the measurement prediction error can be written as the sum of a low rank matrix and a block diagonal matrix, if we assume that the features are observed independently at each pose. This special structure of the covariance matrix allows us to compute its inverse analytically and efficiently. Together with a good bookkeeping strategy, the complexity of the Joint Compatibility (JC) test is reduced to O(1). Contrary to the approximated JC test, ours is both exact and fast. Based on the efficient JC test algorithm and a branch and bound search procedure, we devise an algorithm, called Fast Joint Compatibility Branch and Bound (FastJCBB), to quickly obtain robust data association. The FastJCBB algorithm is essentially modified from the conventional Joint Compatibility Branch and Bound (JCBB) algorithm and both of these algorithms are able to produce exactly the same data association results. However, with the substantial improvement in the efficiency of JC tests, our FastJCBB algorithm is much faster than the conventional JCBB, especially when matching two large feature clouds. It is reported that our FastJCBB algorithm is more than 740 times faster than the conventional JCBB in carrying out one million JC tests when matching two clouds with about 100 features each. Since both FastJCBB and JCBB share the same branch and bound procedure in exploring the interpretation tree, the search complexity remains exponential. Our main contribution is the significant improvement in the efficiency of exploring each node of the interpretation tree.

ICRA Conference 2016 Conference Paper

Ingestible, controllable, and degradable origami robot for patching stomach wounds

• Shuhei Miyashita
• Steven Guitron
• Kazuhiro Yoshida
• Shuguang Li 0005
• Dana D. Damian
• Daniela Rus

Developing miniature robots that can carry out versatile clinical procedures inside the body under the remote instructions of medical professionals has been a long time challenge. In this paper, we present origami-based robots that can be ingested into the stomach, locomote to a desired location, patch a wound, remove a foreign body, deliver drugs, and biodegrade. We designed and fabricated composite material sheets for a biocompatible and biodegradable robot that can be encapsulated in ice for delivery through the esophagus, embed a drug layer that is passively released to a wounded area, and be remotely controlled to carry out underwater maneuvers specific to the tasks using magnetic fields. The performances of the robots are demonstrated in a simulated physical environment consisting of an esophagus and stomach with properties similar to the biological organs.

ICRA Conference 2016 Conference Paper

Printable hydraulics: A method for fabricating robots by 3D co-printing solids and liquids

• Robert MacCurdy
• Robert K. Katzschmann
• Youbin Kim
• Daniela Rus

This paper introduces a novel technique for fabricating functional robots using 3D printers. Simultaneously depositing photopolymers and a non-curing liquid allows complex, pre-filled fluidic channels to be fabricated. This new printing capability enables complex hydraulically actuated robots and robotic components to be automatically built, with no assembly required. The technique is showcased by printing linear bellows actuators, gear pumps, soft grippers and a hexapod robot, using a commercially-available 3D printer. We detail the steps required to modify the printer and describe the design constraints imposed by this new fabrication approach.

IROS Conference 2016 Conference Paper

Printable programmable viscoelastic materials for robots

• Robert MacCurdy
• Jeffrey I. Lipton
• Shuguang Li 0005
• Daniela Rus

Impact protection and vibration isolation are an important component of the mobile robot designer's toolkit; however, current damping materials are available only in bulk or molded form, requiring manual fabrication steps and restricting material property control. In this paper we demonstrate a new method for 3D printing viscoelastic materials with specified material properties. This method allows arbitrary net-shape material geometries to be rapidly fabricated and enables continuously varying material properties throughout the finished part. This new ability allows robot designers to tailor the properties of viscoelastic damping materials in order to reduce impact forces and isolate vibrations. We present a case study for using this material to create jumping robots with programmed levels of bouncing.

ICRA Conference 2016 Conference Paper

Probabilistic visual verification for robotic assembly manipulation

• Changhyun Choi
• Daniela Rus

In this paper we present a visual verification approach for robotic assembly manipulation which enables robots to verify their assembly state. Given shape models of objects and their expected placement configurations, our approach estimates the probability of the success of the assembled state using a depth sensor. The proposed approach takes into account uncertainties in object pose. Probability distributions of depth and surface normal depending on the uncertainties are estimated to classify the assembly state in a Bayesian formulation. The effectiveness of our approach is validated in comparative experiments with other approaches.

ICRA Conference 2016 Conference Paper

The flying monkey: A mesoscale robot that can run, fly, and grasp

• Yash Mulgaonkar
• Brandon Araki
• Je-Sung Koh
• Luis Guerrero-Bonilla
• Daniel M. Aukes
• Anurag Makineni
• Michael T. Tolley
• Daniela Rus

The agility and ease of control make a quadrotor aircraft an attractive platform for studying swarm behavior, modeling, and control. The energetics of sustained flight for small aircraft, however, limit typical applications to only a few minutes. Adding payloads - and the mechanisms used to manipulate them - reduces this flight time even further. In this paper we present the flying monkey, a novel robot platform having three main capabilities: walking, grasping, and flight. This new robotic platform merges one of the world's smallest quadrotor aircraft with a lightweight, single-degree-of-freedom walking mechanism and an SMA-actuated gripper to enable all three functions in a 30 g package. The main goal and key contribution of this paper is to design and prototype the flying monkey that has increased mission life and capabilities through the combination of the functionalities of legged and aerial robots.

ICRA Conference 2015 Conference Paper

3D M-Blocks: Self-reconfiguring robots capable of locomotion via pivoting in three dimensions

• John W. Romanishin
• Kyle Gilpin
• Sebastian Claici
• Daniela Rus

This paper presents the mechanical design of a modular robot called the 3D M-Block, a 50mm cubic module capable of both independent and lattice-based locomotion. The first M-Blocks described in [1] could pivot about one axis of rotation only. In contrast, the 3D M-blocks can exert on demand both forward and backward torques about three orthogonal axes, for a total of six directions. The 3D M-Blocks transform these torques into pivoting motions which allow the new 3D M-Blocks to move more freely than their predecessors. Individual modules can employ pivoting motions to independently roll across a wide variety of surfaces as well as to join and move relative to other M-Blocks as part of a larger collective structure. The 3D M-Block maintains the same form factor and magnetic bonding system as the one-dimensional M-Blocks [1], but a new fabrication process supports more efficient and precise production. The 3D M-blocks provide a robust and capable modular self-reconfigurable robotic platform able to support swarm robot applications through individual module capabilities and self-reconfiguring robot applications using connected lattices of modules.

ICRA Conference 2015 Conference Paper

A Distributed Robot Garden System

• Lindsay M. Sanneman
• Deborah Ajilo
• Joseph DelPreto
• Ankur M. Mehta
• Shuhei Miyashita
• Negin Abdolrahim Poorheravi
• Cami Ramirez
• Sehyuk Yim

Computational thinking is an important part of a modern education, and robotics provides a powerful tool for teaching programming logic in an interactive and engaging way. The robot garden presented in this paper is a distributed multi-robot system capable of running autonomously or under user control from a simple graphical interface. Over 100 origami flowers are actuated with LEDs and printed pouch motors, and are deployed in a modular array around additional swimming and crawling folded robots. The garden integrates state-of-the-art rapid design and fabrication technologies with distributed systems software techniques to create a scalable swarm in which robots can be controlled individually or as a group. The garden can be used to teach basic algorithmic concepts through its distributed algorithm demonstration capabilities and can teach programming concepts through its education-oriented user interface.

IROS Conference 2015 Conference Paper

A soft cube capable of controllable continuous jumping

• Shuguang Li 0005
• Robert K. Katzschmann
• Daniela Rus

Soft-bodied robots are designed to work in the physical world with a high compliance, while most of them lack in highly dynamic motion. In this paper, we present a soft-bodied jumping robot, which leverages its body's elasticity to achieve a highly dynamic passive bouncing motion after an active jumping motion. This robot has a cubic shape. It is covered by silicone foam, and each of its six faces has an opening to allow for jumping actuation. By winding up and releasing an elastic strip, the robot can jump in two directions at any orientation. We present the design, and fabrication process, and experimental results. By comparing this robot with a rigid version of the robot, we show that this soft-bodied robot can use a single jump to travel longer forward than its rigid counterpart.

ICRA Conference 2015 Conference Paper

An untethered miniature origami robot that self-folds, walks, swims, and degrades

• Shuhei Miyashita
• Steven Guitron
• Marvin Ludersdorfer
• Cynthia R. Sung
• Daniela Rus

A miniature robotic device that can fold-up on the spot, accomplish tasks, and disappear by degradation into the environment promises a range of medical applications but has so far been a challenge in engineering. This work presents a sheet that can self-fold into a functional 3D robot, actuate immediately for untethered walking and swimming, and subsequently dissolve in liquid. The developed sheet weighs 0. 31 g, spans 1. 7 cm square in size, features a cubic neodymium magnet, and can be thermally activated to self-fold. Since the robot has asymmetric body balance along the sagittal axis, the robot can walk at a speed of 3. 8 body-length/s being remotely controlled by an alternating external magnetic field. We further show that the robot is capable of conducting basic tasks and behaviors, including swimming, delivering/carrying blocks, climbing a slope, and digging. The developed models include an acetone-degradable version, which allows the entire robot's body to vanish in a liquid. We thus experimentally demonstrate the complete life cycle of our robot: self-folding, actuation, and degrading.

IROS Conference 2015 Conference Paper

Anytime planning of optimal schedules for a mobile sensing robot

• Jingjin Yu
• Javed A. Aslam
• Sertac Karaman
• Daniela Rus

We study the problem in which a mobile sensing robot is tasked to travel among and gather intelligence at a set of spatially distributed points-of-interest (POIs). The quality of the information collected at a POI is characterized by some sensory (reward) function of time. With limited fuel, the robot must balance between spending time traveling to more POIs and performing time-consuming sensing activities at POIs to maximize the overall reward. In a dual formulation, the robot is required to acquire a minimum amount of reward with the least amount of time. We propose an anytime planning algorithm for solving these two NP-hard problems to arbitrary precision for arbitrary reward functions. The algorithm is effective on large instances with tens to hundreds of POIs, as demonstrated with an extensive set of computational experiments. Besides mobile sensor scheduling, our algorithm also applies to automation scenarios such as intelligent and optimal itinerary planning.

IROS Conference 2015 Conference Paper

Autonomous golf cars for public trial of mobility-on-demand service

• Scott Drew Pendleton
• Tawit Uthaicharoenpong
• Zhuang Jie Chong
• James Guo Ming Fu
• Baoxing Qin
• Wei Liu 0024
• Xiaotong Shen
• Zhiyong Weng

We detail the design of autonomous golf cars which were used in public trials in Singapore's Chinese and Japanese Gardens, for the purpose of raising public awareness and gaining user acceptance of autonomous vehicles. The golf cars were designed to be robust, reliable, and safe, while operating under prolonged durations. Considerations that went in to the overall system design included the fact that any member of the public had to not only be able to easily use the system, but to also not have the option to use the system in an unintended manner. This paper details the hardware and software components of the golf cars with these considerations, and also how the booking system and mission planner facilitated users to book for a golf car from any of ten stations within the gardens. We show that the vehicles performed robustly throughout the prolonged operations with a small localization variance, and that users were very receptive from the user survey results.

ICRA Conference 2015 Conference Paper

Coresets for visual summarization with applications to loop closure

• Mikhail Volkov 0002
• Guy Rosman
• Dan Feldman
• John W. Fisher III
• Daniela Rus

In continuously operating robotic systems, efficient representation of the previously seen camera feed is crucial. Using a highly efficient compression coreset method, we formulate a new method for hierarchical retrieval of frames from large video streams collected online by a moving robot. We demonstrate how to utilize the resulting structure for efficient loop-closure by a novel sampling approach that is adaptive to the structure of the video. The same structure also allows us to create a highly-effective search tool for large-scale videos, which we demonstrate in this paper. We show the efficiency of proposed approaches for retrieval and loop closure on standard datasets, and on a large-scale video from a mobile camera.

ICRA Conference 2015 Conference Paper

Dynamics and trajectory optimization for a soft spatial fluidic elastomer manipulator

• Andrew D. Marchese
• Russ Tedrake
• Daniela Rus

The goal of this work is to develop a soft robotic manipulation system that is capable of autonomous, dynamic, and safe interactions with humans and its environment. First, we develop a dynamic model for a multi-body fluidic elastomer manipulator that is composed entirely from soft rubber and subject to the self-loading effects of gravity. Then, we present a strategy for independently identifying all unknown components of the system: the soft manipulator, its distributed fluidic elastomer actuators, as well as drive cylinders that supply fluid energy. Next, using this model and trajectory optimization techniques we find locally optimal open-loop policies that allow the system to perform dynamic maneuvers we call grabs. In 37 experimental trials with a physical prototype, we successfully perform a grab 92% of the time. By studying such an extreme example of a soft robot, we can begin to solve hard problems inhibiting the mainstream use of soft machines.

IROS Conference 2015 Conference Paper

Haptic identification of objects using a modular soft robotic gripper

• Bianca S. Homberg
• Robert K. Katzschmann
• Mehmet Remzi Dogar
• Daniela Rus

This work presents a soft hand capable of robustly grasping and identifying objects based on internal state measurements. A highly compliant hand allows for intrinsic robustness to grasping uncertainty, but the specific configuration of the hand and object is not known, leaving undetermined if a grasp was successful in picking up the right object. A soft finger was adapted and combined to form a three finger gripper that can easily be attached to existing robots, for example, to the wrist of the Baxter robot. Resistive bend sensors were added within each finger to provide a configuration estimate sufficient for distinguishing between a set of objects. With one data point from each finger, the object grasped by the gripper can be identified. A clustering algorithm to find the correspondence for each grasped object is presented for both enveloping grasps and pinch grasps. This hand is a first step towards robust proprioceptive soft grasping.

ICRA Conference 2015 Conference Paper

Local motion planning for collaborative multi-robot manipulation of deformable objects

• Javier Alonso-Mora
• Ross A. Knepper
• Roland Siegwart
• Daniela Rus

This paper presents a formalism that exploits deformability during manipulation of soft objects by robot teams. A hybrid centralized/distributed approach restricts centralized planning to high-level global guidance of the object for consensus. Low-level control is thus delegated to the individual manipulator robots, which retain manipulation and collision avoidance guarantees by passing forces to one another through the object. A distributed receding horizon planner provides local control, formulated as a convex optimization problem in velocity space and incorporating constraints for both collision avoidance and shape maintenance. We demonstrate teams of mobile manipulators autonomously carrying various deformable objects.

ICRA Conference 2015 Conference Paper

Multi-robot grasp planning for sequential assembly operations

• Mehmet Remzi Dogar
• Andrew Spielberg
• Stuart Baker
• Daniela Rus

This paper addresses the problem of finding robot configurations to grasp assembly parts during a sequence of collaborative assembly operations. We formulate the search for such configurations as a constraint satisfaction problem (CSP). Collision constraints in an operation and transfer constraints between operations determine the sets of feasible robot configurations. We show that solving the connected constraint graph with off-the-shelf CSP algorithms can quickly become infeasible even for a few sequential assembly operations. We present an algorithm which, through the assumption of feasible regrasps, divides the CSP into independent smaller problems that can be solved exponentially faster. The algorithm then uses local search techniques to improve this solution by removing a gradually increasing number of regrasps from the plan. The algorithm enables the user to stop the planner anytime and use the current best plan if the cost of removing regrasps from the plan exceeds the cost of executing those regrasps. We present simulation experiments to compare our algorithm's performance to a naive algorithm which directly solves the connected constraint graph. We also present a real robot system which uses the output of our planner to grasp and bring parts together in assembly configurations.

IROS Conference 2015 Conference Paper

Multi-robot navigation in formation via sequential convex programming

• Javier Alonso-Mora
• Stuart Baker
• Daniela Rus

This paper presents a method for navigating a team of robots in formation in 2D and 3D environments with static and dynamic obstacles. The method is local and computes the optimal parameters for the formation within a neighborhood of the robots, allowing for reconfigurations, when required, by considering a set of target formations. The method consists of first computing the largest collision-free convex polytope in a neighborhood of the robots, followed by a constrained optimization via sequential convex programming where the optimal parameters for the formation are obtained. The robots navigate towards the target collision-free formation with individual local planners that account for their dynamics. The approach is efficient and scalable with the number of robots and performed well in simulations with a large team of quadrators and in experiments with two mobile manipulators carrying a rigid object.

ICRA Conference 2015 Conference Paper

Reconfiguration planning for pivoting cube modular robots

• Cynthia R. Sung
• James M. Bern
• John W. Romanishin
• Daniela Rus

In this paper, we present algorithms for self-reconfiguration of modular robots that move by pivoting. The modules are cubes that can pivot about their edges along the x̂, ŷ, or ẑ axes to move on a 3-dimensional substrate. This is a different model from prior work, which usually considers modules that slide along their faces. We analyze the pivoting cube model and give sufficient conditions for reconfiguration to be feasible. In particular, we show that if an initial configuration does not contain any of three subconfigurations, which we call rules, then it can reconfigure into a line. We provide provably correct algorithms for reconfiguration for both 2-D and 3-D systems, and we verify our algorithms via simulation on randomly generated 2-D and 3-D configurations.

IROS Conference 2015 Conference Paper

Towards autonomous navigation of unsignalized intersections under uncertainty of human driver intent

• Volkan Sezer
• Tirthankar Bandyopadhyay
• Daniela Rus
• Emilio Frazzoli
• David Hsu

In a mixed environment of autonomous driverless vehicles and human driven vehicles operating on the same road, identifying intentions of human drivers and interacting with them in a compliant and responsible manner becomes a challenging problem for the driverless vehicles. In this paper, the problem of vehicle interaction at an intersection merging scenario is formulated as an Intention-Aware motion planning problem using the tools from Mixed Observability Markov Decision Process (MOMDP). We utilize the tools from recent intention aware planning framework to demonstrate a merging behavior in the presence of human drivers by trying to infer and act according to the intentions of the human drivers. A driver behavior model for T-junction intersections is developed in order to calculate the probabilistic state transition functions of the MOMDP model. With proposed solution, it is demonstrated that using intention aware planning improves performance in comparison to present time to merge approach by lowering accident probability and intersection navigation duration. The proposed method is tested on a real autonomous vehicle (AV) in the presence of human driven vehicles to validate our approach.

ICRA Conference 2014 Conference Paper

An end-to-end approach to making self-folded 3D surface shapes by uniform heating

• Byoungkwon An
• Shuhei Miyashita
• Michael T. Tolley
• Daniel M. Aukes
• Laura Meeker
• Erik D. Demaine
• Martin L. Demaine
• Robert J. Wood

This paper presents an end-to-end approach for creating 3D shapes by self-folding planar sheets activated by uniform heating. These shapes can be used as the mechanical bodies of robots. The input to this process is a 3D geometry (e. g. an OBJ file). The output is a physical object with the specified geometry. We describe an algorithm pipeline that (1) identifies the overall geometry of the input, (2) computes a crease pattern that causes the sheet to self-fold into the desired 3D geometry when activated by uniform heating, (3) automatically generates the design of a 2D sheet with the desired pattern and (4) automatically generates the design files required to fabricate the 2D structure. We demonstrate these algorithms by applying them to complex 3D shapes. We demonstrate the fabrication of a self-folding object with over 50 faces from automatically generated design files.

ICRA Conference 2014 Conference Paper

An end-to-end system for designing mechanical structures for print-and-fold robots

• Ankur M. Mehta
• Daniela Rus

This work presents a script-based development environment aimed at allowing users to easily design and create mechanical bodies for folded plastic robots. The origami-inspired fabrication process is inexpensive and widely accessible, and the tools developed in this work allow for open source design sharing and modular reuse. Designs are generated by recursively combining mechanical components — from primitive building blocks, through mechanisms and assemblies, to full robots — in a flexible yet well-defined manner. This process was used to design robotic elements of increasing complexity up to a multi-degree-of-freedom compliant manipulator arm, demonstrating the power of this system. The developed system is extensible, opening avenues for further research ultimately leading to the development of a complete robot compiler.

IROS Conference 2014 Conference Paper

Cogeneration of mechanical, electrical, and software designs for printable robots from structural specifications

• Ankur M. Mehta
• Joseph DelPreto
• Benjamin Shaya
• Daniela Rus

Designing and fabricating new robotic systems is typically limited to experts, requiring engineering background, expensive tools, and considerable time. In contrast, to facilitate everyday users developing custom robots for personal use, this work presents a new system to easily create printable foldable robots from high-level structural specifications. A user merely needs to select electromechanical components from a library of basic building blocks and pre-designed mechanisms, then connect them to define custom robot assemblies. The system then generates complete mechanical drawings suitable for fabrication, instructions for the assembly of electronics, and software to control and drive the final robot. Several robots designed in this manner demonstrate the ability and versatility of this process.

ICRA Conference 2014 Conference Paper

Controlling a team of robots with a single input

• Nora Ayanian
• Andrew Spielberg
• Matthew Arbesfeld
• Jason Strauss
• Daniela Rus

We present a novel end-to-end solution for distributed multirobot coordination that translates multitouch gestures into low-level control inputs for teams of robots. Highlighting the need for a holistic solution to the problem of scalable human control of multirobot teams, we present a novel control algorithm with provable guarantees on the robots' motion that lends itself well to input from modern tablet and smartphone interfaces. Concretely, we develop an iOS application in which the user is presented with a team of robots and a bounding box (prism). The user carefully translates and scales the prism in a virtual environment; these prism coordinates are wirelessly transferred to our server and then received as input to distributed onboard robot controllers. We develop a novel distributed multirobot control policy which provides guarantees on convergence to a goal with distance bounded linearly in the number of robots, and avoids interrobot collisions. This approach allows the human user to solve the cognitive tasks such as path planning, while leaving precise motion to the robots. Our system was tested in simulation and experiments, demonstrating its utility and effectiveness.

NeurIPS Conference 2014 Conference Paper

Coresets for k-Segmentation of Streaming Data

• Guy Rosman
• Mikhail Volkov
• Dan Feldman
• John Fisher III
• Daniela Rus

Life-logging video streams, financial time series, and Twitter tweets are a few examples of high-dimensional signals over practically unbounded time. We consider the problem of computing optimal segmentation of such signals by k-piecewise linear function, using only one pass over the data by maintaining a coreset for the signal. The coreset enables fast further analysis such as automatic summarization and analysis of such signals. A coreset (core-set) is a compact representation of the data seen so far, which approximates the data well for a specific task -- in our case, segmentation of the stream. We show that, perhaps surprisingly, the segmentation problem admits coresets of cardinality only linear in the number of segments k, independently of both the dimension d of the signal, and its number n of points. More precisely, we construct a representation of size O(klog n /eps^2) that provides a (1+eps)-approximation for the sum of squared distances to any given k-piecewise linear function. Moreover, such coresets can be constructed in a parallel streaming approach. Our results rely on a novel eduction of statistical estimations to problems in computational geometry. We empirically evaluate our algorithms on very large synthetic and real data sets from GPS, video and financial domains, using 255 machines in Amazon cloud.

IROS Conference 2014 Conference Paper

Correlated Orienteering Problem and its application to informative path planning for persistent monitoring tasks

• Jingjin Yu
• Mac Schwager
• Daniela Rus

We propose a novel non-linear extension to the Orienteering Problem (OP), called the Correlated Orienteering Problem (COP). We use COP to plan informative tours (cyclic paths) for persistent monitoring of an environment with spatial correlations, where the tours are constrained to a fixed length or time budget. The main feature of COP is a quadratic utility function that captures spatial correlations among points of interest that are close to each other. COP may be solved using mixed integer quadratic programming (MIQP) that can plan multiple disjoint tours that maximize the quadratic utility function. We perform extensive characterization of our method to verify its correctness, as well as its applicability to the estimation of a realistic, time-varying, and spatially correlated scalar field.

ICRA Conference 2014 Conference Paper

Design and control of a soft and continuously deformable 2D robotic manipulation system

• Andrew D. Marchese
• Konrad Komorowski
• Cagdas D. Onal
• Daniela Rus

In this paper we describe the design, fabrication, control, and experimental validation of a soft and highly compliant 2D manipulator. The arm consists of several body segments actuated using bi-directional fluidic elastomer actuators and is fabricated using a novel composite molding process. We use a cascaded PI and PID computation and novel fluidic drive cylinders to provide closed-loop control of curvature for each soft and highly compliant body segment. Furthermore, we develop algorithms to compute the arm's forward and inverse kinematics in a manner consistent with piece-wise constant curvature continuum manipulators. These computation and control systems enable this highly compliant robot to autonomously follow trajectories. Experimental results with a robot consisting of six segments show that controlled movement of a soft and highly compliant manipulator is feasible.

ICRA Conference 2014 Conference Paper

Learning pedestrian activities for semantic mapping

• Baoxing Qin
• Zhuang Jie Chong
• Tirthankar Bandyopadhyay
• Marcelo H. Ang
• Emilio Frazzoli
• Daniela Rus

This paper proposes a semantic mapping method based on pedestrian activity in the urban road environment. Pedestrian activity patterns are learned from pedestrian tracks collected by a mobile platform. With the learned knowledge of pedestrian activity, semantic mapping is performed using Bayesian classification techniques. The proposed method is tested in real experiments, and shows promising results in recognizing four activity-related semantic properties of the urban road environment: pedestrian path, entrance/exit, pedestrian crossing and sidewalk.

ICRA Conference 2014 Conference Paper

Persistent monitoring of events with stochastic arrivals at multiple stations

• Jingjin Yu
• Sertac Karaman
• Daniela Rus

This paper is concerned with a novel mobile sensor scheduling problem, involving a single robot tasked with monitoring several events of interest that occur at different locations. Of particular interest is the monitoring of events that can not be easily forecast. Prominent examples range from natural phenomena (e. g. , monitoring abnormal seismic activity around a volcano using a ground robot) to urban activities (e. g. , monitoring early formations of traffic congestion in the Boston area using an aerial robot). Motivated by these examples, this paper focuses on problems where the precise occurrence time of the events is not known a priori, but some statistics for their inter-arrival times are available from past observations. The robot's task is to monitor the events to optimize the following two objectives: (i) maximize the number of events observed and (ii) minimize the delay between two consecutive observations of events occurring at the same location. Provided with only one robot, it is crucial to optimize these objectives in a balanced way, so that they are optimized at each station simultaneously. Our main theoretical result is that this complex mobile sensor scheduling problem can be reduced to a quasi-convex program, which can be solved in polynomial time. In other words, a globally optimal solution can be computed in time that is polynomial in the number of locations. We also provide computational experiments that validate our theoretical results.

ICRA Conference 2014 Conference Paper

Pouch Motors: Printable/inflatable soft actuators for robotics

• Ryuma Niiyama
• Daniela Rus
• Sangbae Kim

We propose a new family of fluidic soft actuators called Pouch Motors. The pouch motors are developed to create printable actuators for enhancing mass-fabrication of robots from sheet materials using easily accessible tools. The pouch motor consists of one or more gas-tight bladders (called pouches) fabricated by heat bonding. We developed two types of actuators from inflatable pouches: the linear pouch motor and the rotational pouch motor. Our theoretical analysis predicts the static force-length and moment-angle relationships of these actuators under pressure control. We compare the theoretical bounds with actual results achieved using several fabricated devices. We developed a fabrication process of pouch motors using a heat stamping technique that allows mass-manufacturing. We also demonstrate three robot bodies with embedded pouch motors: a parallel gripper, a robotic arm with antagonistic actuation, and legged walking robot with a self-contained miniature pneumatic system.

ICRA Conference 2014 Conference Paper

Self-folding printable elastic electric devices: Resistor, capacitor, and inductor

• Shuhei Miyashita
• Laura Meeker
• Maurice Göldi
• Yoshihiro Kawahara
• Daniela Rus

This paper presents a methodology and validation of print-and-self-fold electric devices. For printing functional structures for robotic use, we realize electric circuitry based on metallic polyester film (MPF). By exploiting the unique material properties of MPF, we developed fundamental electric devices, namely a resistor, capacitor, and inductor. The developed polyvinyl chloride laminated MPF sheet shows reliable self-folding processes under a heat application, and it configures 3D electric devices. Due to the pre-resolved kinematic design, these devices feature elasticity, making them suitable as sensors and actuators in soft circuits. Here we testify to a self-assembled variable resistor and capacitive strain sensor. An actuation mechanism consisting of a folded contractible coil is also considered and shown. Finally, an RLC circuit obtained from the integration of all the developed devices is demonstrated, in which the coil based actuator is controlled by reading a variable capacitive strain sensor.

ICRA Conference 2014 Conference Paper

Visual precis generation using coresets

• Rohan Paul
• Dan Feldman
• Daniela Rus
• Paul Newman 0001

Given an image stream, our on-line algorithm will select the semantically-important images that summarize the visual experience of a mobile robot. Our approach consists of data pre-clustering using coresets followed by a graph based incremental clustering procedure using a topic based image representation. A coreset for an image stream is a set of representative images that semantically compresses the data corpus, in the sense that every frame has a similar representative image in the coreset. We prove that our algorithm efficiently computes the smallest possible coreset under natural well-defined similarity metric and up to provably small approximation factor. The output visual summary is computed via a hierarchical tree of coresets for different parts of the image stream. This allows multi-resolution summarization (or a video summary of specified duration) in the batch setting and a memory-efficient incremental summary for the streaming case.

IROS Conference 2014 Conference Paper

Whole arm planning for a soft and highly compliant 2D robotic manipulator

• Andrew D. Marchese
• Robert K. Katzschmann
• Daniela Rus

Soft continuum manipulators have the advantage of being more compliant and having more degrees of freedom than rigid redundant manipulators. This attribute should allow soft manipulators to autonomously execute highly dexterous tasks. However, current approaches to motion planning, inverse kinematics, and even design limit the capacity of soft manipulators to take full advantage of their inherent compliance. We provide a computational approach to whole arm planning for a soft planar manipulator that advances the arm's end effector pose in task space while simultaneously considering the arm's entire envelope in proximity to a confined environment. The algorithm solves a series of constrained optimization problems to determine locally optimal inverse kinematics. Due to inherent limitations in modeling the kinematics of a highly compliant soft robot and the local optimality of the planner's solutions, we also rely on the increased softness of our newly designed manipulator to accomplish the whole arm task, namely the arm's ability to harmlessly collide with the environment. We detail the design and fabrication of the new modular manipulator as well as the planner's central algorithm. We experimentally validate our approach by showing that the robotic system is capable of autonomously advancing the soft arm through a pipe-like environment in order to reach distinct goal states.

IROS Conference 2013 Conference Paper

A lightweight modular 12-DOF print-and-fold hexapod

• Daniel E. Soltero
• Brian J. Julian
• Cagdas D. Onal
• Daniela Rus

This paper presents the design, fabrication and operation of a hexapod fabricated using a combination of printing and folding flat sheets of polyester. The polyester sheets are cut and engraved with crease patterns, which are then manually folded to create 3D functional modules, inspired by the Japanese art of Origami. These modules, when connected, form a hexapod with two degrees of freedom per leg. All custom mechanical parts are manufactured in a planar fashion using a laser cutter. We created this print-and-fold hexapod as a miniature version of a commercially available platform, to which we compare several metrics, such as weight, walking speed, and cost of transportation. Our print-and-fold hexapod has a mass of 195 g, can walk at speeds of up to 38. 1 cm/sec (two body lengths per second), and can be manufactured and assembled from scratch by a single person in approximately seven hours. Experimental results of gait control and trajectory tracking are provided.

IROS Conference 2013 Conference Paper

Decentralized robotic assembly with physical ordering and timing constraints

• T. Ryan Schoen
• Daniela Rus

Our prior work presented a system for decentralized robotic assembly[1]: given a team of robots, a cache of components, and a desired structure specified as a blueprint, the algorithm computes a sequence of part deliveries and assembly steps to achieve the desired structure, while considering physical dependencies and reachability constraints for the goal structure. In this paper we introduce a new algorithm that extends our prior result to incorporate the duration of each assembly operation. We also extend the algorithm to be adaptive to the availability of parts. When a part is not available, the assembly sequence is recomputed. The algorithms are provably convergent and their execution does not depend on the size of the robot team. We implement the algorithms using a team of four youBot robots that can (1) locate and identify parts; (2) use communication to achieve coordinated hand-off of parts; and (3) create complex log-cabin style structures.

ICRA Conference 2013 Conference Paper

IkeaBot: An autonomous multi-robot coordinated furniture assembly system

• Ross A. Knepper
• Todd Layton
• John W. Romanishin
• Daniela Rus

We present an automated assembly system that directs the actions of a team of heterogeneous robots in the completion of an assembly task. From an initial user-supplied geometric specification, the system applies reasoning about the geometry of individual parts in order to deduce how they fit together. The task is then automatically transformed to a symbolic description of the assembly—a sort of blueprint. A symbolic planner generates an assembly sequence that can be executed by a team of collaborating robots. Each robot fulfills one of two roles: parts delivery or parts assembly. The latter are equipped with specialized tools to aid in the assembly process. Additionally, the robots engage in coordinated co-manipulation of large, heavy assemblies. We provide details of an example furniture kit assembled by the system.

ICRA Conference 2013 Conference Paper

Improving the performance of multi-robot systems by task switching

• Cynthia R. Sung
• Nora Ayanian
• Daniela Rus

We consider the problem of task assignment for a multi-robot system where each robot must attend to one or more queues of tasks. We assume that individual robots have no knowledge of tasks in the environment that are not in their queue. Robots in communication with each other may share information about active tasks and exchange queues to achieve lower cost for the system. We show that allowing this kind of task switching causes tasks to be completed more efficiently. In addition, we present conditions under which queues can be guaranteed to make progress, and we support these claims with simulation and experimental results. This work has potential applications in manufacturing, environmental exploration, and pickup-delivery tasks.

ICRA Conference 2013 Conference Paper

Incremental synthesis of control policies for heterogeneous multi-agent systems with linear temporal logic specifications

• Tichakorn Wongpiromsarn
• Alphan Ulusoy
• Calin Belta
• Emilio Frazzoli
• Daniela Rus

We consider automatic synthesis of control policies for non-independent, heterogeneous multi-agent systems with the objective of maximizing the probability of satisfying a given specification. The specification is expressed as a formula in linear temporal logic. The agents are modeled by Markov decision processes with a common set of actions. These actions, however, may or may not affect the behaviors of all the agents. To alleviate the well-known state explosion problem, an incremental approach is proposed where only a small subset of agents is incorporated in the synthesis procedure initially and more agents are successively added until the limitations on computational resources are reached. The proposed algorithm runs in an anytime fashion, where the probability of satisfying the specification increases as the algorithm progresses.

ICRA Conference 2013 Conference Paper

K-robots clustering of moving sensors using coresets

• Dan Feldman
• Stephanie Gil
• Ross A. Knepper
• Brian J. Julian
• Daniela Rus

We present an approach to position k servers (e. g. mobile robots) to provide a service to n independently moving clients; for example, in mobile ad-hoc networking applications where inter-agent distances need to be minimized, connectivity constraints exist between servers, and no a priori knowledge of the clients' motion can be assumed. Our primary contribution is an algorithm to compute and maintain a small representative set, called a kinematic coreset, of the n moving clients. We prove that, in any given moment, the maximum distance between the clients and any set of k servers is approximated by the coreset up to a factor of (1 ± ε), where ε > 0 is an arbitrarily small constant. We prove that both the size of our coreset and its update time is polynomial in k log(n)/ε. Although our optimization problem is NP-hard (i. e. , takes time exponential in the number of servers to solve), solving it on the small coreset instead of the original clients results in a tractable controller. The approach is validated in a small scale hardware experiment using robot servers and human clients, and in a large scale numerical simulation using thousands of clients.

IROS Conference 2013 Conference Paper

M-blocks: Momentum-driven, magnetic modular robots

• John W. Romanishin
• Kyle Gilpin
• Daniela Rus

In this paper, we describe a novel self-assembling, self-reconfiguring cubic robot that uses pivoting motions to change its intended geometry. Each individual module can pivot to move linearly on a substrate of stationary modules. The modules can use the same operation to perform convex and concave transitions to change planes. Each module can also move independently to traverse planar unstructured environments. The modules achieve these movements by quickly transferring angular momentum accumulated in a self-contained flywheel to the body of the robot. The system provides a simplified realization of the modular actions required by the sliding cube model using pivoting. We describe the principles, the unit-module hardware, and extensive experiments with a system of eight modules.

IROS Conference 2013 Conference Paper

Mapping with synthetic 2D LIDAR in 3D urban environment

• Zhuang Jie Chong
• Baoxing Qin
• Tirthankar Bandyopadhyay
• Marcelo H. Ang
• Emilio Frazzoli
• Daniela Rus

In this paper, we report a fully automated detailed mapping of a challenging urban environment using single LIDAR. To improve scan matching, extended correlative scan matcher is proposed. Also, a Monte Carlo loop closure detection is implemented to perform place recognition efficiently. Automatic recovery of the pose graph map in the presence of false place recognition is realized through a heuristic based loop closure rejection. This mapping framework is evaluated through experiments on the real world dataset obtained from NUS campus environment.

IROS Conference 2013 Conference Paper

On mutual information-based control of range sensing robots for mapping applications

• Brian J. Julian
• Sertac Karaman
• Daniela Rus

In this paper we examine the correlation between the information content and the spatial realization of range measurements taken by a mapping robot. To do so, we consider the task of constructing an occupancy grid map with a binary Bayesian filter. Using a narrow beam-based sensor model (versus an additive white Gaussian noise model), we prove that any controller tasked to maximize a mutual information reward function is eventually attracted to unexplored space. This intuitive behavior is derived solely from the geometric dependencies of the occupancy grid mapping algorithm and the monotonie properties of mutual information. Since it is a function of both the robot's position and the uncertainty of the surrounding cells, mutual information encodes geometric relationships that are fundamental to robot control, thus yielding geometrically relevant reward surfaces on which the robot can navigate. Lastly, we present the results of two experiments employing an omnidirectional ground robot equipped with a laser rangefinder.

ICRA Conference 2013 Conference Paper

On the completeness of ensembles of motion planners for decentralized planning

• Ross A. Knepper
• Daniela Rus

We provide a set of sufficient conditions to establish the completeness of an ensemble of motion planners-that is, a set of loosely-coupled motion planners that produce a unified result. The planners are assumed to divide the total planning problem across some parameter space(s), such as task space, state space, action space, or time. Robotic applications have employed ensembles of planners for decades, although the concept has not been formally unified or analyzed until now. We focus on applications in multi-robot navigation and collision avoidance. We show that individual resolutionor probabilistically-complete planners that meet certain communication criteria constitute a (respectively, resolution- or probabilistically-) complete ensemble of planners. This ensemble of planners, in turn, guarantees that the robots are free of deadlock, livelock, and starvation.

IROS Conference 2013 Conference Paper

Pose and paste - An intuitive interface for remote navigation of a multi-robot system

• Michael Lichtenstern
• Michael Angermann
• Martin Frassl
• Gunther Berthold
• Brian J. Julian
• Daniela Rus

We present Pose and Paste (P&P) — an intuitive interface designed to facilitate interaction between a single user and a number of robots equipped with cameras. With this interface, a user wearing a head-mounted display is able to cycle through the real-time video streams originating from the robots' cameras. The user is also able to select a robot and remotely position it by simply walking or turning his/her head, i. e. , control the robot's motion in a master/slave-type fashion. We report the results of an initial hardware experiment where a user located in the USA is tasked to position two quadrotor robots within a motion capture laboratory located in Germany. These results suggest that P&P is a feasible approach to remotely inspect disaster affected sites. Lastly, we conduct a user study to compare P&P with a baseline interface composed of a traditional computer monitor and a video game controller. The quantitative results and qualitative discussions resulting from this user study highlight how such multi-robot interfaces can be further improved.

ICRA Conference 2013 Conference Paper

Robot self-assembly by folding: A printed inchworm robot

• Samuel M. Felton
• Michael T. Tolley
• Cagdas D. Onal
• Daniela Rus
• Robert J. Wood

Printing and folding are fast and inexpensive methods for prototyping complex machines. Self-assembly of the folding step would expand the possibilities of this method to include applications where external manipulation is costly, such as micro-assembly, mass production, and space applications. This paper presents a method for self-folding of printed robots from two-dimensional materials based on shape memory polymers actuated by joule heating using embedded circuits. This method was shown to be capable of sequential folding, angle-controlled folds, slot-and-tab assembly, and mountain and valley folds. An inchworm robot was designed to demonstrate the merits of this technique. Upon the application of sufficient current, the robot was able to fold into its functional form with fold angle deviations within six degrees. This printed robot demonstrated locomotion at a speed of two millimeters per second.

ICRA Conference 2013 Conference Paper

Robust real-time underwater digital video streaming using optical communication

• Marek Doniec
• Anqi Xu 0003
• Daniela Rus

We present a real-time video delivery solution based on free-space optical communication for underwater applications. This solution comprises of AquaOptical II, a high-bandwidth wireless optical communication device, and a two-layer digital encoding scheme designed for error-resistant communication of high resolution images. Our system can transmit digital video reliably through a unidirectional underwater channel, with minimal infrastructural overhead. We present empirical evaluation of this system's performance for various system configurations, and demonstrate that it can deliver high quality video at up to 15 Hz, with near-negligible communication latencies of 100 ms. We further characterize the corresponding end-to-end latencies, i. e. from time of image acquisition until time of display, and reveal optimized results of under 200 ms, which facilitates a wide range of applications such as underwater robot tele-operation and interactive remote seabed monitoring.

IROS Conference 2013 Conference Paper

Self-folding shape memory laminates for automated fabrication

• Michael T. Tolley
• Samuel M. Felton
• Shuhei Miyashita
• Lily Xu
• ByungHyun Shin
• Monica Zhou
• Daniela Rus
• Robert J. Wood

Nature regularly uses self-folding as an efficient approach to automated fabrication. In engineered systems, however, the use of self-folding has been primarily restricted to the assembly of small structures using exotic materials and/or complex infrastructures. In this paper we present three approaches to the self-folding of structures using low-cost, rapid-prototyped shape memory laminates. These structures require minimal deployment infrastructure, and are activated by light, heat, or electricity. We compare the fabrication of a fundamental structure (a cube) using each approach, and test ways to control fold angles in each case. Finally, for each self-folding approach we present a unique structure that the approach is particularly suited to fold, and discuss the advantages and disadvantages of each approach.

IROS Conference 2013 Conference Paper

Self-pop-up cylindrical structure by global heating

• Shuhei Miyashita
• Cagdas D. Onal
• Daniela Rus

In this study, we demonstrate a new approach to autonomous folding for the body of a 3D robot from a 2D sheet using heat. We approach this challenge by folding a 0. 27 mm sheet-like material into a structure. We utilize the thermal deformation of a contractive sheet sandwiched by rigid structural layers. During this “baking” process, the heat applied on the entire sheet induces contraction of the contracting layer and, thus, forms an instructed bend in the sheet. To attain the targeted folding angles, the V-fold Spans method is used. The targeted angle θ out can be kinematically encoded into crease geometry. The realization of this angle in the folded structure can be approximately controlled by a contraction angle θ in. The process is non-reversible, is reliable, and it is relatively fast. Our method can be applied simultaneously to all the folds in multi-creased origami structures. We demonstrate the use of this method to create a light-weight mobile robot.

ICRA Conference 2013 Conference Paper

Synthetic 2D LIDAR for precise vehicle localization in 3D urban environment

• Zhuang Jie Chong
• Baoxing Qin
• Tirthankar Bandyopadhyay
• Marcelo H. Ang
• Emilio Frazzoli
• Daniela Rus

This paper presents a precise localization algorithm for vehicles in 3D urban environment with only one 2D LIDAR and odometry information. A novel idea of synthetic 2D LIDAR is proposed to solve the localization problem on a virtual 2D plane. A Monte Carlo Localization scheme is adopted for vehicle position estimation, based on synthetic LIDAR measurements and odometry information. The accuracy and robustness of the proposed algorithm are demonstrated by performing real time localization in a 1. 5 km driving test around the NUS campus area.

ICRA Conference 2012 Conference Paper

A distributed algorithm for 2D shape duplication with smart pebble robots

• Kyle Gilpin
• Daniela Rus

We present our digital fabrication technique for manufacturing active objects in 2D from a collection of smart particles. Given a passive model of the object to be formed, we envision submerging this original in a vat of smart particles, executing the new shape duplication algorithm described in this paper, and then brushing aside any extra modules to reveal both the original object and an exact copy, side-by-side. Extensions to the duplication algorithm can be used to create a magnified version of the original or multiple copies of the model object. Our novel duplication algorithm uses a distributed approach to identify the geometric specification of the object being duplicated and then forms the duplicate from spare modules in the vicinity of the original. This paper details the duplication algorithm and the features that make it robust to (1) an imperfect packing of the modules around the original object; (2) missing communication links between neighboring modules; and (3) missing modules in the vicinity of the duplicate object(s). We show that the algorithm requires O(1) storage space per module and that the algorithm exchanges O(n) messages per module. Finally, we present experimental results from 60 hardware trials and 150 simulations. These experiments demonstrate the algorithm working correctly and reliably despite broken communication links and missing modules.

IROS Conference 2012 Conference Paper

Autonomy for mobility on demand

• Zhuang Jie Chong
• Baoxing Qin
• Tirthankar Bandyopadhyay
• Tichakorn Wongpiromsarn
• Brice Rebsamen
• P. Dai
• S. Kim
• Marcelo H. Ang

We present an autonomous vehicle providing mobility-on-demand service in a crowded urban environment. The focus in developing the vehicle has been to attain autonomous driving with minimal sensing and low cost, off-the-shelf sensors to ensure the system's economic viability. The autonomous vehicle has successfully completed over 50 km handling numerous mobility requests during the course of multiple demonstrations. The video provides an overview of our approach, with special comments on our localization and perception modules showcasing one such request being serviced.

IROS Conference 2012 Conference Paper

Communication coverage for independently moving robots

• Stephanie Gil
• Dan Feldman
• Daniela Rus

We consider the task of providing communication coverage to a group of sensing robots (sensors) moving independently to collect data. We provide communication via controlled placement of router vehicles that relay messages from any sensor to any other sensor in the system under the assumptions of 1) no cooperation from the sensors, and 2) only sensor-router or router-router communication over a maximum distance of R is reliable. We provide a formal framework and design provable exact and approximate (faster) algorithms for finding optimal router vehicle locations that are updated according to sensor movement. Using vehicle limitations, such as bounded control effort and maximum velocities of the sensors, our algorithm approximates areas that each router can reach while preserving connectivity and returns an expiration time window over which these positions are guaranteed to maintain communication of the entire system. The expiration time is compared against computation time required to update positions as a decision variable for choosing either the exact or approximate solution for maintaining connectivity with the sensors on-line.

ICRA Conference 2012 Conference Paper

Controlling the locomotion of a separated inner robot from an outer robot using electropermanent magnets

• Andrew D. Marchese
• Haruhiko Asada
• Daniela Rus

This paper presents the design, modeling, and experimental verification of a novel, programmable connection mechanism for robots separated by a surface. The connector uses electropermanent magnets (EPMs) [1] to establish a continuum of clamping force between the robots, enabling the motion of one robot to slave the other during a variety of maneuvers. The authors design a novel, solid-state EPM arrangement capable of generating up to an estimated 890N of clamping force under environmental loading conditions. A relationship between geometric and environmental variables and connection assembly performance is first modeled and subsequently experimentally characterized. By implementing these connectors in a custom manufactured pair of assembly robots, the authors demonstrate the connection assembly and magnetizing hardware can be compactly fit within an autonomous robot application. We offer this mechanism as a repeatable, easily-automated alternative to robotic systems that depend on mechanic means to regulate clamping force [2].

ICRA Conference 2012 Conference Paper

Curb-intersection feature based Monte Carlo Localization on urban roads

• Baoxing Qin
• Zhuang Jie Chong
• Tirthankar Bandyopadhyay
• Marcelo H. Ang
• Emilio Frazzoli
• Daniela Rus

One of the most prominent features on an urban road is the curb, which defines the boundary of a road surface. An intersection is a junction of two or more roads, appearing where no curb exists. The combination of curb and intersection features and their idiosyncrasies carry significant information about the urban road network that can be exploited to improve a vehicle's localization. This paper introduces a Monte Carlo Localization (MCL) method using the curb-intersection features on urban roads. We propose a novel idea of “Virtual LIDAR” to get the measurement models for these features. Under the MCL framework, above road observation is fused with odometry information, which is able to yield precise localization. We implement the system using a single tilted 2D LIDAR on our autonomous test bed and show robust performance in the presence of occlusion from other vehicles and pedestrians.

ICRA Conference 2012 Conference Paper

Distributed coverage with mobile robots on a graph: Locational optimization

• Seung-kook Yun
• Daniela Rus

This paper presents decentralized algorithms for coverage with mobile robots on a graph. Coverage is an important capability of multi-robot systems engaged in a number of different applications, including placement for environmental modeling, deployment for maximal quality surveillance, and even coordinated construction. We use distributed vertex substitution for locational optimization, and the controllers minimize the corresponding cost functions. We prove that the proposed controller with two-hop communication guarantees convergence to the locally optimal configuration. We evaluate the algorithms in simulations and compare them to the coverage algorithms in a continuous domain.

IROS Conference 2012 Conference Paper

Generating informative paths for persistent sensing in unknown environments

• Daniel E. Soltero
• Mac Schwager
• Daniela Rus

We present an online algorithm for a robot to shape its path to a locally optimal configuration for collecting information in an unknown dynamic environment. As the robot travels along its path, it identifies both where the environment is changing, and how fast it is changing. The algorithm then morphs the robot's path online to concentrate on the dynamic areas in the environment in proportion to their rate of change. A Lyapunov-like stability proof is used to show that, under our proposed path shaping algorithm, the path converges to a locally optimal configuration according to a Voronoi-based coverage criterion. The path shaping algorithm is then combined with a previously introduced speed controller to produce guaranteed persistent monitoring trajectories for a robot in an unknown dynamic environment. Simulation and experimental results with a quadrotor robot support the proposed approach.

ICRA Conference 2012 Conference Paper

How was your day? Online visual workspace summaries using incremental clustering in topic space

• Rohan Paul
• Daniela Rus
• Paul Newman 0001

Someday mobile robots will operate continually. Day after day, they will be in receipt of a never ending stream of images. In anticipation of this, this paper is about having a mobile robot generate apt and compact summaries of its life experience. We consider a robot moving around its environment both revisiting and exploring, accruing images as it goes. We describe how we can choose a subset of images to summarise the robot's cumulative visual experience. Moreover we show how to do this such that the time cost of generating an summary is largely independent of the total number of images processed. No one day is harder to summarise than any other.

IROS Conference 2012 Conference Paper

Incremental temporal logic synthesis of control policies for robots interacting with dynamic agents

• Tichakorn Wongpiromsarn
• Alphan Ulusoy
• Calin Belta
• Emilio Frazzoli
• Daniela Rus

We consider the synthesis of control policies from temporal logic specifications for robots that interact with multiple dynamic environment agents. Each environment agent is modeled by a Markov chain whereas the robot is modeled by a finite transition system (in the deterministic case) or Markov decision process (in the stochastic case). Existing results in probabilistic verification are adapted to solve the synthesis problem. To partially address the state explosion issue, we propose an incremental approach where only a small subset of environment agents is incorporated in the synthesis procedure initially and more agents are successively added until we hit the constraints on computational resources. Our algorithm runs in an anytime fashion where the probability that the robot satisfies its specification increases as the algorithm progresses.

AAAI Conference 2012 Conference Paper

Parsing Outdoor Scenes from Streamed 3D Laser Data Using Online Clustering and Incremental Belief Updates

• Rudolph Triebel
• Rohan Paul
• Daniela Rus
• Paul Newman

In this paper, we address the problem of continually parsing a stream of 3D point cloud data acquired from a laser sensor mounted on a road vehicle. We leverage an online star clustering algorithm coupled with an incremental belief update in an evolving undirected graphical model. The fusion of these techniques allows the robot to parse streamed data and to continually improve its understanding of the world. The core competency produced is an ability to infer object classes from similarities based on appearance and shape features, and to concurrently combine that with a spatial smoothing algorithm incorporating geometric consistency. This formulation of feature-space star clustering modulating the potentials of a spatial graphical model is entirely novel. In our method, the two sources of information: feature similarity and geometrical consistency are fed continually into the system, improving the belief over the class distributions as new data arrives. The algorithm obviates the need for hand-labeled training data and makes no apriori assumptions on the number or characteristics of object categories. Rather, they are learnt incrementally over time from streamed input data. In experiments performed on real 3D laser data from an outdoor scene, we show that our approach is capable of obtaining an everimproving unsupervised scene categorization.

ICRA Conference 2012 Conference Paper

Programming and controlling self-folding robots

• Byoungkwon An
• Daniela Rus

This paper describes a robot in the form of a self-folding sheet that is capable of origami-style autonomous folding. We describe the hardware device we designed and fabricated. The device is a sheet with a box-pleated pattern and an integrated electronic substrate and actuators. The sheet is programmed and controlled to achieve different shapes using an idea called sticker programming. We describe the sticker controller and its instantiation. We also describe the algorithms for programming and controlling a given sheet to self-fold into a desired shape. Finally we present experiments with a 4×4 hardware device and an 8×8 hardware device.

IROS Conference 2012 Conference Paper

Semantic categorization of outdoor scenes with uncertainty estimates using multi-class gaussian process classification

• Rohan Paul
• Rudolph Triebel
• Daniela Rus
• Paul Newman 0001

This paper presents a novel semantic categorization method for 3D point cloud data using supervised, multiclass Gaussian Process (GP) classification. In contrast to other approaches, and particularly Support Vector Machines, which probably are the most used method for this task to date, GPs have the major advantage of providing informative uncertainty estimates about the resulting class labels. As we show in experiments, these uncertainty estimates can either be used to improve the classification by neglecting uncertain class labels or - more importantly - they can serve as an indication of the under-representation of certain classes in the training data. This means that GP classifiers are much better suited in a lifelong learning framework, where not all classes are represented initially, but instead new training data arrives during the operation of the robot.

ICRA Conference 2012 Conference Paper

Stochastic distributed multi-agent planning and applications to traffic

• Sejoon Lim
• Daniela Rus

This paper proposes a method for multi-agent path planning on a road network in the presence of congestion. We suggest a distributed method to find paths for multiple agents by introducing a probabilistic path choice achieving global goals such as the social optimum. This approach, which shows that the global goals can be achieved by local processing using only local information, can be parallelized and sped-up using massive parallel processing. The probabilistic assignment reliably copes with the case of random choices of unidentified agents or random route changes of agents who ignore our path guidance. We provide the analytical result on convergence and running time. We demonstrate and evaluate our algorithm by an implementation using asynchronous computation on multi-core computers.

ICRA Conference 2012 Conference Paper

Stochastic motion planning with path constraints and application to optimal agent, resource, and route planning

• Sejoon Lim
• Daniela Rus

We present algorithms for a motion planning for multiple agents whose goals are to visit multiple locations with probabilistic guarantees for achieving the goal. Though much research has been done in stochastic shortest path algorithms, the existing algorithms focus on the single-origin single-destination problem for one agent. This paper formulates a general framework for the stochastic shortest path problem with visit node constraints designed to achieve a specific goal with multiple agents, multiple resources, and multiple destinations. The constraints are defined by a set of sequences of nodes to be visited. Given predetermined constraints, our motion planning problem consists of finding the best agents, resources, and destinations, and the path through a sequence of nodes representing them. The technique in this paper solves the problem at the same level of complexity as solving the single-origin single-destination problem by parallelization. We demonstrate the algorithm by a Web-based traffic navigation guide system and evaluate the algorithm's performance.

IROS Conference 2012 Conference Paper

Trajectory clustering for motion prediction

• Cynthia R. Sung
• Dan Feldman
• Daniela Rus

We investigate a data-driven approach to robotic path planning and analyze its performance in the context of interception tasks. Trajectories of moving objects often contain repeated patterns of motion, and learning those patterns can yield interception paths that succeed more often. We therefore propose an original trajectory clustering algorithm for extracting motion patterns from trajectory data and demonstrate its effectiveness over the more common clustering approach of using k-means. We use the results to build a Hidden Markov Model of a target's motion and predict movement. Our simulations show that these predictions lead to more effective interception. The results of this work have potential applications in coordination of multi-robot systems, tracking and surveillance tasks, and dynamic obstacle avoidance.

IROS Conference 2011 Conference Paper

A scalable information theoretic approach to distributed robot coordination

• Brian J. Julian
• Michael Angermann
• Mac Schwager
• Daniela Rus

This paper presents a scalable information theoretic approach to infer the state of an environment by distributively controlling robots equipped with sensors. The robots iteratively estimate the environment state using a recursive Bayesian filter, while continuously moving to improve the quality of the estimate by following the gradient of mutual information. Both the filter and the controller use a novel algorithm for approximating the robots' joint measurement probabilities, which combines consensus (for decentralization) and sampling (for scalability). The approximations are shown to approach the true joint measurement probabilities as the size of the consensus rounds grows or as the network becomes complete. The resulting gradient controller runs in constant time with respect to the number of robots, and linear time with respect to the number of sensor measurements and environment discretization cells, while traditional mutual information methods are exponential in all of these quantities. Furthermore, the controller is proven to be convergent between consensus rounds and, under certain conditions, is locally optimal. The complete distributed inference and coordination algorithm is demonstrated in experiments with five quad-rotor flying robots and simulations with 100 robots.

IROS Conference 2011 Conference Paper

Collision avoidance for persistent monitoring in multi-robot systems with intersecting trajectories

• Daniel E. Soltero
• Stephen L. Smith 0001
• Daniela Rus

Persistent robot tasks such as monitoring and cleaning are concerned with controlling mobile robots to act in a changing environment in a way that guarantees that the uncertainty in the system (due to change and to the actions of the robot) remains bounded for all time. Prior work in persistent robot tasks considered only robot systems with collision-free paths that move following speed controllers. In this paper we describe a solution to multi-robot persistent monitoring, where robots have intersecting trajectories. We develop collision and deadlock avoidance algorithms that are based on stopping policies, and quantify the impact of the stopping times on the overall stability of the speed controllers.

IROS Conference 2011 Conference Paper

Constraint-aware coordinated construction of generic structures

• David Stein 0007
• T. Ryan Schoen
• Daniela Rus

This paper presents a constraint-aware decentralized approach to construction with teams of robots. We present an extension to existing work on a distributed controller for robotic construction of simple structures. Our previous work described a set of adaptive algorithms for constructing truss structures given a target geometry using continuous and graph-based equal-mass partitioning [1], [2]. Using this work as a foundation, we present an algorithm which performs construction tasks and conforms to physical constraints while considering those constraints to parallelize tasks. This is accomplished by defining a mass function which reflects the priority of part placement and prevents physically impossible states. This mass function generates a set of pointmasses in ℝ n, and we present a novel algorithm for finding a locally optimal, equal-mass, convex tessellation of such a set.

ICRA Conference 2011 Conference Paper

Decentralized self-repair to maintain connectivity and coverage in networked multi-robot systems

• Anna Derbakova
• Nikolaus Correll
• Daniela Rus

We present a suite of algorithms that enable a team of mobile robots to repair connectivity in a wireless mesh network. Each robot carries a wireless router and can act as a mobile access point. The algorithms are distributed, with each robot computing it's trajectory using its position, the positions of its neighbors within communication range, and the position of a gateway node. The algorithms are validated via an analytical model as well as field experiments with 7 Create robots.

ICRA Conference 2011 Conference Paper

Making self-disassembling objects with multiple components in the Robot Pebbles system

• Kyle Gilpin
• Kent Koyanagi
• Daniela Rus

This paper describes several novel algorithms for shape formation by subtraction in programmable matter systems. These algorithms allow the simultaneous formation of multiple different shapes from a single block of host material. The resulting shapes are allowed to intertwine in arbitrarily complex ways. We also present a proof that the algorithms operate correctly to form the desired shapes. Finally, we show experimental results from close to 100 trials using both the Robot Pebbles hardware and a unique software simulator. Multiple trials of several different experiments demonstrate the algorithms operating correctly.

IROS Conference 2011 Conference Paper

Optimal multi-robot path planning with Temporal Logic constraints

• Alphan Ulusoy
• Stephen L. Smith 0001
• Xu Chu Ding
• Calin Belta
• Daniela Rus

In this paper we present a method for automatically planning optimal paths for a group of robots that satisfy a common high level mission specification. Each robot's motion in the environment is modeled as a weighted transition system. The mission is given as a Linear Temporal Logic formula. In addition, an optimizing proposition must repeatedly be satisfied. The goal is to minimize the maximum time between satisfying instances of the optimizing proposition. Our method is guaranteed to compute an optimal set of robot paths. We utilize a timed automaton representation in order to capture the relative position of the robots in the environment. We then obtain a bisimulation of this timed automaton as a finite transition system that captures the joint behavior of the robots and apply our earlier algorithm for the single robot case to optimize the group motion. We present a simulation of a persistent monitoring task in a road network environment.

ICRA Conference 2011 Conference Paper

Persistent monitoring of changing environments using a robot with limited range sensing

• Stephen L. Smith 0001
• Mac Schwager
• Daniela Rus

This paper presents controllers that enable a mobile robot to persistently monitor or sweep a changing environment. The changing environment is modeled as an accumulation function which grows in areas that are not within range of the robot, and decreases in areas that are within range of the robot. The robot must continually move through the environment to prevent the accumulation of any area from growing unbounded. We consider the case in which a predefined path is given for the robot, and we focus on controlling the robot's speed along the path. We characterize necessary and sufficient conditions on the speed controller of the robot for keeping the accumulation function bounded. We then search among the space of speed controllers that are parametrized by a finite set of basis functions. We develop a linear program to compute the optimal speed controller; that which minimizes the accumulation over the environment. Simulation results illustrate the performance of the controllers.

ICRA Conference 2011 Conference Paper

Persistent ocean monitoring with underwater gliders: Towards accurate reconstruction of dynamic ocean processes

• Ryan N. Smith
• Mac Schwager
• Stephen L. Smith 0001
• Daniela Rus
• Gaurav S. Sukhatme

This paper proposes a path planning algorithm and a velocity control algorithm for underwater gliders to persistently monitor a patch of ocean. The algorithms address a pressing need among ocean scientists to collect high-value data for studying ocean events of scientific and environmental interest, such as the occurrence of harmful algal blooms. The path planner optimizes a cost function that blends two competing factors: it maximizes the information value of the path, while minimizing the deviation from the path due to ocean currents. The speed control algorithm then optimizes the speed along the planned path so that higher resolution samples are collected in areas of higher information value. The resulting paths are closed circuits that can be repeatedly traversed to collect long term ocean data in dynamic environments. The algorithms were tested during sea trials on an underwater glider operating off the coast of southern California over the course of several weeks. The results show significant improvements in data resolution and path reliability compared to a sampling path that is typically used in the region.

IROS Conference 2011 Conference Paper

Soft robot actuators using energy-efficient valves controlled by electropermanent magnets

• Andrew D. Marchese
• Cagdas D. Onal
• Daniela Rus

This paper presents the design, fabrication, and evaluation of a novel type of valve that uses an electropermanent magnet [1]. This valve is then used to build actuators for a soft robot. The developed EPM valves require only a brief (5 ms) pulse of current to turn flow on or off for an indefinite period of time. EPMvalves are characterized and demonstrated to be well suited for the control of elastomer fluidic actuators. The valves drive the pressurization and depressurization of fluidic channels within soft actuators. Furthermore, the forward locomotion of a soft, multi-actuator rolling robot is driven by EPM valves. The small size and energy-efficiency of EPM valves may make them valuable in soft mobile robot applications.

ICRA Conference 2011 Conference Paper

Time scales and stability in networked multi-robot systems

• Mac Schwager
• Nathan Michael
• Vijay Kumar 0001
• Daniela Rus

This paper examines the dynamic interplay between decentralized controllers and mesh networking protocols for controlling groups of robots. A proportional controller is used to maintain robots in a formation based on estimates of the robots' states observed through the network. The state information is propagated through the network using a flooding algorithm, which introduces topology-dependent time delays. The coupled interaction of information flow over the network with the dynamics of the robots is modeled as a linear dynamical system. With this model it is shown that systems made up of robots with stable first order dynamics are stable for all network update times, positive feedback gains, and connected communication graphs. With higher order robot dynamics it is found that stability is a complex and counter intuitive function of feedback gain and network update time. A performance metric is proposed for analyzing the convergence rate of the multi-robot system. Experiments with flying quadrotor robots verify the predictions of the model and the performance metric.

ICRA Conference 2011 Conference Paper

Towards printable robotics: Origami-inspired planar fabrication of three-dimensional mechanisms

• Cagdas D. Onal
• Robert J. Wood
• Daniela Rus

This work presents a technique which allows the application of 2-D fabrication methods to build 3-D robotic systems. The ability to print robots introduces a fast and low-cost fabrication method to modern, real-world robotic applications. To this end, we employ laser-engraved origami patterns to build a new class of robotic systems for mobility and manipulation. Origami is suitable for printable robotics as it uses only a flat sheet as the base structure for building complicated functional shapes, which can be utilized as robot bodies. An arbitrarily complex folding pattern can be used to yield an array of functionalities, in the form of actuated hinges or active spring elements. For actuation, we use compact NiTi coil actuators placed on the body to move parts of the structure on-demand. We demonstrate, as a proof-of-concept case study, the end-to-end fabrication and assembly of a simple mobile robot that can undergo worm-like peristaltic locomotion.

ICRA Conference 2010 Conference Paper

Adaptation to robot failures and shape change in decentralized construction

• Seung-kook Yun
• Daniela Rus

Our prior work [1] presented a decentralized algorithm for coordinating the construction of a truss structure out of multiple components. In this paper, we discuss adaptation in decentralized construction. We partition construction in two tasks, tool delivery and assembly. Each task is performed by a networked team of specialized robots. We analyze the performance of the algorithms using the balls into bins problem, and show their adaptation to failure of robots, dynamic constraints, multiple types of elements and reconfiguration. The algorithms can be used for general types of source elements.

ICRA Conference 2010 Conference Paper

Complete SE 3 underwater robot control with arbitrary thruster configurations

• Marek Doniec
• Iuliu Vasilescu
• Carrick Detweiler
• Daniela Rus

We present a control algorithm for autonomous underwater robots with modular thruster configuration. The algorithm can handle arbitrary thruster configurations. It maintains the robot's desired attitude while solving for translational motion. The attitude can be arbitrarily chosen from the special orthogonal group SO 3 allowing the robot all possible orientations. The desired translational velocities can be chosen from R 3 allowing the robot to follow arbitrary trajectories underwater. If the robot is not fully holonomic then the controller chooses the closest possible solution using least squares and outputs the error vector. We verify the controller with experiments using our autonomous underwater robot AMOUR. We achieve roll errors of 1. 0 degree (2. 1 degrees standard deviation) and pitch errors of 1. 5 degrees (1. 8 degrees standard deviation). We also demonstrate experimentally that the controller can handle both nonholonomic and fully holonomic thruster configurations of the robot. In the later case we show how depth can be maintained while performing 360 degree rolls. Further, we demonstrate an input device that allows a user to control the robot's attitude while moving along a desired trajectory.

IROS Conference 2010 Conference Paper

Distributed Coverage Control on Surfaces in 3D Space

• Andreas Breitenmoser
• Jean-Claude Metzger
• Roland Siegwart
• Daniela Rus

This paper addresses the problem of deploying a group of networked robots on a non-planar surface embedded in 3D space. Two distributed coverage control algorithms are presented that both provide a solution to the problem by discrete coverage of a graph. The first method computes shortest paths and runs the Lloyd algorithm on the graph to obtain a centroidal Voronoi tessellation. The second method uses the Euclidean distance measure and locally exchanges mesh cells between approximated Voronoi regions to reach an optimal robot configuration. Both methods are compared and evaluated in simulations and in experiments with five robots on a curved surface.

IROS Conference 2010 Conference Paper

Experiments in decentralized robot construction with tool delivery and assembly robots

• Adrienne Bolger
• Matthew Faulkner 0003
• David Stein 0007
• Lauren White
• Seung-kook Yun
• Daniela Rus

Our prior work [1] presented a decentralized algorithm for coordinating the construction of truss shaped objects out of multiple components (rods and connectors). In this paper, we consider how to transfer the theory to practice, implementing the algorithm to create a decentralized multi robot construction system. The system is composed of mobile manipulators and smarts parts with an embedded communication device. We discuss the delivery and assembly algorithms that comprise this system and the assumptions behind them. We present data from extensive hardware experiments with 4 robots coordinating an assembly task.

IROS Conference 2010 Conference Paper

Making shapes from modules by magnification

• Byoungkwon An
• Daniela Rus

We present a distributed algorithm for creating a modular shape by magnification. The input to the algorithm is presented with a small scale version of the desired shape and a magnification factor m. The output of the system is the object that corresponds to the m-fold magnification of the input shape. We describe and analyze a distributed algorithm for this capability and present simulation results. Making shapes by magnification can be viewed as a programming interface for creating objects by programming matter.

IROS Conference 2010 Conference Paper

Optimal path planning under temporal logic constraints

• Stephen L. Smith 0001
• Jana Tumova
• Calin Belta
• Daniela Rus

In this paper we present a method for automatically generating optimal robot trajectories satisfying high level mission specifications. The motion of the robot in the environment is modeled as a weighted transition system. The mission is specified by a general linear temporal logic formula. In addition, we require that an optimizing proposition must be repeatedly satisfied. The cost function that we seek to minimize is the maximum time between satisfying instances of the optimizing proposition. For every environment model, and for every formula, our method computes a robot trajectory which minimizes the cost function. The problem is motivated by robotic monitoring and data gathering. In this setting, the optimizing proposition is satisfied at locations where data can be uploaded, and the formula specifies a an infinite horizon data collection mission. Our method utilizes Büchi automata to produce an automaton (which can be thought of as a graph) whose runs satisfy the temporal logic formula. We then present a graph algorithm which computes a path corresponding to the optimal robot trajectory. We also present an implementation for a robot performing a data gathering mission.

ICRA Conference 2010 Conference Paper

Optimizing communication in air-ground robot networks using decentralized control

• Stephanie Gil
• Mac Schwager
• Brian J. Julian
• Daniela Rus

We develop a distributed controller to position a team of aerial vehicles in a configuration that optimizes communication-link quality, to support a team of ground vehicles performing a collaborative task. We propose a gradient-based control approach where agents' positions locally minimize a physically motivated cost function. The contributions of this paper are threefold. We formulate of a cost function that incorporates a continuous, physical model of signal quality, SIR. We develop a non-smooth gradient-based controller that positions aerial vehicles to acheive optimized signal quality amongst all vehicles in the system. This controller is provably convergent while allowing for non-differentiability due to agents moving in or out of communication with one another. Lastly, we guarantee that given certain initial conditions or certain values of the control parameters, aerial vehicles will never disconnect the connectivity graph. We demonstrate our controller on hardware experiments using AscTec Hummingbird quadrotors and provide aggregate results over 10 trials. We also provide hardware-in-the-loop and MATALB simulation results, which demonstrate positioning of the aerial vehicles to minimize the cost function H and improve signal-quality amongst all communication links in the ground/air robot team.

ICRA Conference 2010 Conference Paper

Peristaltic locomotion with antagonistic actuators in soft robotics

• Sangok Seok
• Cagdas D. Onal
• Robert J. Wood
• Daniela Rus
• Sangbae Kim

This paper presents a soft robotic platform that exhibits peristaltic locomotion. The design principle is based on the unique antagonistic arrangement of radial/circular and longitudinal muscle groups of Oligochaeta. Sequential antagonistic motion is achieved in a flexible braided mesh-tube structure with NiTi coil actuators. A numerical model for the mesh structure describes how peristaltic motion induces robust locomotion and details the deformation by the contraction of NiTi actuators. Several peristaltic locomotion modes are modeled, tested, and compared on the basis of locomotion speed. The entire mechanical structure is made of flexible mesh materials and can withstand significant external impacts during locomotion. This approach can enable a completely soft robotic platform by employing a flexible control unit and energy sources.

ICRA Conference 2010 Conference Paper

Robot pebbles: One centimeter modules for programmable matter through self-disassembly

• Kyle Gilpin
• Ara N. Knaian
• Daniela Rus

This paper describes the design, fabrication, and experimental results of a programmable matter system capable of 2D shape formation through subtraction. The system is composed of autonomous 1cm modules which use custom-designed electropermanent magnets to bond, communicate, and share power with their neighbors. Given an initial block composed of many of these modules latched together in a regular crystalline structure, our system is able to form shapes by detaching the unnecessary modules. Many experiments show that the modules in our system are able to distribute data at 9600bps to their neighbors with a 98. 5% success rate after four retries, and the connectors are able to support over 85 times the weight of a single module.

IROS Conference 2010 Conference Paper

Using optical communication for remote underwater robot operation

• Marek Doniec
• Carrick Detweiler
• Iuliu Vasilescu
• Daniela Rus

Underwater vehicles are typically operated using a tether or a slow acoustic link. We present an underwater optical communication system that enables a high-throughput and low-latency link to an underwater robot. The optical link allows the robot to operate in cluttered environments without the need for a tether. We demonstrate the performance of the system in a number of experiments which characterize the optical link and demonstrate remote control of the robot using a human input device.

ICRA Conference 2010 Conference Paper

Voronoi coverage of non-convex environments with a group of networked robots

• Andreas Breitenmoser
• Mac Schwager
• Jean-Claude Metzger
• Roland Siegwart
• Daniela Rus

This paper presents a solution to decentralized Voronoi coverage in non-convex polygonal environments. We show that complications arise when existing approaches to Voronoi coverage are applied for deploying a group of robots in non-convex environments. We present an algorithm that is guaranteed to converge to a local optimum. Our algorithm combines classical Voronoi coverage with the Lloyd algorithm and the local path planning algorithm TangentBug to compute the motion of the robots around obstacles and corners. We present the algorithm and prove convergence and optimality. We also discuss experimental results from an implementation with five robots.

ICRA Conference 2009 Conference Paper

Ad-hoc wireless network coverage with networked robots that cannot localize

• Nikolaus Correll
• Jonathan Bachrach
• Daniel Vickery
• Daniela Rus

We study a fully distributed, reactive algorithm for deployment and maintenance of a mobile communication backbone that provides an area around a network gateway with wireless network access for higher-level agents. Possible applications of such a network are distributed sensor networks as well as communication support for disaster or military operations. The algorithm has minimalist requirements on the individual robotic node and does not require any localization. This makes the proposed solution suitable for deployment of large numbers of comparably cheap mobile communication nodes and as a backup solution for more capable systems in GPS-denied environments. Robots keep exploring the configuration space by random walk and stop only if their current location satisfies user-specified constraints on connectivity (number of neighbors). Resulting deployments are robust and convergence is analyzed using both kinematic simulation with a simplified collision and communication model as well as a probabilistic macroscopic model. The approach is validated on a team of 9 iRobot Create robots carrying wireless access points in an indoor environment.

IROS Conference 2009 Conference Paper

Building a distributed robot garden

• Nikolaus Correll
• Nikos Aréchiga
• Adrienne Bolger
• Mario Bollini
• Benjamin Charrow
• Adam Clayton
• Felipe Dominguez
• Kenneth Donahue

This paper describes the architecture and implementation of a distributed autonomous gardening system. The garden is a mesh network of robots and plants. The gardening robots are mobile manipulators with an eye-in-hand camera. They are capable of locating plants in the garden, watering them, and locating and grasping fruit. The plants are potted cherry tomatoes enhanced with sensors and computation to monitor their well-being (e. g. soil humidity, state of fruits) and with networking to communicate servicing requests to the robots. Task allocation, sensing and manipulation are distributed in the system and de-centrally coordinated. We describe the architecture of this system and present experimental results for navigation, object recognition and manipulation.

ICRA Conference 2009 Conference Paper

Distributed coverage control for mobile sensors with location-dependent sensing models

• Ajay Deshpande
• Sameera Poduri
• Daniela Rus
• Gaurav S. Sukhatme

This paper addresses the problem of coverage control of a network of mobile sensors. In the current literature, this is commonly formulated as a locational optimization problem under the assumption that sensing performance is independent of the locations of sensors. We extend this work to a more general framework where the sensor model is location-dependent. We propose a distributed control law and coordination algorithm. If the global sensing performance function is known a priori, we prove that the algorithm is guaranteed to converge. To validate this algorithm, we conduct experiments with indoor and outdoor deployments of Cyclops cameras and model its sensing performance. This model is used to simulate deployments on 1D pathways and study the coverage obtained. We also examine the coverage in the case when the global sensing function is not known and is estimated in an online fashion.

ICRA Conference 2009 Conference Paper

Optimal coverage for multiple hovering robots with downward facing cameras

• Mac Schwager
• Brian J. Julian
• Daniela Rus

This paper presents a distributed control strategy for deploying hovering robots with multiple downward facing cameras to collectively monitor an environment. Information per pixel is proposed as an optimization criterion for multi-camera placement problems. This metric is used to derive a specific cost function for multiple downward facing cameras mounted on hovering robot platforms. The cost function leads to a gradient-based distributed controller for positioning the robots. A convergence proof using LaSalle's invariance principle is given to show that the robots converge to locally optimal positions. The controller is demonstrated in experiments with three flying quad-rotor robots.

ICRA Conference 2009 Conference Paper

Planning the reconfiguration of grounded truss structures with truss climbing robots that carry truss elements

• Seung-kook Yun
• David Alan Hjelle
• Eric Schweikardt
• Hod Lipson
• Daniela Rus

In this paper we describe an optimal reconfiguration planning algorithm that morphs a grounded truss structure of known geometry into a new geometry. The plan consists of a sequence of paths to move truss elements to their new locations that generate the new truss geometry. The trusses are grounded and remain connected at all time. Intuitively, the algorithm grows gradually the new truss structure from the old one. The truss elements are rigid bars joined with 18-way connectors. The paper also introduces the design of a truss-climbing robot that can execute the plan.

ICRA Conference 2008 Conference Paper

A ladybug exploration strategy for distributed adaptive coverage control

• Mac Schwager
• Francesco Bullo
• David Skelly
• Daniela Rus

A control strategy inspired by the hunting tactics of ladybugs is presented to simultaneously achieve sensor coverage and exploration of an area with a group of networked robots. The controller is distributed in that it requires only information local to each robot, and adaptive in that it modifies its behavior based on information in the environment. The ladybug controller is developed as a modification to a basic coverage control law, first for the non-adaptive case, then for the adaptive case. Stability is proven for both cases with a Lyapunov-type proof. Results of numerical simulations are presented.

IROS Conference 2008 Conference Paper

An optical external localization system and applications to indoor tracking

• Srujan Linga
• Binayak Roy
• Haruhiko Asada
• Daniela Rus

Precise robot positioning is important for many applications in indoor environments. Current solutions to the indoor localization problem are either both unreliable and inaccurate, or very expensive. In this paper we propose, design and build a low-cost, robust and accurate indoor localization system using laser light sources. The system calculates the coordinates of a robotic arm by using triangulation algorithms with precisely measured values of the angles of the receiver with respect to the three laser emitters. A system of three rotating lasers and receiver unit was built and deployed in the wing of an aircraft. Using this system, a robotic arm could be localized accurately within error margins defined approximately by Gaussian distributions centered at the object’s true coordinate values and with standard deviations of 0. 19 mm, 0. 11 mm and 0. 34 mm in the x, y and z coordinate directions respectively. The system was also used to detect vertical drop in the robotic arm due to its weight as it extends to perform fitting operations on the skin of the wing. Feedback from the laser localization system was used to adjust the position of the tip of the robotic arm in order to perform a sequence of high precision docking tasks within the aircraft wing

ICRA Conference 2008 Conference Paper

Consensus learning for distributed coverage control

• Mac Schwager
• Jean-Jacques E. Slotine
• Daniela Rus

A decentralized controller is presented that causes a network of robots to converge to a near optimal sensing configuration, while simultaneously learning the distribution of sensory information in the environment. A consensus (or flocking) term is introduced in the learning law to allow sharing of parameters among neighbors, greatly increasing learning convergence rates. Convergence and consensus is proven using a Lyapunov-type proof. The controller with parameter consensus is shown to perform better than the basic controller in numerical simulations.

IROS Conference 2008 Conference Paper

Optimal distributed planning for self assembly of modular manipulators

• Seung-kook Yun
• Daniela Rus

We describe algorithms to build self-assembling robot systems composed of active modular robots and passive bars. The distributed algorithms are based on locally optimal matching. We demonstrate how to build an active structure by the cooperative aggregation and disassembly of modular robotic manipulators. A target structure is modeled as a dynamic graph. We prove that the same optimality - quadratic competitive ratio - as for the static graph can be achieved for the algorithms. We demonstrate how this algorithm can be used to build truss-like structures.

ICRA Conference 2008 Conference Paper

Self assembly of modular manipulators with active and passive modules

• Seung-kook Yun
• Daniela Rus

We describe self-assembling robot arm systems composed of active modular robots and passive bars. We present a case study where the robotic module is the Shady3D robot and the passive component is a rigid bar with embedded IR LEDs. We propose algorithms that demonstrate the cooperative aggregation of modular robotic manipulators with greater capability and workspace out of these two types of elements. We present results from physical experiments in which two 3DOF Shady3D robots and one rigid bar coordinate to self-assemble into a 6DOF manipulator. We then demonstrate cooperative algorithms for forward and inverse kinematics, grasping, and mobility with this arm.

ICRA Conference 2007 Conference Paper

Decentralized, Adaptive Control for Coverage with Networked Robots

• Mac Schwager
• Jean-Jacques E. Slotine
• Daniela Rus

A decentralized, adaptive control law is presented to drive a network of mobile robots to a near-optimal sensing configuration. The control law is adaptive in that it integrates sensor measurements to provide a converging estimate of the distribution of sensory information in the environment. It is decentralized in that it requires only information local to each robot. A Lyapunov-type proof is used to show that the control law causes the network to converge to a near-optimal sensing configuration, and the controller is demonstrated in numerical simulations. This technique suggests a broader application of adaptive control methodologies to decentralized control problems in unknown dynamical environments.

ICRA Conference 2007 Conference Paper

Energy-efficient Autonomous Four-rotor Flying Robot Controlled at 1 kHz

• Daniel Gurdan
• Jan Stumpf
• Michael Achtelik
• Klaus-Michael Doth
• Gerhard Hirzinger
• Daniela Rus

We describe an efficient, reliable, and robust four-rotor flying platform for indoor and outdoor navigation. Currently, similar platforms are controlled at low frequencies due to hardware and software limitations. This causes uncertainty in position control and instable behavior during fast maneuvers. Our flying platform offers a 1 kHz control frequency and motor update rate, in combination with powerful brushless DC motors in a light-weight package. Following a minimalistic design approach this system is based on a small number of low-cost components. Its robust performance is achieved by using simple but reliable highly optimized algorithms. The robot is small, light, and can carry payloads of up to 350g.

ICRA Conference 2007 Conference Paper

Experiments with Underwater Robot Localization and Tracking

• Peter Corke
• Carrick Detweiler
• Matthew Dunbabin
• Michael Hamilton 0001
• Daniela Rus
• Iuliu Vasilescu

This paper describes a novel experiment in which two very different methods of underwater robot localization are compared. The first method is based on a geometric approach in which a mobile node moves within a field of static nodes, and all nodes are capable of estimating the range to their neighbours acoustically. The second method uses visual odometry, from stereo cameras, by integrating scaled optical flow. The fundamental algorithmic principles of each localization technique is described. We also present experimental results comparing acoustic localization with GPS for surface operation, and a comparison of acoustic and visual methods for underwater operation.

ICRA Conference 2007 Conference Paper

Miche: Modular Shape Formation by Self-Dissasembly

• Kyle Gilpin
• Keith Kotay
• Daniela Rus

We describe the design, implementation, and experimentation with a collection of robots that, starting from an amorphous arrangement, can be assembled into arbitrary shapes and then commanded to self-disassemble in an organized manner. Each of the 28 modules in the system is implemented as a 1. 8-inch autonomous cube-shaped robot able to connect to and communicate with its immediate neighbors. Two cooperating microprocessors control each module's magnetic connection mechanisms and infrared communication interfaces. When assembled into a structure, the modules form a system that can be virtually sculpted using a computer interface. We report on the hardware design and experiments from hundreds of trials.

IROS Conference 2007 Conference Paper

Optimal distributed planning of multi-robot placement on a 3D truss

• Seung-kook Yun
• Daniela Rus

This paper considers the problem of allocating tasks among robots that operate on a 3D truss. Each robot is commanded to navigate to a different location for work. When the information about the robots' initial and desired locations are centrally known, this problem reduced to a classical disjoint-path problem. In this paper we consider the distributed problem where each robot knows its own goals only and we wish to plan an optimal set of steps for each robot that minimizes energy while fulfilling the task requirements. The challenge is to cope with possible path collisions. We present and analyze a distributed algorithm. We describe a simulation of this algorithm and show data from a physical experiment.

ICRA Conference 2007 Conference Paper

Shady3D: A Robot that Climbs 3D Trusses

• Yeoreum Yoon
• Daniela Rus

This paper describes a truss climbing robot we designed and prototyped. The robot has a minimalist design with three motive degrees of freedom that enable movement along three-dimensional truss structures. This robot can form a six-degree-of-freedom structure by connecting to another identical module using a passive bar as a medium. We present the design and implementation of this robot, control algorithms for moving the robot in a 3D truss structure, and hardware experiments

ICRA Conference 2006 Conference Paper

Data Muling over Underwater Wireless Sensor Networks using an Autonomous Underwater Vehicle

• Matthew Dunbabin
• Peter Corke
• Iuliu Vasilescu
• Daniela Rus

We present algorithms, systems, and experimental results for underwater data muling. In data muling a mobile agent interacts with static agents to upload, download, or transport data to a different physical location. We consider a system comprising an autonomous underwater vehicle (AUV) and many static underwater sensor nodes (USN) networked together optically and acoustically. The AUV can locate the static nodes using vision and hover above the static nodes for data upload. We describe the hardware and software architecture of this underwater system, as well as experimental data

ICRA Conference 2006 Conference Paper

Distributed Construction by Mobile Robots with Enhanced Building Blocks

• Justin Werfel
• Yaneer Bar-Yam
• Daniela Rus
• Radhika Nagpal

We describe a system in which autonomous robots assemble two-dimensional structures out of square building blocks. A fixed set of local control rules is sufficient for a group of robots to collectively build arbitrary solid structures. We present and compare four versions in which blocks are (1) inert and indistinguishable, (2) uniquely labeled, (3) able to be relabeled by robots, (4) capable of some computation and local communication. Added block capabilities increase the availability of nonlocal structural knowledge, thereby increasing robustness and significantly speeding construction. In this way we extend the principle of stigmergy (storing information in the environment) used by social insects, by increasing the capabilities of the blocks that represent that environmental information. Finally, we describe hardware experiments using a prototype capable of building arbitrary solid 2-D structures

ICRA Conference 2006 Conference Paper

Hierarchical Control for Self-assembling Mobile Trusses with Passive and Active Links

• Carrick Detweiler
• Marsette Vona
• Keith Kotay
• Daniela Rus

This paper explores the space of active modular trusses, ranging from a passive truss with one independent active climbing module to fully self-reconfiguring dynamically controllable trusses comprised of active modules and passive struts. We describe a hardware design for truss climbing and present hierarchical algorithms for controlling hyper-redundant modular trusses

ICRA Conference 2005 Conference Paper

Autonomous Modular Optical Underwater Robot (AMOUR) Design, Prototype and Feasibility Study

• Iuliu Vasilescu
• Paulina Varshavskaya
• Keith Kotay
• Daniela Rus

We propose a novel modular underwater robot which can self-reconfigure by stacking and unstacking its component modules. Applications for this robot include underwater monitoring, exploration, and surveillance. Our current prototype is a single module which contains several subsystems that later will be segregated into different modules. This robot functions as a testbed for the subsystems which are needed in the modular implementation. We describe the module design and discuss the propulsion, docking, and optical ranging subsystems in detail. Experimental results demonstrate depth control, linear motion, target module detection, and docking capabilities.

ICRA Conference 2005 Conference Paper

Efficient Locomotion for a Self-Reconfiguring Robot

• Keith Kotay
• Daniela Rus

In this paper we describe a modular self-reconfiguring robot composed of Molecule robot modules. We present the architecture of this robot and discuss how self-reconfiguration can be used as a locomotion gait for this system. We present two types of locomotion algorithms for this robot: a statically stable tumbling algorithm and a dynamically stable algorithm that achieves locomotion by modifying the center of mass of the robot. For each algorithm we analyze the efficiency of the self-reconfiguration gait for locomotion. Finally we present experimental data for the tumbling algorithm implemented on a four-module Molecule robot.

ICRA Conference 2005 Conference Paper

Reconfiguration Planning Among Obstacles for Heterogeneous Self-Reconfiguring Robots

• Robert Fitch
• Zack J. Butler
• Daniela Rus

Most reconfiguration planners for self-reconfiguring robots do not consider the placement of specific modules within the configuration. Recently, we have begun to investigate heterogeneous reconfiguration planning in lattice-based systems, in which there are various classes of modules. The start and goal configurations specify the class of each module, in addition to placement. Our previous work presents solutions for this problem with unrestricted free space available to the robot during reconfiguration, and also free space limited to a thin connected region over the entire surface of the configuration. In this paper, we further this restriction and define free space by an arbitrarily-shaped bounding region. This addresses the important problem of reconfiguration among obstacles, and reconfiguration over a rigid surface. Our algorithm plans module trajectories through the volume of the structure, and is divided into two phases: shape-forming, and sorting the goal configuration to correctly position modules by class. The worst-case running time for the first phase is O(n 2 ) with O(n 2 ) moves for an n-module robot, and a loose upper bound for the second phase is O(n 4 ) time and moves. However, we show this bound to be Θ (n 2 )time and moves in common instances.

ICRA Conference 2005 Conference Paper

Voronoi Toolpaths for PCB Mechanical Etch: Simple and Intuitive Algorithms with the 3D GPU

• Marsette Vona
• Daniela Rus

We describe VIsolate (Voronoi Isolate), a system which performs geometric computations associated with toolpath planning for mechanical etch (also called isolation routing) of printed-circuit boards, including the computation of a novel Voronoi-based toolpath with some advantages over the current industry practice. We highlight how we use the 3D Graphics Processing Unit (GPU) to implement simple, intuitive algorithms in VIsolate, including polygon overlap detection, 2D offset, and constrained generalized Voronoi diagram computation, building on a method from [1]. Thus, this work also illustrates how we can employ the GPU as a rudimentary ‘ mind’s eye’ for the machine, allowing us to rapidly implement visually-intuitive geometric algorithms.

ICRA Conference 2004 Conference Paper

Autonomous Deployment and Repair of a Sensor Network using an Unmanned Aerial Vehicle

• Peter Corke
• Stefan Hrabar
• Ronald A. Peterson
• Daniela Rus
• Srikanth Saripalli
• Gaurav S. Sukhatme

We describe a sensor network deployment method using autonomous flying robots. Such networks are suitable for tasks such as large-scale environmental monitoring or for command and control in emergency situations. We describe in detail the algorithms used for deployment and for measuring network connectivity and provide experimental data we collected from field trials. A particular focus is on determining gaps in connectivity of the deployed network and generating a plan for a second, repair, pass to complete the connectivity. This project is the result of a collaboration between three robotics labs (CSIRO, USC, and Dartmouth.).

ICRA Conference 2004 Conference Paper

Controlling Mobile Sensors for Monitoring Events with Coverage Constraints

• Zack J. Butler
• Daniela Rus

Sensor networks are systems of many small units that work together to monitor a given environment. Endowing such sensor units with mobility can allow them to reactively converge on more interesting portions of their environment. This enables the concentration of sensing resources where they are most useful and provides robustness by delivering redundancy at the point of interest. However, when converging, in general the sensors should not leave any portion of the environment unsensed. In this paper, we review distributed methods for controlling the sensors and describe a family of distributed methods for retaining coverage while allowing the convergence to proceed where possible. The coverage methods are based on the Voronoi diagram of the sensors' positions, and can use different amounts of communication and computation to produce complete coverage of the environment. We also describe extensions that serve to make coverage more uniform or allow specific areas to be left uncovered. We present implementations of these algorithms in simulation and describe results and avenues of future work.

IROS Conference 2004 Conference Paper

Generic distributed assembly and repair algorithms for self-reconfiguring robots

• Keith Kotay
• Daniela Rus

In this paper we present generic distributed algorithms for assembling and repairing shapes using modular self-reconfiguring robots. The algorithms work in the sliding cube model. Each module independently evaluates a set of local rules using different evaluation models. Two methods are used to determine the correctness of the algorithms - a graph analysis technique which can prove the rule set is correct for specific instances of the algorithm, and a statistical technique which can produce arbitrary bounds on the likelihood that the rule set functions correctly. An extension of the assembly algorithm can be used to produce arbitrary non-cantilevered convex shapes without holes. The algorithms have been implemented and evaluated in simulation.

ICRA Conference 2004 Conference Paper

Interacting with Sensor Networks

• Ronald A. Peterson
• Daniela Rus

We develop distributed algorithms for sensor networks that respond by directing a target (robot or human) through a region. The sensor network models the event levels sensed across a geographical area, adapts to changes, and guides a moving object incrementally across the network. We describe a device we call a flashlight for interacting with the sensor field. This interaction includes collecting navigation information from the sensors in the local neighborhood, activating and deactivating specified areas of the sensor network, and detecting events in the sensor network. We report on hardware experiments using a physical sensor network consisting of Mote sensors.

IROS Conference 2004 Conference Paper

Learning distributed control for modular robots

• Paulina Varshavskaya
• Leslie Pack Kaelbling
• Daniela Rus

We propose to automate controller design for distributed modular robots. In this paper, we present some initial experiments with learning distributed controllers for synthesizing compliant locomotion gaits for modular, self-reconfigurable robots. We use both centralized and distributed policy search and find that the learning approach is promising, as locomotion tasks are learnt well. We also find that the additive nature of the robotic platforms can help speed up learning if we increase the robot size incrementally.

ICRA Conference 2004 Conference Paper

Virtual Fences for Controlling Cows

• Zack J. Butler
• Peter Corke
• Ronald A. Peterson
• Daniela Rus

We describe a moving virtual fence algorithm for herding cows. Each animal in the herd is given a smart collar consisting of a GPS, PDA, wireless networking and a sound amplifier. Using the GPS, the animal's location can be verified relative to the fence boundary. When approaching the perimeter, the animal is presented with a sound stimulus whose effect is to move away. We have developed the virtual fence control algorithm for moving a herd. We present simulation results and data from experiments with 8 cows equipped with smart collars.

IROS Conference 2003 Conference Paper

Reconfiguration planning for heterogeneous self-reconfiguring robots

• Robert Fitch
• Zack J. Butler
• Daniela Rus

Current research in self-reconfiguring robots focuses predominantly on systems of identical modules. However, allowing modules of varying types, with different sensors, for example, is of practical interest. In this paper, we propose the development of an algorithmic basis for heterogeneous self-reconfiguring systems. We demonstrate algorithmic feasibility by presenting O(n/sup 2/) time centralized and O(n/sup 3/) time decentralized solutions to the reconfiguration problem for n non-identical modules. As our centralized time bound is equal to the best published homogeneous solution, we argue that space, as opposed to time, is the critical resource in the reconfiguration problem. Our results encourage the development both of applications that use heterogeneous self-reconfiguration, and also heterogeneous hardware systems.

IJCAI Conference 2003 Conference Paper

Self-reconfiguring Robots: Successes and Challenges

• Daniela Rus

ICRA Conference 2002 Conference Paper

Distributed Goal Recognition Algorithms for Modular Robots

• Zack J. Butler
• Robert Fitch
• Daniela Rus
• Yuhang Wang

Modular robots are systems composed of a number of independent units that can be reconfigured to fit the task at hand. When the modules are computationally independent, they form a large distributed system with no central controller. We are concerned with the ability of such modular robots to easily recognize the achievement (or lack thereof) of a given goal configuration. We present algorithms for a class of 2D and 3D modular robots, along with correctness and running time analysis. We have successfully implemented the 2D algorithm on the second-generation Crystalline Atomic robot, a self-reconfigurable modular robot under development in our laboratory and we present implementation details and experimental results.

IROS Conference 2002 Conference Paper

Experiments in distributed locomotion with a unit-compressible modular robot

• Zack J. Butler
• Robert Fitch
• Daniela Rus

Effective algorithms for modular self-reconfiguring robots should be distributed and parallel. In previous work, we explored general algorithms for locomotion and self-replication and their instantiations to systems in which modules move over the surface of the robot. In this work, we present several algorithms applied to the Crystal robot-two new distributed locomotion algorithms designed specifically for unit-compressible actuation, as well as the adaptation of a generic division algorithm to the Crystal. We also present the integration of a locomotion algorithm with a distributed goal recognition algorithm developed previously. This allows the robot to reconfigure and recognize the achievement of its goal, all without the use of a central controller. We have instantiated all of these algorithms on the Crystal hardware, and we present results of our experiments. These experiments empirically verify the utility of our distributed algorithms on a self-reconfiguring system.

ICRA Conference 2002 Conference Paper

Generic Decentralized Control for a Class of Self-Reconfigurable Robots

• Zack J. Butler
• Keith Kotay
• Daniela Rus
• Kohji Tomita

Previous work on self-reconfiguring modular robots has concentrated primarily on hardware and reconfiguration algorithms for particular systems. We introduce a type of generic locomotion algorithm for self-reconfigurable robots. The algorithms presented are inspired by cellular automata, using geometric rules to control module actions. The actuation model used is a general one, presuming that modules can generally move over the surface of a group of modules. These algorithms can then be instantiated on to a variety of particular systems. Correctness proofs of the rule sets are also given for the generic geometry, with the intent that this analysis can carry over to the instantiated algorithms to provide different systems with correct locomotion algorithms.

IROS Conference 2001 Conference Paper

3D rectilinear motion planning with minimum bend paths

• Robert Fitch
• Zack J. Butler
• Daniela Rus

Computing rectilinear shortest paths in two dimensions has been solved optimally using a number of different techniques. A variety of related problems have been solved, including minimizing the number of bends in the path, the total rectilinear distance, or some combination of both. However, solutions to the 3D versions of these problems are less common. We propose a solution to the 3D minimum-bend path problem, which has theoretical as well as practical interest. Applications include motion planning problems where straight line motion is preferred over taking arbitrary turns. We employ our results in motion planning for self-repair in self-reconfigurable robots.

IROS Conference 2001 Conference Paper

Distributed motion planning for modular robots with unit-compressible modules

• Zack J. Butler
• Sean Byrnes
• Daniela Rus

The ability of self-reconfigurable robots to solve a variety of robot tasks comes in part from their use of a large number of modules. Effective use of these systems requires parallel actuation and planning, both for efficiency and independence from a central controller. This paper presents the PacMan algorithm, a technique for distributed actuation and planning. This algorithm was developed for systems with unit-compressible modules, such as the crystalline robot. We also describe some analytical properties of the PacMan planning and actuation, and discuss simulation and hardware experiments.

YNIMG Journal 2001 Journal Article

The fMRI data center: An introduction

• Jeffrey S. Grethe
• John D. Van Horn
• Jeffrey B. Woodward
• Souheil Inati
• Peter J. Kostelec
• Javed A. Aslam
• Daniel Rockmore
• Daniela Rus

IROS Conference 2000 Conference Paper

A basis for self-reconfiguring robots using crystal modules

• Daniela Rus
• Marsette Vona

We discuss a basis for creating self-reconfiguring robots and instantiate it for crystal modules. Crystalline robots consist of modules that can aggregate together to form distributed robot systems. Crystalline modules are actuated by expanding and contracting each unit. This actuation mechanism permits automated shape metamorphosis. We describe the crystalline module concept and its physical implementation. We prove that crystalline robots are general self-reconfiguring robots.

ICRA Conference 2000 Conference Paper

A Physical Implementation of the Self-Reconfiguring Crystalline Robot

• Daniela Rus
• Marsette Vona

We discuss a physical implementation of the crystalline robot system. Crystalline robots consist of modules that can aggregate together to form distributed robot systems. Crystalline modules are actuated by expanding and contracting each unit. This actuation mechanism permits automated shape metamorphosis. We describe the crystalline module concept and a physical implementation of a robot system with ten units. We describe experiments with this robot.

IROS Conference 2000 Conference Paper

Algorithms for self-reconfiguring molecule motion planning

• Keith Kotay
• Daniela Rus

In this paper we present algorithms for planning the motion of robotic "molecules" (modules) on a substrate of other molecules. Our approach is to divide self-reconfiguration planning into three levels: trajectory planning, configuration planning, and task-level planning. This paper focuses on algorithms for trajectory planning, moving a single molecule from a start location to a goal location, and configuration planning, moving a set of molecules from a starting configuration to a goal configuration. We also present our scaffold planning approach in which the interior of a structure contains three-dimensional tunnels. This allows molecules to move within a structure as well as on the surface, simplifying molecule motion planning as well as increasing parallelism. In addition, we present a new gripper-type connection mechanism for the molecule which does not require power to maintain connections.

ICRA Conference 2000 Conference Paper

Distributed Manipulation of Multiple Objects using Ropes

• Bruce Randall Donald
• Larry Gariepy
• Daniela Rus

This paper describes a system in which multiple robots cooperate to move multiple objects such as groups of boxes using a constrained prehensile manipulation mode, by wrapping ropes around them. The system consists of three manipulation skills: tying ropes around objects, effecting rotations using a flossing manipulation gait, and effecting translations using a ratcheting manipulation gait. We present algorithms for these operations, a numerical analysis for the motion of groups of boxes, and experimental results.

ICRA Conference 1999 Conference Paper

A Mobile Manipulator

• Matthew T. Mason
• Dinesh K. Pai
• Daniela Rus
• Lee R. Taylor
• Michael A. Erdmann

This paper describes a mobile manipulator that uses its wheels for manipulation as well as locomotion. This robot, named the mobipulator, looks like a small car with four independently powered wheels, none of them steered. It is designed to manipulate paper and other objects on the surface of a desk. The wheels are used for locomotion or for manipulation, switching functions dynamically as the task demands. So far we have preliminary demonstrations of a variety of motions, and performance data for the task of moving a sheet of paper in a square while maintaining constant orientation.

SODA Conference 1999 Conference Paper

A Practical Clustering Algorithm for Static and Dynamic Information Organization

• Javed A. Aslam
• Katya Pelekhov
• Daniela Rus

ICRA Conference 1999 Conference Paper

Self-Reconfiguration Planning with Compressible Unit Modules

• Daniela Rus
• Marsette Vona

We discuss a robotic system composed of crystalline modules. Crystalline modules can aggregate together to form distributed robot systems. Crystalline modules can move relative to each other by expanding and contracting. This actuation mechanism permits automated shape metamorphosis. We describe the crystalline module concept and show the basic motions that enable a crystalline robot system to self-reconfigure. We present an algorithm for general self-reconfiguration and describe simulation experiments.

IROS Conference 1998 Conference Paper

Motion synthesis for the self-reconfiguring molecule

• Keith Kotay
• Daniela Rus

In this paper we present a geometric approach to specifying and planning the motion of robotic molecules on a substrate of molecules in O(n) time, where n is the number of molecules in the substrate. We describe a language for specifying the molecule motion. We give algorithms for performing global translations, rotations, and stacking of molecular structures. We show that these motions are sufficient to guarantee certain classes of motion for molecular structures. We also examine a geometric approach to synthesizing language expressions for moving a molecule on a substrate of other molecules.

IROS Conference 1998 Conference Paper

Self-reconfigurable molecule robots as 3D metamorphic robots

• Craig D. McGray
• Daniela Rus

This paper describes a three-dimensional self-reconfiguring system that is capable of reconfiguration and the associated planning in polynomial time. The approach is to reduce a system composed of molecule robots to metamorphic robots. Having done so, we are able to apply polynomial-time planning algorithms that have previously been used only in two-dimensional systems.

ICRA Conference 1998 Conference Paper

The Self-Reconfiguring Robotic Molecule

• Keith Kotay
• Daniela Rus
• Marsette Vona
• Craig D. McGray

We discuss a robotic module called a molecule. Molecules can be the basis for building self-reconfiguring robots. They support multiple modalities of locomotion and manipulation. We describe the design, functionality, and control of the molecule. We show how a set of molecules can aggregate as active three-dimensional structures that can move and change shape. Finally, we discuss our molecule experiments.

IROS Conference 1997 Conference Paper

Task-reconfigurable robots: navigators and manipulators

• Keith Kotay
• Daniela Rus

Task-reconfigurable robots consist of a set of one or more identical autonomous modules that can adapt their shape and function to tasks. We describe a module, the Inchworm robot, that can function as a climbing robot, a manipulator, or a leg in a multi-legged walker. This module can make autonomous transitions between these states. We present the control algorithms that enable our robot to be a versatile navigator and manipulator and report on our experimental results.

IROS Conference 1996 Conference Paper

Navigating 3D steel web structures with an inchworm robot

• Keith Kotay
• Daniela Rus

We wish to navigate across complicated three-dimensional structures. We describe a robot that can propel itself on a web of surfaces oriented around arbitrary directions in three-space. This robot is an inchworm-like robot with a simple, modular, and flexible design. We present control algorithms for the task-level skills that allow the robot to walk vertically, horizontally, and inverted, and the algorithms that allow the robot to make transitions between surfaces. Finally, we discuss our experiments.

IROS Conference 1995 Conference Paper

Moving furniture with teams of autonomous robots

• Daniela Rus
• Bruce Randall Donald
• Jim Jennings

The authors wish to organize furniture in a room with a team of robots that can push objects. The authors show how coordinated pushing by robots can change the pose (position and orientation) of objects and then they ask whether planning, global control, and explicit communication are necessary for cooperatively changing the pose of objects. The authors answer in the negative and present, as witnesses, four cooperative manipulation protocols that use different amounts of state, sensing, and communication. The authors analyze these protocols in the information invariant framework. The authors formalize the notion of resource tradeoffs for robot protocols and give the tradeoffs for the specific protocols discussed here.

ICRA Conference 1994 Conference Paper

Analyzing Teams of Cooperating Mobile Robots

• Bruce Randall Donald
• James S. Jennings
• Daniela Rus

Donald (1993) described a manipulation task for cooperating mobile robots that can push large, heavy objects. There, the author asked whether explicit local and global communication between the agents can be removed from a family of pushing protocols. In this paper, the authors answer in the affirmative. They do so by using the general methods of Donald for analyzing information invariants. The authors discuss several measures for the information complexity of the task of pushing with cooperating mobile robots, and they present a methodology for creating new manipulation strategies out of existing ones. The authors develop and analyze synchronous and asynchronous manipulation protocols for a small team of cooperating mobile robots than can push large boxes. The protocols described have been implemented in several forms on the Cornell mobile robots in the authors' laboratory. >

IROS Conference 1993 Conference Paper

Coordinated manipulation of polygonal objects

• Daniela Rus

In this paper the author presents an efficient algorithm for the coordinated manipulation of polygonal objects with independent robot agents and an efficient decision procedure for whether a polygonal object can undergo any desired reorientation. This algorithm has good stability properties, in that it is not sensitive to small errors in the initial data and it relies on a simple control scheme. The result is a contribution towards realistic task-level planning for robots.

IROS Conference 1993 Conference Paper

Towards task-directed coordinated manipulation

• Allen Back
• Daniela Rus

The authors analyze the feasibility of the finger tracking paradigm for multi-finger manipulation introduced by D. Rus (1992) and relate it to task-directed programming issues. By using the geometry of singularities that arise in solving the instantaneous motion problem they are able to characterize regions of feasibility associated with an initial grasp and quantify how much reorientation is possible starting from this grasp. This results in a provably correct global strategy for reorientation.

ICRA Conference 1992 Conference Paper

Dexterous rotations of polyhedra

• Daniela Rus

Studies a strategy for dexterous manipulation, called finger tracking. It is shown that the paradigm of finger tracking may be used to control the fingers of a robot hand to generate rotational motions of the grasped object. The notion of manipulation refers to the reorientation of an object by a mechanical hand by some degrees, about some axis. The reorientation is accomplished by fine finger motions, and the hand never drops the object in the process. The hand can maintain planar rotational motions. This provides for a simple primitive for high-level, task-directed algorithms, toward relieving users from the complexity of low-level manipulation control. >