Martial Hebert

ICLR Conference 2025 Conference Paper

Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models

• Shuhong Zheng
• Zhipeng Bao
• Ruoyu Zhao
• Martial Hebert
• Yu-Xiong Wang

Beyond high-fidelity image synthesis, diffusion models have recently exhibited promising results in dense visual perception tasks. However, most existing work treats diffusion models as a standalone component for perception tasks, employing them either solely for off-the-shelf data augmentation or as mere feature extractors. In contrast to these isolated and thus sub-optimal efforts, we introduce an integrated, versatile, diffusion-based framework, Diff-2-in-1, that can simultaneously handle both multi-modal data generation and dense visual perception, through a unique exploitation of the diffusion-denoising process. Within this framework, we further enhance discriminative visual perception via multi-modal generation, by utilizing the denoising network to create multi-modal data that mirror the distribution of the original training set. Importantly, Diff-2-in-1 optimizes the utilization of the created diverse and faithful data by leveraging a novel self-improving learning mechanism. Comprehensive experimental evaluations validate the effectiveness of our framework, showcasing consistent performance improvements across various discriminative backbones and high-quality multi-modal data generation characterized by both realism and usefulness. Our project website is available at https://zsh2000.github.io/diff-2-in-1.github.io/.

NeurIPS Conference 2024 Conference Paper

Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding

• Yunze Man
• Shuhong Zheng
• Zhipeng Bao
• Martial Hebert
• Liang-Yan Gui
• Yu-Xiong Wang

Complex 3D scene understanding has gained increasing attention, with scene encoding strategies built on top of visual foundation models playing a crucial role in this success. However, the optimal scene encoding strategies for various scenarios remain unclear, particularly compared to their image-based counterparts. To address this issue, we present the first comprehensive study that probes various visual encoding models for 3D scene understanding, identifying the strengths and limitations of each model across different scenarios. Our evaluation spans seven vision foundation encoders, including image, video, and 3D foundation models. We evaluate these models in four tasks: Vision-Language Scene Reasoning, Visual Grounding, Segmentation, and Registration, each focusing on different aspects of scene understanding. Our evaluation yields key intriguing findings: Unsupervised image foundation models demonstrate superior overall performance, video models excel in object-level tasks, diffusion models benefit geometric tasks, language-pretrained models show unexpected limitations in language-related tasks, and the mixture-of-vision-expert (MoVE) strategy leads to consistent performance improvement. These insights challenge some conventional understandings, provide novel perspectives on leveraging visual foundation models, and highlight the need for more flexible encoder selection in future vision-language and scene understanding tasks.

ICRA Conference 2023 Conference Paper

Discovering Multiple Algorithm Configurations

• Leonid Keselman
• Martial Hebert

Many practitioners in robotics regularly depend on classic, hand-designed algorithms. Often the performance of these algorithms is tuned across a dataset of annotated examples which represent typical deployment conditions. Automatic tuning of these settings is traditionally known as algorithm configuration. In this work, we extend algorithm configuration to automatically discover multiple modes in the tuning dataset. Unlike prior work, these configuration modes represent multiple dataset instances and are detected automatically during the course of optimization. We propose three methods for mode discovery: a post hoc method, a multistage method, and an online algorithm using a multi-armed bandit. Our results characterize these methods on synthetic test functions and in multiple robotics application domains: stereoscopic depth estimation, differentiable rendering, motion planning, and visual odometry. We show the clear benefits of detecting multiple modes in algorithm configuration space.

IROS Conference 2023 Conference Paper

Optimizing Algorithms from Pairwise User Preferences

• Leonid Keselman
• Katherine Shih
• Martial Hebert
• Aaron Steinfeld

Typical black-box optimization approaches in robotics focus on learning from metric scores. However, that is not always possible, as not all developers have ground truth available. Learning appropriate robot behavior in human-centric contexts often requires querying users, who typically cannot provide precise metric scores. Existing approaches leverage human feedback in an attempt to model an implicit reward function; however, this reward may be difficult or impossible to effectively capture. In this work, we introduce SortCMA to optimize algorithm parameter configurations in high dimensions based on pairwise user preferences. SortCMA efficiently and robustly leverages user input to find parameter sets without directly modeling a reward. We apply this method to tuning a commercial depth sensor without ground truth, and to robot social navigation, which involves highly complex preferences over robot behavior. We show that our method succeeds in optimizing for the user's goals and perform a user study to evaluate social navigation results.

ICML Conference 2022 Conference Paper

Generative Modeling for Multi-task Visual Learning

• Zhipeng Bao
• Martial Hebert
• Yu-Xiong Wang

Generative modeling has recently shown great promise in computer vision, but it has mostly focused on synthesizing visually realistic images. In this paper, motivated by multi-task learning of shareable feature representations, we consider a novel problem of learning a shared generative model that is useful across various visual perception tasks. Correspondingly, we propose a general multi-task oriented generative modeling (MGM) framework, by coupling a discriminative multi-task network with a generative network. While it is challenging to synthesize both RGB images and pixel-level annotations in multi-task scenarios, our framework enables us to use synthesized images paired with only weak annotations (i. e. , image-level scene labels) to facilitate multiple visual tasks. Experimental evaluation on challenging multi-task benchmarks, including NYUv2 and Taskonomy, demonstrates that our MGM framework improves the performance of all the tasks by large margins, consistently outperforming state-of-the-art multi-task approaches in different sample-size regimes.

ICLR Conference 2021 Conference Paper

Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis

• Zhipeng Bao
• Yu-Xiong Wang
• Martial Hebert

We propose a novel task of joint few-shot recognition and novel-view synthesis: given only one or few images of a novel object from arbitrary views with only category annotation, we aim to simultaneously learn an object classifier and generate images of that type of object from new viewpoints. While existing work copes with two or more tasks mainly by multi-task learning of shareable feature representations, we take a different perspective. We focus on the interaction and cooperation between a generative model and a discriminative model, in a way that facilitates knowledge to flow across tasks in complementary directions. To this end, we propose bowtie networks that jointly learn 3D geometric and semantic representations with a feedback loop. Experimental evaluation on challenging fine-grained recognition datasets demonstrates that our synthesized images are realistic from multiple viewpoints and significantly improve recognition performance as ways of data augmentation, especially in the low-data regime.

ICRA Conference 2021 Conference Paper

ZePHyR: Zero-shot Pose Hypothesis Rating

• Brian Okorn
• Qiao Gu
• Martial Hebert
• David Held

Pose estimation is a basic module in many robot manipulation pipelines. Estimating the pose of objects in the environment can be useful for grasping, motion planning, or manipulation. However, current state-of-the-art methods for pose estimation either rely on large annotated training sets or simulated data. Further, the long training times for these methods prohibit quick interaction with novel objects. To address these issues, we introduce a novel method for zero-shot object pose estimation in clutter. Our approach uses a hypothesis generation and scoring framework, with a focus on learning a scoring function that generalizes to objects not used for training. We achieve zero-shot generalization by rating hypotheses as a function of unordered point differences. We evaluate our method on challenging datasets with both textured and untextured objects in cluttered scenes and demonstrate that our method significantly outperforms previous methods on this task. We also demonstrate how our system can be used by quickly scanning and building a model of a novel object, which can immediately be used by our method for pose estimation. Our work allows users to estimate the pose of novel objects without requiring any retraining. Additional information can be found on our website https://bokorn.github.io/zephyr/

IROS Conference 2020 Conference Paper

Learning Orientation Distributions for Object Pose Estimation

• Brian Okorn
• Mengyun Xu
• Martial Hebert
• David Held

For robots to operate robustly in the real world, they should be aware of their uncertainty. However, most methods for object pose estimation return a single point estimate of the object's pose. In this work, we propose two learned methods for estimating a distribution over an object's orientation. Our methods take into account both the inaccuracies in the pose estimation as well as the object symmetries. Our first method, which regresses from deep learned features to an isotropic Bingham distribution, gives the best performance for orientation distribution estimation for non-symmetric objects. Our second method learns to compare deep features and generates a non-parameteric histogram distribution. This method gives the best performance on objects with unknown symmetries, accurately modeling both symmetric and non-symmetric objects, without any requirement of symmetry annotation. We show that both of these methods can be used to augment an existing pose estimator. Our evaluation compares our methods to a large number of baseline approaches for uncertainty estimation across a variety of different types of objects. Code available at https://bokorn.github.io/orientation-distributions/.

ICRA Conference 2020 Conference Paper

Learning Shape-based Representation for Visual Localization in Extremely Changing Conditions

• Hae-Gon Jeon
• Sunghoon Im 0001
• Jean Oh
• Martial Hebert

Visual localization is an important task for applications such as navigation and augmented reality, but is a challenging problem when there are changes in scene appearances through day, seasons, or environments. In this paper, we present a convolutional neural network (CNN)-based approach for visual localization across normal to drastic appearance variations such as pre- and post-disaster cases. Our approach aims to address two key challenges: (1) to reduce the biases based on scene textures as in traditional CNNs, our model learns a shape-based representation by training on stylized images; (2) to make the model robust against layout changes, our approach uses the estimated dominant planes of query images as approximate scene coordinates. Our method is evaluated on various scenes including a simulated disaster dataset to demonstrate the effectiveness of our method in significant changes of scene layout. Experimental results show that our method provides reliable camera pose predictions in various changing conditions.

IROS Conference 2020 Conference Paper

MAPPER: Multi-Agent Path Planning with Evolutionary Reinforcement Learning in Mixed Dynamic Environments

• Zuxin Liu
• Baiming Chen
• Hongyi Zhou
• Guru Koushik
• Martial Hebert
• Ding Zhao

Multi-agent navigation in dynamic environments is of great industrial value when deploying a large scale fleet of robot to real-world applications. This paper proposes a decentralized partially observable multi-agent path planning with evolutionary reinforcement learning (MAPPER) method to learn an effective local planning policy in mixed dynamic environments. Reinforcement learning-based methods usually suffer performance degradation on long-horizon tasks with goal-conditioned sparse rewards, so we decompose the long-range navigation task into many easier sub-tasks under the guidance of a global planner, which increases agents' performance in large environments. Moreover, most existing multi-agent planning approaches assume either perfect information of the surrounding environment or homogeneity of nearby dynamic agents, which may not hold in practice. Our approach models dynamic obstacles' behavior with an image-based representation and trains a policy in mixed dynamic environments without homogeneity assumption. To ensure multi-agent training stability and performance, we propose an evolutionary training approach that can be easily scaled to large and complex environments. Experiments show that MAPPER is able to achieve higher success rates and more stable performance when exposed to a large number of non-cooperative dynamic obstacles compared with traditional reaction-based planner LRA* and the state-of-the-art learning-based method.

IROS Conference 2019 Conference Paper

DISC: A Large-scale Virtual Dataset for Simulating Disaster Scenarios

• Hae-Gon Jeon
• Sunghoon Im 0001
• Byeong-Uk Lee
• Dong-Geol Choi
• Martial Hebert
• In-So Kweon

In this paper, we present the first large-scale synthetic dataset for visual perception in disaster scenarios, and analyze state-of-the-art methods for multiple computer vision tasks with reference baselines. We simulated before and after disaster scenarios such as fire and building collapse for fifteen different locations in realistic virtual worlds. The dataset consists of more than 300K high-resolution stereo image pairs, all annotated with ground-truth data for semantic segmentation, depth, optical flow, surface normal estimation and camera pose estimation. To create realistic disaster scenes, we manually augmented the effects with 3D models using physical-based graphics tools. We use our dataset to train state-of-the-art methods and evaluate how well these methods can recognize the disaster situations and produce reliable results on virtual scenes as well as real-world images. The results obtained from each task are then used as inputs to the proposed visual odometry network for generating 3D maps of buildings on fire. Finally, we discuss challenges for future research.

AAAI Conference 2019 Conference Paper

Learning Anytime Predictions in Neural Networks via Adaptive Loss Balancing

• Hanzhang Hu
• Debadeepta Dey
• Martial Hebert
• J. Andrew Bagnell

This work considers the trade-off between accuracy and testtime computational cost of deep neural networks (DNNs) via anytime predictions from auxiliary predictions. Specifically, we optimize auxiliary losses jointly in an adaptive weighted sum, where the weights are inversely proportional to average of each loss. Intuitively, this balances the losses to have the same scale. We demonstrate theoretical considerations that motivate this approach from multiple viewpoints, including connecting it to optimizing the geometric mean of the expectation of each loss, an objective that ignores the scale of losses. Experimentally, the adaptive weights induce more competitive anytime predictions on multiple recognition data-sets and models than non-adaptive approaches including weighing all losses equally. In particular, anytime neural networks (ANNs) can achieve the same accuracy faster using adaptive weights on a small network than using static constant weights on a large one. For problems with high performance saturation, we also show a sequence of exponentially deepening ANNs can achieve near-optimal anytime results at any budget, at the cost of a const fraction of extra computation.

ICRA Conference 2017 Conference Paper

Learning robust failure response for autonomous vision based flight

• Dhruv Mauria Saxena
• Vince Kurtz
• Martial Hebert

The ability of autonomous mobile robots to react to and recover from potential failures of on-board systems is an important area of ongoing robotics research. With increasing emphasis on robust systems and long-term autonomy, mobile robots must be able to respond safely and intelligently to dangerous situations. Recent developments in computer vision have made autonomous vision based navigation possible. However, vision systems are known to be imperfect and prone to failure due to variable lighting, terrain changes, and other environmental variables. We describe a system for learning simple failure recovery maneuvers based on experience. This involves both recognizing when the vision system is prone to failure, and associating failures with appropriate responses that will most likely help the robot recover. We implement this system on an autonomous quadrotor and demonstrate that behaviors learned with our system are effective in recovering from situational perception failure, thereby improving reliability in cluttered and uncertain environments.

NeurIPS Conference 2017 Conference Paper

Learning to Model the Tail

• Yu-Xiong Wang
• Deva Ramanan
• Martial Hebert

We describe an approach to learning from long-tailed, imbalanced datasets that are prevalent in real-world settings. Here, the challenge is to learn accurate "few-shot'' models for classes in the tail of the class distribution, for which little data is available. We cast this problem as transfer learning, where knowledge from the data-rich classes in the head of the distribution is transferred to the data-poor classes in the tail. Our key insights are as follows. First, we propose to transfer meta-knowledge about learning-to-learn from the head classes. This knowledge is encoded with a meta-network that operates on the space of model parameters, that is trained to predict many-shot model parameters from few-shot model parameters. Second, we transfer this meta-knowledge in a progressive manner, from classes in the head to the "body'', and from the "body'' to the tail. That is, we transfer knowledge in a gradual fashion, regularizing meta-networks for few-shot regression with those trained with more training data. This allows our final network to capture a notion of model dynamics, that predicts how model parameters are likely to change as more training data is gradually added. We demonstrate results on image classification datasets (SUN, Places, and ImageNet) tuned for the long-tailed setting, that significantly outperform common heuristics, such as data resampling or reweighting.

NeurIPS Conference 2017 Conference Paper

Predictive-State Decoders: Encoding the Future into Recurrent Networks

• Arun Venkatraman
• Nicholas Rhinehart
• Wen Sun
• Lerrel Pinto
• Martial Hebert
• Byron Boots
• Kris Kitani
• J. Bagnell

Recurrent neural networks (RNNs) are a vital modeling technique that rely on internal states learned indirectly by optimization of a supervised, unsupervised, or reinforcement training loss. RNNs are used to model dynamic processes that are characterized by underlying latent states whose form is often unknown, precluding its analytic representation inside an RNN. In the Predictive-State Representation (PSR) literature, latent state processes are modeled by an internal state representation that directly models the distribution of future observations, and most recent work in this area has relied on explicitly representing and targeting sufficient statistics of this probability distribution. We seek to combine the advantages of RNNs and PSRs by augmenting existing state-of-the-art recurrent neural networks with Predictive-State Decoders (PSDs), which add supervision to the network's internal state representation to target predicting future observations. PSDs are simple to implement and easily incorporated into existing training pipelines via additional loss regularization. We demonstrate the effectiveness of PSDs with experimental results in three different domains: probabilistic filtering, Imitation Learning, and Reinforcement Learning. In each, our method improves statistical performance of state-of-the-art recurrent baselines and does so with fewer iterations and less data.

UAI Conference 2016 Conference Paper

Efficient Feature Group Sequencing for Anytime Linear Prediction

• Hanzhang Hu
• Alexander Grubb
• J. Andrew Bagnell
• Martial Hebert

We consider anytime linear prediction in the common machine learning setting, where features are in groups that have costs. We achieve anytime (or interruptible) predictions by sequencing the computation of feature groups and reporting results using the computed features at interruption. We extend Orthogonal Matching Pursuit (OMP) and Forward Regression (FR) to learn the sequencing greedily under this group setting with costs. We theoretically guarantee that our algorithms achieve near-optimal linear predictions at each budget when a feature group is chosen. With a novel analysis of OMP, we improve its theoretical bound to the same strength as that of FR. In addition, we develop a novel algorithm that consumes cost 4B to approximate the optimal performance of any cost B, and prove that with cost less than 4B, such an approximation is impossible. To our knowledge, these are the first anytime bounds at all budgets. We test our algorithms on two real-world data-sets and evaluate them in terms of anytime linear prediction performance against costweighted Group Lasso and alternative greedy algorithms.

IJCAI Conference 2016 Conference Paper

Inference Machines for Nonparametric Filter Learning

• Arun Venkatraman
• Wen Sun
• Martial Hebert
• Byron Boots
• J. Andrew Bagnell

Data-driven approaches for learning dynamic models for Bayesian filtering often try to maximize the data likelihood given parametric forms for the transition and observation models. However, this objective is usually nonconvex in the parametrization and can only be locally optimized. Furthermore, learning algorithms typically do not provide performance guarantees on the desired Bayesian filtering task. In this work, we propose using inference machines to directly optimize the filtering performance. Our procedure is capable of learning partially-observable systems when the state space is either unknown or known in advance. To accomplish this, we adapt PREDICTIVE STATE INFERENCE MACHINES (PSIMs) by introducing the concept of hints, which incorporate prior knowledge of the state space to accompany the predictive state representation. This allows PSIM to be applied to the larger class of filtering problems which require prediction of a specific parameter or partial component of state. Our PSIM+HINTS adaptation enjoys theoretical advantages similar to the original PSIM algorithm, and we showcase its performance on a variety of robotics filtering problems.

IROS Conference 2016 Conference Paper

Introspective perception: Learning to predict failures in vision systems

• Shreyansh Daftry
• Sam Zeng
• J. Andrew Bagnell
• Martial Hebert

As robots aspire for long-term autonomous operations in complex dynamic environments, the ability to reliably take mission-critical decisions in ambiguous situations becomes critical. This motivates the need to build systems that have situational awareness to assess how quali ed they are at that moment to make a decision. We call this self-evaluating capability as introspection. In this paper, we take a small step in this direction and propose a generic framework for introspective behavior in perception systems. Our goal is to learn a model to reliably predict failures in a given system, with respect to a task, directly from input sensor data. We present this in the context of vision-based autonomous MAV flight in outdoor natural environments, and show that it effectively handles uncertain situations.

AAAI Conference 2016 Conference Paper

Learning by Transferring from Unsupervised Universal Sources

• Yu-Xiong Wang
• Martial Hebert

Category classifiers trained from a large corpus of annotated data are widely accepted as the sources for (hypothesis) transfer learning. Sources generated in this way are tied to a particular set of categories, limiting their transferability across a wide spectrum of target categories. In this paper, we address this largelyoverlooked yet fundamental source problem by both introducing a systematic scheme for generating universal source hypotheses and proposing a principled, scalable approach to automatically tuning the transfer process. Our approach is based on the insights that expressive source hypotheses could be generated without any supervision and that a sparse combination of such hypotheses facilitates recognition of novel categories from few samples. We demonstrate improvements over the state-of-the-art on object and scene classification in the small sample size regime.

NeurIPS Conference 2016 Conference Paper

Learning from Small Sample Sets by Combining Unsupervised Meta-Training with CNNs

• Yu-Xiong Wang
• Martial Hebert

This work explores CNNs for the recognition of novel categories from few examples. Inspired by the transferability properties of CNNs, we introduce an additional unsupervised meta-training stage that exposes multiple top layer units to a large amount of unlabeled real-world images. By encouraging these units to learn diverse sets of low-density separators across the unlabeled data, we capture a more generic, richer description of the visual world, which decouples these units from ties to a specific set of categories. We propose an unsupervised margin maximization that jointly estimates compact high-density regions and infers low-density separators. The low-density separator (LDS) modules can be plugged into any or all of the top layers of a standard CNN architecture. The resulting CNNs significantly improve the performance in scene classification, fine-grained recognition, and action recognition with small training samples.

AAAI Conference 2016 Conference Paper

Online Instrumental Variable Regression with Applications to Online Linear System Identification

• Arun Venkatraman
• Wen Sun
• Martial Hebert
• J. Bagnell
• Byron Boots

Instrumental variable regression (IVR) is a statistical technique utilized to recover unbiased estimators when there are errors in the independent variables. Estimator bias in learned time series models can yield poor performance in applications such as long-term prediction and filtering where the recursive use of the model results in the accumulation of propagated error. However, prior work addressed the IVR objective in the batch setting, where it is necessary to store the entire dataset in memory - an infeasible requirement in large dataset scenarios. In this work, we develop Online Instrumental Variable Regression (OIVR), an algorithm that is capable of updating the learned estimator with streaming data. We show that the online adaptation of IVR enjoys a no-regret performance guarantee with respect to the original batch setting by taking advantage of any no-regret online learning algorithm inside OIVR for the underlying update steps. We experimentally demonstrate the efficacy of our algorithm in combination with popular no-regret online algorithms for the task of learning predictive dynamical system models and on a prototypical econometrics instrumental variable regression problem.

AAAI Conference 2015 Conference Paper

Improving Multi-Step Prediction of Learned Time Series Models

• Arun Venkatraman
• Martial Hebert
• J.. Bagnell

IROS Conference 2015 Conference Paper

Inferring door locations from a teammate's trajectory in stealth human-robot team operations

• Jean Oh
• Luis Ernesto Navarro-Serment
• Arne Suppé
• Anthony Stentz
• Martial Hebert

Robot perception is generally viewed as the interpretation of data from various types of sensors such as cameras. In this paper, we study indirect perception where a robot can perceive new information by making inferences from non-visual observations of human teammates. As a proof-of-concept study, we specifically focus on a door detection problem in a stealth mission setting where a team operation must not be exposed to the visibility of the team's opponents. We use a special type of the Noisy-OR model known as BN2O model of Bayesian inference network to represent the inter-visibility and to infer the locations of the doors, i. e. , potential locations of the opponents. Experimental results on both synthetic data and real person tracking data achieve an F-measure of over. 9 on average, suggesting further investigation on the use of non-visual perception in human-robot team operations.

AAAI Conference 2015 Conference Paper

Toward Mobile Robots Reasoning Like Humans

• Jean Oh
• Arne Suppé
• Felix Duvallet
• Abdeslam Boularias
• Luis Navarro-Serment
• Martial Hebert
• Anthony Stentz
• Jerry Vinokurov

Robots are increasingly becoming key players in human-robot teams. To become effective teammates, robots must possess profound understanding of an environment, be able to reason about the desired commands and goals within a specific context, and be able to communicate with human teammates in a clear and natural way. To address these challenges, we have developed an intelligence architecture that combines cognitive components to carry out high-level cognitive tasks, semantic perception to label regions in the world, and a natural language component to reason about the command and its relationship to the objects in the world. This paper describes recent developments using this architecture on a fielded mobile robot platform operating in unknown urban environments. We report a summary of extensive outdoor experiments; the results suggest that a multidisciplinary approach to robotics has the potential to create competent human-robot teams.

ICRA Conference 2015 Conference Paper

Visual chunking: A list prediction framework for region-based object detection

• Nicholas Rhinehart
• Jiaji Zhou
• Martial Hebert
• J. Andrew Bagnell

We consider detecting objects in an image by iteratively selecting from a set of arbitrarily shaped candidate regions. Our generic approach, which we term visual chunking, reasons about the locations of multiple object instances in an image while expressively describing object boundaries. We design an optimization criterion for measuring the performance of a list of such detections as a natural extension to a common per-instance metric. We present an efficient algorithm with provable performance for building a high-quality list of detections from any candidate set of region-based proposals. We also develop a simple class-specific algorithm to generate a candidate region instance in near-linear time in the number of low-level superpixels that outperforms other region generating methods. In order to make predictions on novel images at testing time without access to ground truth, we develop learning approaches to emulate these algorithms' behaviors. We demonstrate that our new approach outperforms sophisticated baselines on benchmark datasets.

ICRA Conference 2014 Conference Paper

Visual sensing for developing autonomous behavior in snake robots

• Hugo Ponte
• Max Queenan
• Chaohui Gong
• Christoph Mertz
• Matthew J. Travers
• Florian Enner
• Martial Hebert
• Howie Choset

Snake robots are uniquely qualified to investigate a large variety of settings including archaeological sites, natural disaster zones, and nuclear power plants. For these applications, modular snake robots have been tele-operated to perform specific tasks using images returned to it from an onboard camera in the robots head. In order to give the operator an even richer view of the environment and to enable the robot to perform autonomous tasks we developed a structured light sensor that can make three-dimensional maps of the environment. This paper presents a sensor that is uniquely qualified to meet the severe constraints in size, power and computational footprint of snake robots. Using range data, in the form of 3D pointclouds, we show that it is possible to pair high-level planning with mid-level control to accomplish complex tasks without operator intervention.

ICRA Conference 2013 Conference Paper

Efficient 3-D scene analysis from streaming data

• Hanzhang Hu
• Daniel Munoz
• J. Andrew Bagnell
• Martial Hebert

Rich scene understanding from 3-D point clouds is a challenging task that requires contextual reasoning, which is typically computationally expensive. The task is further complicated when we expect the scene analysis algorithm to also efficiently handle data that is continuously streamed from a sensor on a mobile robot. Hence, we are typically forced to make a choice between 1) using a precise representation of the scene at the cost of speed, or 2) making fast, though inaccurate, approximations at the cost of increased misclassifications. In this work, we demonstrate that we can achieve the best of both worlds by using an efficient and simple representation of the scene in conjunction with recent developments in structured prediction in order to obtain both efficient and state-of-the-art classifications. Furthermore, this efficient scene representation naturally handles streaming data and provides a 300% to 500% speedup over more precise representations.

ICRA Conference 2013 Conference Paper

Efficient temporal consistency for streaming video scene analysis

• Ondrej Miksik
• Daniel Munoz
• J. Andrew Bagnell
• Martial Hebert

We address the problem of image-based scene analysis from streaming video, as would be seen from a moving platform, in order to efficiently generate spatially and temporally consistent predictions of semantic categories over time. In contrast to previous techniques which typically address this problem in batch and/or through graphical models, we demonstrate that by learning visual similarities between pixels across frames, a simple filtering algorithm is able to achieve high performance predictions in an efficient and online/causal manner. Our technique is a meta-algorithm that can be efficiently wrapped around any scene analysis technique that produces a per-pixel semantic category distribution. We validate our approach over three different scene analysis techniques on three different datasets that contain different semantic object categories. Our experiments demonstrate that our approach is very efficient in practice and substantially improves the consistency of the predictions over time.

AAMAS Conference 2013 Conference Paper

Enhancing Robot Perception Using Human Teammates

• Jean Oh
• Arne Suppe
• Anthony Stentz
• Martial Hebert

In robotics research, perception is one of the most challenging tasks. In contrast to existing approaches that rely only on computer vision, we propose an alternative method for improving perception by learning from human teammates. To evaluate, we apply this idea to a door detection problem. A set of preliminary experiments has been completed using software agents with real vision data. Our results demonstrate that information inferred from teammate observations significantly improves the perception precision.

ICRA Conference 2013 Conference Paper

Exploiting domain knowledge for Object Discovery

• Alvaro Collet
• Bo Xiong
• Corina Gurau
• Martial Hebert
• Siddhartha S. Srinivasa

In this paper, we consider the problem of Lifelong Robotic Object Discovery (LROD) as the long-term goal of discovering novel objects in the environment while the robot operates, for as long as the robot operates. As a first step towards LROD, we automatically process the raw video stream of an entire workday of a robotic agent to discover objects. We claim that the key to achieve this goal is to incorporate domain knowledge whenever available, in order to detect and adapt to changes in the environment. We propose a general graph-based formulation for LROD in which generic domain knowledge is encoded as constraints. Our formulation enables new sources of domain knowledge—metadata—to be added dynamically to the system, as they become available or as conditions change. By adding domain knowledge, we discover 2. 7· more objects and decrease processing time 190 times. Our optimized implementation, HerbDisc, processes 6 h 20 min of RGBD video of real human environments in 18 min 30 s, and discovers 121 correct novel objects with their 3D models.

AAAI Conference 2013 Conference Paper

Gradient Networks: Explicit Shape Matching Without Extracting Edges

• Edward Hsiao
• Martial Hebert

We present a novel framework for shape-based template matching in images. While previous approaches required brittle contour extraction, considered only local information, or used coarse statistics, we propose to match the shape explicitly on low-level gradients by formulating the problem as traversing paths in a gradient network. We evaluate our algorithm on a challenging dataset of objects in cluttered environments and demonstrate significant improvement over state-of-theart methods for shape matching and object detection.

ICRA Conference 2013 Conference Paper

Learning monocular reactive UAV control in cluttered natural environments

• Stéphane Ross
• Narek Melik-Barkhudarov
• Kumar Shaurya Shankar
• Andreas Wendel
• Debadeepta Dey
• J. Andrew Bagnell
• Martial Hebert

Autonomous navigation for large Unmanned Aerial Vehicles (UAVs) is fairly straight-forward, as expensive sensors and monitoring devices can be employed. In contrast, obstacle avoidance remains a challenging task for Micro Aerial Vehicles (MAVs) which operate at low altitude in cluttered environments. Unlike large vehicles, MAVs can only carry very light sensors, such as cameras, making autonomous navigation through obstacles much more challenging. In this paper, we describe a system that navigates a small quadrotor helicopter autonomously at low altitude through natural forest environments. Using only a single cheap camera to perceive the environment, we are able to maintain a constant velocity of up to 1. 5m/s. Given a small set of human pilot demonstrations, we use recent state-of-the-art imitation learning techniques to train a controller that can avoid trees by adapting the MAVs heading. We demonstrate the performance of our system in a more controlled environment indoors, and in real natural forest environments outdoors.

ICRA Conference 2013 Conference Paper

Multi-armed recommendation bandits for selecting state machine policies for robotic systems

• Pyry Matikainen
• P. Michael Furlong
• Rahul Sukthankar
• Martial Hebert

We investigate the problem of selecting a state-machine from a library to control a robot. We are particularly interested in this problem when evaluating such state machines on a particular robotics task is expensive. As a motivating example, we consider a problem where a simulated vacuuming robot must select a driving state machine well-suited for a particular (unknown) room layout. By borrowing concepts from collaborative filtering (recommender systems such as Netflix and Amazon.com), we present a multi-armed bandit formulation that incorporates recommendation techniques to efficiently select state machines for individual room layouts. We show that this formulation outperforms the individual approaches (recommendation, multi-armed bandits) as well as the baseline of selecting the `average best' state machine across all rooms.

IROS Conference 2012 Conference Paper

An integrated system for autonomous robotics manipulation

• J. Andrew Bagnell
• Felipe Cavalcanti
• Lei Cui 0005
• Thomas Galluzzo
• Martial Hebert
• Moslem Kazemi
• Matthew Klingensmith
• Jacqueline Libby

We describe the software components of a robotics system designed to autonomously grasp objects and perform dexterous manipulation tasks with only high-level supervision. The system is centered on the tight integration of several core functionalities, including perception, planning and control, with the logical structuring of tasks driven by a Behavior Tree architecture. The advantage of the implementation is to reduce the execution time while integrating advanced algorithms for autonomous manipulation. We describe our approach to 3-D perception, real-time planning, force compliant motions, and audio processing. Performance results for object grasping and complex manipulation tasks of in-house tests and of an independent evaluation team are presented.

AAAI Conference 2012 Conference Paper

Using Expectations to Drive Cognitive Behavior

• Unmesh Kurup
• Christian Lebiere
• Anthony Stentz
• Martial Hebert

Generating future states of the world is an essential component of high level cognitive tasks such as planning. We explore the notion that such future state generation is more widespread and forms an integral part of cognition. We call these generated states expectations, and propose that cognitive systems constantly generate expectations, match them to observed behavior and react when a difference exists between the two. We describe an ACT R model that performs expectation driven cognition on two tasks pedestrian tracking and behavior classification. The model generates expectations of pedestrian movements to track them. The model also uses differences in expectations to identify distinctive features that differentiate these tracks. During learning, the model learns the association between these features and the various behaviors. During testing, it classifies pedestrian tracks by recalling the behavior associated with the features of each track. We tested the model on both single and multiple behavior datasets and compared the results against a k NN classifier. The k NN classifier outperformed the model in correct classifications, but the model had fewer incorrect classifications in the multiple behavior case, and both systems had about equal incorrect classifications in the single behavior case.

ICRA Conference 2011 Conference Paper

3-D scene analysis via sequenced predictions over points and regions

• Xuehan Xiong
• Daniel Munoz
• J. Andrew Bagnell
• Martial Hebert

We address the problem of understanding scenes from 3-D laser scans via per-point assignment of semantic labels. In order to mitigate the difficulties of using a graphical model for modeling the contextual relationships among the 3-D points, we instead propose a multi-stage inference procedure to capture these relationships. More specifically, we train this procedure to use point cloud statistics and learn relational information (e. g. , tree-trunks are below vegetation) over fine (point-wise) and coarse (region-wise) scales. We evaluate our approach on three different datasets, that were obtained from different sensors, and demonstrate improved performance.

ICRA Conference 2011 Conference Paper

Structure discovery in multi-modal data: A region-based approach

• Alvaro Collet
• Siddhartha S. Srinivasa
• Martial Hebert

The ability of a perception system to discern what is important in a scene and what is not is an invaluable asset, with multiple applications in object recognition, people detection and SLAM, among others. In this paper, we aim to analyze all sensory data available to separate a scene into a few physically meaningful parts, which we term structure, while discarding background clutter. In particular, we consider the combination of image and range data, and base our decision in both appearance and 3D shape. Our main contribution is the development of a framework to perform scene segmentation that preserves physical objects using multi-modal data. We combine image and range data using a novel mid-level fusion technique based on the concept of regions that avoids any pixel-level correspondences between data sources. We associate groups of pixels with 3D points into multi-modal regions that we term regionlets, and measure the structure-ness of each regionlet using simple, bottom-up cues from image and range features. We show that the highest-ranked regionlets correspond to the most prominent objects in the scene. We verify the validity of our approach on 105 scenes of household environments.

NeurIPS Conference 2010 Conference Paper

Estimating Spatial Layout of Rooms using Volumetric Reasoning about Objects and Surfaces

• Abhinav Gupta
• Martial Hebert
• Takeo Kanade
• David Blei

There has been a recent push in extraction of 3D spatial layout of scenes. However, none of these approaches model the 3D interaction between objects and the spatial layout. In this paper, we argue for a parametric representation of objects in 3D, which allows us to incorporate volumetric constraints of the physical world. We show that augmenting current structured prediction techniques with volumetric reasoning significantly improves the performance of the state-of-the-art.

IROS Conference 2010 Conference Paper

People helping robots helping people: Crowdsourcing for grasping novel objects

• Alexander Sorokin
• Dmitry Berenson
• Siddhartha S. Srinivasa
• Martial Hebert

For successful deployment, personal robots must adapt to ever-changing indoor environments. While dealing with novel objects is a largely unsolved challenge in AI, it is easy for people. In this paper we present a framework for robot supervision through Amazon Mechanical Turk. Unlike traditional models of teleoperation, people provide semantic information about the world and subjective judgements. The robot then autonomously utilizes the additional information to enhance its capabilities. The information can be collected on demand in large volumes and at low cost. We demonstrate our approach on the task of grasping unknown objects.

NeurIPS Conference 2009 Conference Paper

An Integer Projected Fixed Point Method for Graph Matching and MAP Inference

• Marius Leordeanu
• Martial Hebert
• Rahul Sukthankar

Graph matching and MAP inference are essential problems in computer vision and machine learning. We introduce a novel algorithm that can accommodate both problems and solve them efficiently. Recent graph matching algorithms are based on a general quadratic programming formulation, that takes in consideration both unary and second-order terms reflecting the similarities in local appearance as well as in the pairwise geometric relationships between the matched features. In this case the problem is NP-hard and a lot of effort has been spent in finding efficiently approximate solutions by relaxing the constraints of the original problem. Most algorithms find optimal continuous solutions of the modified problem, ignoring during the optimization the original discrete constraints. The continuous solution is quickly binarized at the end, but very little attention is put into this final discretization step. In this paper we argue that the stage in which a discrete solution is found is crucial for good performance. We propose an efficient algorithm, with climbing and convergence properties, that optimizes in the discrete domain the quadratic score, and it gives excellent results either by itself or by starting from the solution returned by any graph matching algorithm. In practice it outperforms state-or-the art algorithms and it also significantly improves their performance if used in combination. When applied to MAP inference, the algorithm is a parallel extension of Iterated Conditional Modes (ICM) with climbing and convergence properties that make it a compelling alternative to the sequential ICM. In our experiments on MAP inference our algorithm proved its effectiveness by outperforming ICM and Max-Product Belief Propagation.

ICRA Conference 2009 Conference Paper

Onboard contextual classification of 3-D point clouds with learned high-order Markov Random Fields

• Daniel Munoz
• Nicolas Vandapel
• Martial Hebert

Contextual reasoning through graphical models such as Markov random fields often show superior performance against local classifiers in many domains. Unfortunately, this performance increase is often at the cost of time consuming, memory intensive learning and slow inference at testing time. Structured prediction for 3-D point cloud classification is one example of such an application. In this paper we present two contributions. First we show how efficient learning of a random field with higher-order cliques can be achieved using subgradient optimization. Second, we present a context approximation using random fields with high-order cliques designed to make this model usable online, onboard a mobile vehicle for environment modeling. We obtained results with the mobile vehicle on a variety of terrains, at 1/3 Hz for a map 25 times 50 meters and a vehicle speed of 1-2 m/s.

IROS Conference 2009 Conference Paper

Planning-based prediction for pedestrians

• Brian D. Ziebart
• Nathan D. Ratliff
• Garratt Gallagher
• Christoph Mertz
• Kevin M. Peterson
• J. Andrew Bagnell
• Martial Hebert
• Anind K. Dey

We present a novel approach for determining robot movements that efficiently accomplish the robot's tasks while not hindering the movements of people within the environment. Our approach models the goal-directed trajectories of pedestrians using maximum entropy inverse optimal control. The advantage of this modeling approach is the generality of its learned cost function to changes in the environment and to entirely different environments. We employ the predictions of this model of pedestrian trajectories in a novel incremental planner and quantitatively show the improvement in hindrance-sensitive robot trajectory planning provided by our approach.

IROS Conference 2007 Conference Paper

Potential negative obstacle detection by occlusion labeling

• Nicholas Heckman
• Jean-François Lalonde
• Nicolas Vandapel
• Martial Hebert

In this paper, we present an approach for potential negative obstacle detection, based on missing data interpretation that extends traditional techniques driven by data only, which capture the occupancy of the scene. The approach is decomposed into three steps: three-dimensional (3D) data accumulation and low level classification, 3D occluder propagation, and context-based occlusion labeling. The approach is validated using logged laser data collected in various outdoor natural terrains and also demonstrated live on-board the Demo-III experimental unmanned vehicle (XUV).

ICML Conference 2006 Conference Paper

Efficient MAP approximation for dense energy functions

• Marius Leordeanu
• Martial Hebert

IROS Conference 2006 Conference Paper

Opportunistic Use of Vision to Push Back the Path-Planning Horizon

• Bart C. Nabbe
• Derek Hoiem
• Alexei A. Efros
• Martial Hebert

Mobile robots need maps or other forms of geometric information about the environment to navigate. The mobility sensors (LADAR, stereo, etc.) on these robotic vehicles can however populate these maps only up to a distance of a few tens of meters. A navigation system has no knowledge about the world beyond this sensing horizon. As a result, path planners that rely only on this knowledge are unable to anticipate obstacles sufficiently early and have no choice but to resort to an inefficient local obstacle avoidance behavior. However, recent developments in the computer vision community allows us to collect geometric information about the environment far beyond this sensing horizon. The coarse 3D geometric estimation that can be recovered is derived from an appearance-based model. That uses a multiple-hypothesis framework to robustly estimate scene structure from a single image and estimating confidences for each geometric label. This 3D geometric estimation is used with a previously presented navigation strategy that reasons about sensor constraints and plans for measurements while navigating towards the goal. The validity of the sensing method and navigation strategy is supported by results from simulations as well as field experiments with a real robotic platform. These results also show that significant reduction in path length can be achieved by using this framework

ICRA Conference 2005 Conference Paper

Analysis and Removal of Artifacts in 3-D LADAR Data

• John Tuley
• Nicolas Vandapel
• Martial Hebert

Errors in laser based range measurements can be divided into two categories: intrinsic sensor errors (range drift with temperature, systematic and random errors), and errors due to the interaction of the laser beam with the environment. The former have traditionally received attention and can be modeled. The latter in contrast have long been observed but not well characterized. We propose to do so in this paper. In addition, we present a sensor independent method to remove such artifacts. The objective is to improve the overall quality of 3-D scene reconstruction to perform terrain classification of scenes with vegetation.

ICRA Conference 2004 Conference Paper

Classifier Fusion for Outdoor Obstacle Detection

• Cristian Dima
• Nicolas Vandapel
• Martial Hebert

This work describes an approach for using several levels of data fusion in the domain of autonomous off-road navigation. We are focusing on outdoor obstacle detection, and we present techniques that leverage on data fusion and machine learning for increasing the reliability of obstacle detection systems. We are combining color and infrared (IR) imagery with range information from a laser range finder. We show that in addition to fusing data at the pixel level, performing high level classifier fusion is beneficial in our domain. Our general approach is to use machine learning techniques for automatically deriving effective models of the classes of interest (obstacle and non-obstacle for example). We train classifiers on different subsets of the features we extract from our sensor suite and show how different classifier fusion schemes can be applied for obtaining a multiple classifier system that is more robust than any of the classifiers presented as input. We present experimental results we obtained on data collected with both the experimental unmanned vehicle (XUV) and a CMU developed robotic tractor.

ICRA Conference 2004 Conference Paper

Enabling Learning from Large Datasets: Applying Active Learning to Mobile Robotics

• Cristian Dima
• Martial Hebert
• Anthony Stentz

Autonomous navigation in outdoor, off-road environments requires solving complex classification problems. Obstacle detection, road following and terrain classification are examples of tasks which have been successfully approached using supervised machine learning techniques for classification. Large amounts of training data are usually necessary in order to achieve satisfactory generalization. In such cases, manually labeling data becomes an expensive and tedious process. This work describes a method for reducing the amount of data that needs to be presented to a human trainer. The algorithm relies on kernel density estimation in order to identify "interesting" scenes in a dataset. Our method does not require any interaction with a human expert for selecting the images, and only minimal amounts of tuning are necessary. We demonstrate its effectiveness in several experiments using data collected with two different vehicles. We first show that our method automatically selects those scenes from a large dataset that a person would consider "important" for classification tasks. Secondly, we show that by labeling only few of the images selected by our method, we obtain classification performance that is comparable to the one reached after labeling hundreds of images from the same dataset.

IROS Conference 2004 Conference Paper

Finding organized structures in 3-D ladar data

• Nicolas Vandapel
• Martial Hebert

In this paper, we address the problem of finding organized thin structures in three-dimensional (3-D) data. Linear and planar structures segmentation received much attention but thin structures organized in complex patterns remain a challenge for segmentation algorithms. We are interested especially in the problems posed by repetitive and symmetric structures acquired with a laser range finder. The method relies on 3-D data projections along specific directions and 2-D histograms comparison. The sensitivity of the classification algorithm to the parameter settings is evaluated and a segmentation method proposed. We illustrate our approach with data from a concertina wire in terrain with vegetation.

ICRA Conference 2004 Conference Paper

Natural Terrain Classification using 3-D Ladar Data

• Nicolas Vandapel
• Daniel F. Huber
• Anuj Kapuria
• Martial Hebert

Because of the difficulty of interpreting laser data in a meaningful way, safe navigation in vegetated terrain is still a daunting challenge. In this paper, we focus on the segmentation of ladar data using local 3-D point statistics into three classes: clutter to capture grass and tree canopy, linear to capture thin objects like wires or tree branches, and finally surface to capture solid objects like ground terrain surface, rocks or tree trunks. We present the details of the method proposed, the modifications we made to implement it on-board an autonomous ground vehicle. Finally, we present results from field tests using this rover and results produced from different stationary laser sensors.

IROS Conference 2004 Conference Paper

Path planning with hallucinated worlds

• Bart C. Nabbe
• Sanjiv Kumar
• Martial Hebert

We describe an approach that integrates midrange sensing into a dynamic path planning algorithm. The algorithm is based on measuring the reduction in path cost that would be caused by taking a sensor reading from candidate locations. The planner uses this measure in order to decide where to take the next sensor reading. Ideally, one would like to evaluate a path based on a map that is as close as possible to the true underlying world. In practice, however, the map is only sparsely populated by data derived from sensor readings. A key component of the approach described in this paper is a mechanism to infer (or "hallucinate") more complete maps from sparse sensor readings. We show how this hallucination mechanism is integrated with the planner to produce better estimates of the gain in path cost occurred when taking sensor readings. We show results on a real robot as well as a statistical analysis on a large set of randomly generated path planning problems on elevation maps from real terrain.

NeurIPS Conference 2003 Conference Paper

Discriminative Fields for Modeling Spatial Dependencies in Natural Images

• Sanjiv Kumar
• Martial Hebert

In this paper we present Discriminative Random Fields (DRF), a discrim- inative framework for the classification of natural image regions by incor- porating neighborhood spatial dependencies in the labels as well as the observed data. The proposed model exploits local discriminative models and allows to relax the assumption of conditional independence of the observed data given the labels, commonly used in the Markov Random Field (MRF) framework. The parameters of the DRF model are learned using penalized maximum pseudo-likelihood method. Furthermore, the form of the DRF model allows the MAP inference for binary classifica- tion problems using the graph min-cut algorithms. The performance of the model was verified on the synthetic as well as the real-world images. The DRF model outperforms the MRF model in the experiments.

ICRA Conference 2003 Conference Paper

Fast 3D tracking of non-rigid objects

• Nobuhiro Okada
• Martial Hebert

A fast 3D tracking method of non-rigid objects is proposed. The method consists of fast ICP and modified RPM techniques for range images. By using the techniques, the authors aim real time 3D non-rigid object tracking. The authors developed an actual tracking system and show the effectiveness of the method by some experimental results.

IROS Conference 2003 Conference Paper

Quality assessment of traversability maps from aerial LIDAR data for an unmanned ground vehicle

• Nicolas Vandapel
• Raghavendra Donamukkala
• Martial Hebert

In this paper we address the problem of assessing quantitatively the quality of traversability maps computed from data collected by an airborne laser range finder. Such data is used to plan paths for an unmanned ground vehicle (UGV) prior to the execution of long range traverses. Little attention has been devoted to the problem we address in this paper. We use a unique data set of geodetic control points, real robot navigation data, ground LIDAR (light detection and ranging) data and aerial imagery, collected during a week long demonstration to support our work.

IROS Conference 2003 Conference Paper

Robust extraction of multiple structures from non-uniformly sampled data

• Ranjith Unnikrishnan
• Martial Hebert

The extraction of multiple coherent structures from point clouds is crucial to the problem of scene modeling. While many statistical methods exist for robust estimation from noisy data, they are inadequate for addressing issues of scale, semi-structured clutter, and large point density variation together with the computational restriction of autonomous navigation. This paper extends an approach of nonparametric projection-pursuit based regression to compensate for the non-uniform and directional nature of data sampled in outdoor environments. The proposed algorithm is employed for extraction of planar structures and clutter grouping. Results are shown for scene abstraction of 3D range data in large urban scenes.

IROS Conference 2003 Conference Paper

Toward generating labeled maps from color and range data for robot navigation

• Caroline Pantofaru
• Ranjith Unnikrishnan
• Martial Hebert

This paper addresses the problem of extracting information from range and color data acquired by a mobile robot in urban environments. Our approach extracts geometric structures from clouds of 3-D points and regions from the corresponding color images, labels them based on prior models of the objects expected in the environment - buildings in the current experiments - and combines the two sources of information into a composite labeled map. Ultimately, our goal is to generate maps that are segmented into objects of interest, each of which is labeled by its type, e. g. , building, vegetation, etc. Such a map provides a higher-level representation of the environment than the geometric maps normally used for mobile robot navigation. The techniques presented here are a step toward the automatic construction of such labeled maps.

IROS Conference 2003 Conference Paper

Where and when to look: how to extend the myopic planning horizon

• Bart C. Nabbe
• Martial Hebert

In this paper we describe an approach towards integrating mid-range sensing data into a dynamic path planning algorithm. The key problem, sensing for planning is addressed in the context of outdoor navigation. An algorithmic approach is described towards solving these problems and both simulation results and initial experimental results for outdoor navigation using wide baseline stereo data are presented.

ICRA Conference 2002 Conference Paper

Robust Tracking and Structure from Motion with Sample Based Uncertainty Representation

• Peng Chang
• Martial Hebert

Geometric reconstruction of the environment from images is critical in autonomous mapping and robot navigation. Geometric reconstruction involves feature tracking, i. e. , locating corresponding image features in consecutive images, and structure from motion (SFM), i. e. , recovering the 3D structure of the environment from a set of correspondences between images. Although algorithms for feature tracking and structure from motion are well-established, their use in practical mobile robot applications is still difficult because of occluded features, non-smooth motion between frames, and ambiguous patterns in images. We show how a sampling-based representation can be used in place of the traditional Gaussian representation of uncertainty. We show how sampling can be used for both feature tracking and SFM and we show how they are combined in this framework. The approach is exercised in the context of a mobile robot navigating through an outdoor environment with an omnidirectional camera.

ICRA Conference 2002 Conference Paper

Toward Practical Cooperative Stereo for Robotic Colonies

• Bart C. Nabbe
• Martial Hebert

In this paper we describe an approach towards cooperative stereo. The key problems, significant different views, different scale and occlusion, will be addressed in the context of a distributed robotic system. An algorithmic approach is described towards solving these problems and results on real scenes are presented.

ICRA Conference 2000 Conference Paper

3-D Map Reconstruction from Range Data

• Daniel F. Huber
• Owen T. Carmichael
• Martial Hebert

We present techniques for building models of complex environments from range data gathered at multiple viewpoints. The challenges in this problem are: the matching of unregistered views without prior knowledge of pose, the use of very large data sets, and the manipulation of data sets of different resolutions and from different sensors. Our approach is unique in that no prior knowledge of the relative viewpoints is needed in order to register the data. We show results in building maps of interior environment from range finding data, building large terrain maps from ground-based and from aerial data, and from an operational for mapping from stereo data for hazardous environment characterization. The paper summarizes the major results obtained so far in this area.

ICRA Conference 2000 Conference Paper

Active and Passive Range Sensing for Robotics

• Martial Hebert

In this paper, we present a brief survey of the technologies currently available for range sensing, of their use in robotics applications, and of emerging technologies for future systems. The paper is organized by type of sensing: laser range finders, triangulation range finders, and passive stereo. A separate section focuses on current development in the area of nonscanning sensors, a critical area to achieve range sensing performance comparable to that of conventional cameras. The presentation of the different technologies is based on many recent examples from robotics research.

ICRA Conference 2000 Conference Paper

Invariant Filtering for Simultaneous Localization and Mapping

• Matthew C. Deans
• Martial Hebert

This paper presents an algorithm for simultaneous localization and map building for a mobile robot moving in an unknown environment. The robot can measure only the bearings to identifiable targets and its own relative motion. The approach is to recursively estimate features of the environment which are invariant to the robot pose in order to decouple the pose error from the map error. The highly nonlinear nature of this problem requires more explicit reasoning about the spatial relationships between landmarks and between the robot and landmarks than those used in previous methods.

ICRA Conference 1999 Conference Paper

3-D Cueing: A Data Filter for Object Recognition

• Owen T. Carmichael
• Martial Hebert

Presents a method for quickly filtering range data points to make object recognition in large 3D data sets feasible. The general approach, called "3D cueing", uses shape signatures from object models as the basis for a fast, probabilistic classification system which rates scene points in terms of their likelihood of belonging to a model. This algorithm which could be used as a front-end for any traditional 3D matching technique, is demonstrated using several models and cluttered scenes in which the model occupies between 1% and 50% of the data points.

IROS Conference 1999 Conference Paper

A new approach to 3-D terrain mapping

• Daniel F. Huber
• Martial Hebert

We discuss the problem of building larger high-resolution three-dimensional representations of unstructured terrain using terrestrial range sensors, which operate at the scale of meters to hundreds of meters. Issues specific to this sensing modality include widely varying resolution, absence of reliably detectable features, and very large data sets. We have developed a map building algorithm that registers and integrates sequences of range images, and we demonstrate its capabilities by building large terrain maps (260/spl times/166 meters) using ground-based and low-altitude terrestrial range sensors.

ICRA Conference 1998 Conference Paper

Active Laser Radar for High Performance Measurements

• John A. Hancock
• Dirk Langer
• Martial Hebert
• Ryan Sullivan
• Darin Ingimarson
• Eric Hoffmann
• Markus Mettenleiter
• Christoph Fröhlich

Laser scanners, or laser radars (ladar), have been used for a number of years for mobile robot navigation and inspection tasks. Although previous scanners were sufficient for low speed applications, they often did not have the range or angular resolution necessary for mapping at the long distances. Many also did not provide an ample field of view with high accuracy and high precision. In this paper we will present the development of state-of-the-art, high speed, high accuracy, 3D laser radar technology. This work has been a joint effort between CMU and K2T and Z+F. The scanner mechanism provides an unobstructed 360/spl deg/ horizontal field of view, and a 70/spl deg/ vertical field of view. Resolution of the scanner is variable with a maximum resolution of approximately 0. 06 degrees per pixel in both azimuth and elevation. The laser is amplitude-modulated, continuous-wave with an ambiguity interval of 52 m, a range resolution of 1. 6 mm, and a maximum pixel rate of 625 kHz. This paper will focus on the design and performance of the laser radar and will discuss several potential applications for the technology. It reports on performance data of the system including noise, drift over time, precision, and accuracy with measurements. Influences of ambient light, surface material of the target and ambient temperature for range accuracy are discussed. Example data of applications will be shown and improvements will also be discussed.

ICRA Conference 1998 Conference Paper

Experiments in Autonomous Driving with Concurrent Goals and Multiple Vehicles

• Barry Brumitt
• Martial Hebert

In this paper we report on experiments with a system for autonomously driving two vehicles based on complex mission specifications. We show that the system is able to plan local paths in obstacle fields based on sensor data, to plan and update global paths to goals based on frequent obstacle map updates, and to modify mission execution, e. g. , the ordering of the goals, based on the updated paths to the goals. Two recently developed sensors are used for obstacle detection: a high-speed laser rangefinder and a video-rate stereo system. An updated version of a dynamic path planner D* is used for online computation of routes. A new mission planning and execution monitoring tool, GRAMMPS, is used for managing the allocation and ordering of goals between vehicles. We report on experiments conducted in an outdoor test site with two HMMWVs. Implementation details and performance analysis, including failure modes, are described based on a series of twelve experiments, each over 1/2 km distance with up to nine goals. This system is the first multivehicle and multigoal system to be demonstrated in real, natural environments with this degree of generality. The work reported here includes a number of results not previously published, including the use of a real-time stereo machine, a high performance laser rangefinder and the GRAMMPS planning system.

IROS Conference 1998 Conference Paper

Laser intensity-based obstacle detection

• John A. Hancock
• Martial Hebert
• Charles E. Thorpe

We present a novel method for obstacle detection for automated highway environments. Laser range scanners have frequently been used for obstacle detection for mobile robots. Although most laser scanners provide intensity information in addition to range, laser intensity has been ignored by most researchers. We show that laser intensity, on its own, is sufficient (and better) for detecting obstacles at long ranges in mild terrain such as an automated highway.

IROS Conference 1998 Conference Paper

Omni-directional visual servoing for human-robot interaction

• Peng Chang
• Martial Hebert

We describe a visual servoing system developed as a human-robot interface to drive a mobile robot toward any chosen target. An omni-directional camera is used to get the 360 degree of field of view, and an efficient tracking technique is developed to track the target. The use of the omni-directional geometry eliminates, many of the problems common in visual tracking and makes the use of visual servoing a practical alternative for robot-human interaction. The experiments demonstrate that it is an effective and robust way to guide a robot. In particular the experiments show robustness of the tracker to loss of template, vehicle motion, and change in scale and orientation.

IROS Conference 1998 Conference Paper

Unconstrained registration of large 3D point sets for complex model building

• Owen T. Carmichael
• Martial Hebert

We present a method for building models of complex environments from range data gathered at multiple viewpoints. Our approach is unique in that no prior knowledge of the relative positions of the viewpoints is needed in order to register data from them. Furthermore, we present a technique for specification and utilization of so-called "common-sense" constraints on the transformations between views to improve the accuracy and speed of the registration process. Results are shown from our effort to map a 60 m by 20 m multiple-room storage area containing a cluttered array of objects.

IROS Conference 1995 Conference Paper

3-D object modeling and recognition for telerobotic manipulation

• Andrew E. Johnson 0002
• Patrick Leger
• Regis Hoffman
• Martial Hebert
• James Osborn

This paper describes a system that semi-automatically builds a virtual world for remote operations by constructing 3-D models of a robot's work environment. With a minimum of human interaction, planar and quadric surface representations of objects typically found in man-made facilities are generated from laser rangefinder data. The surface representations are used to recognize complex models of objects in the scene. These object models are incorporated into a larger world model that can be viewed and analyzed by the operator, accessed by motion planning and robot safeguarding algorithms, and ultimately used by the operator to command the robot through graphical programming and other high level constructs. Limited operator interaction, combined with assumptions about the robots task environment, make the problem of modeling and recognizing objects tractable and yields a solution that can be readily incorporated into many telerobotic control schemes.

IROS Conference 1995 Conference Paper

A complete navigation system for goal acquisition in unknown environments

• Anthony Stentz
• Martial Hebert

Most autonomous outdoor navigation systems tested on actual robots have centered on local navigation tasks such as avoiding obstacles or following roads. Global navigation has been limited to simple wandering, path tracking, straight-line goal seeking behaviors, or executing a sequence of scripted local behaviors. These capabilities are insufficient for unstructured and unknown environments, where replanning may be needed to account for new information discovered in every sensor image. To address these problems, the authors developed a complete system that integrates local and global navigation. The local system uses a scanning laser rangefinder to detect and avoid obstacles. The global system uses an incremental path planning algorithm to optimally replan the global path for each detected obstacle. A control arbiter steers the robot to achieve the proper balance between safety and goal acquisition. This system was tested on a real robot and successfully drove it 1. 4 kilometers to find a goal given no a priori map of the environment.

IROS Conference 1995 Conference Paper

Experience with rover navigation for lunar-like terrains

• Reid G. Simmons
• Eric Krotkov
• Lonnie Chrisman
• Fábio Gagliardi Cozman
• Richard Goodwin
• Martial Hebert
• Lalitesh Katragadda
• Sven Koenig

Reliable navigation is critical for a lunar rover, both for autonomous traverses and safeguarded remote teleoperation. This paper describes an implemented system that has autonomously driven a prototype wheeled lunar rover over a kilometer in natural, outdoor terrain. The navigation system uses stereo terrain maps to perform local obstacle avoidance, and arbitrates steering recommendations from both the user and the rover. The paper describes the system architecture, each of the major components, and the experimental results to date.

ICRA Conference 1995 Conference Paper

Mapping and Positioning for a Prototype Lunar Rover

• Eric Krotkov
• Martial Hebert

In this paper, we describe practical, effective approaches to outdoor mapping and positioning, and present results from systems implemented for a prototype lunar rover. For mapping, we have developed a binocular head and mounted it on a motion-averaging mast. This head provides images to a normalized correlation matcher, that intelligently selects what part of the image to process (saving time), and subsamples the images (again saving time) without subsampling disparities (which would reduce accuracy). The mapping system has operated successfully during long-duration field exercises, processing streams of thousands of images. The positioning system employs encoders, inclinometers, a compass, and a turn-rate sensor to maintain the position and orientation of the rover as it traverses. The system succeeds in the face of significant sensor noise by virtue of sensor modelling, plus extensive filtering and data screening.

ICRA Conference 1994 Conference Paper

An Integrated System for Autonomous Off-Road Navigation

• Dirk Langer
• Julio K. Rosenblatt
• Martial Hebert

In this paper, we report on experiments with a core system for autonomous navigation in outdoor natural terrain. The system consists of three parts: a perception module which processes range images to identify untraversable regions of the terrain, a local map management module which maintains a representation of the environment in the vicinity of the vehicle, and a planning module which issues commands to the vehicle controller. Our approach uses reactive planning for generating commands to drive the vehicle along with "early traversability evaluation, " in which the perception module decides which parts of the terrain are traversable as soon as a new image is taken. We argue that our approach leads to a robust and efficient navigation system. We illustrate our approach by an experiment in which a vehicle travelled autonomously for one kilometer through unmapped cross-country terrain. >

ICRA Conference 1994 Conference Paper

Building 3-D Models from Unregistered Range Images

• Ken Higuchi
• Martial Hebert
• Katsushi Ikeuchi

The authors describe an approach to building a three-dimensional model from a set of range images. The authors' goal is to build models of free-form surfaces obtained from arbitrary viewing directions, with no initial estimate of the relative viewing directions. The approach is based on building discrete meshes representing the surfaces observed in each of the range images, to map each of the meshes to a spherical image, and to compute the transformations between the views by matching the spherical images. The meshes are built using an iterative fitting algorithm previously developed; the spherical images are built by matching the nodes of the surface meshes to the nodes of a reference mesh on the unit sphere and by storing a measure of curvature at every node. The authors describe the algorithms used for building such models from range images and for matching them. The authors give results obtained using range images of complex objects. >

ICRA Conference 1994 Conference Paper

Pixel-Based Range Processing for Autonomous Driving

• Martial Hebert

We describe a pixel-based approach to range processing for obstacle detection and autonomous driving as an alternative to the traditional image- or map-based approaches. The pixel-based approach eliminates the delays due to image acquisition and map building and permits the integration of traversability evaluation and path generation into a single module without the latency involved in distributed systems. We describe the algorithm used for updating a local map using individual range pixels, for detecting obstacles on the fly, and for generating steering commands. We illustrate the performance of the algorithm using an implementation of a cross-country driving system with a scanning laser range finder. >

ICRA Conference 1994 Conference Paper

Real-Thme 3-D Pose Estimation Using a High-Speed Range Sensor

• David A. Simon
• Martial Hebert
• Takeo Kanade

This paper describes a system which can perform full 3-D pose estimation of a single arbitrarily shaped, rigid object at rates up to 10 Hz. A triangular mesh model of the object to be tracked is generated offline using conventional range sensors. Real-time range data of the object is sensed by the CMU high speed VLSI range sensor. Pose estimation is performed by registering the real-time range data to the triangular mesh model using an enhanced implementation of the Iterative Closest Point (ICP) Algorithm introduced by Besl and McKay (1992). The method does not require explicit feature extraction or specification of correspondence. Pose estimation accuracies of the order of 1% of the object size in translation, and 1 degree in rotation have been measured. >

IROS Conference 1992 Conference Paper

Task Oriented Vision

• Katsushi Ikeuchi
• Martial Hebert

IROS Conference 1991 Conference Paper

3-D measurements from imaging laser radars: how good are they?

• Martial Hebert
• Eric Krotkov

The authors analyze a class of imaging range finders-amplitude-modulated continuous-wave laser radars-in the context of computer vision and robotics. The analysis develops measurement models from the fundamental principles of laser radar operation, and identifies the nature and cause of key problems that plague measurements from this class of sensors. They classify the problems as fundamental (e. g. related to the signal-to-noise ratio), as architectural (e. g. limited by encoding distance by angles (0. 2 pi )), and as artifacts of particular hardware implementations (e. g. insufficient temperature compensation). Experimental results from two different scanning laser range finders designed for autonomous navigation illustrate and support the analysis.

ICRA Conference 1991 Conference Paper

A three-finger gripper for manipulation in unstructured environments

• C. Francois
• Katsushi Ikeuchi
• Martial Hebert

A gripper is described for manipulation in natural, unstructured environments. The specific manipulation task is to pick up surface material such as pebbles or small rocks in a natural terrain. The application is to give autonomous sampling capabilities to an autonomous vehicle for planetary exploration. The authors describe the task analysis process that led to the selection of a configuration with three soft fingers. They carry out a complete analysis of the stability of a grasp for this gripper including an analysis of the deformation of the fingers at the points of contact. The implementation of a grasp selection algorithm is described, and results on three-dimensional representations of objects computed from range data are presented. >

ICRA Conference 1991 Conference Paper

Building qualitative elevation maps from side scan sonar data for autonomous underwater navigation

• Dirk Langer
• Martial Hebert

Deriving a terrain model from sensor data is an important task for the autonomous navigation of a mobile robot. An approach is presented for autonomous underwater vehicles using a side scan sonar system. Some general aspects of the type of data and filtering techniques to improve it are discussed. An estimated bottom contour is derived using a geometric reflection model and information about shadows and highlights. Several techniques of surface reconstruction and their limitations are presented. A method is presented for feature extraction which is important for future data matching/fusion procedures. >

IROS Conference 1991 Conference Paper

Trajectory generation with curvature constraint based on energy minimization

• Herve Delingette
• Martial Hebert
• Katsushi Ikeuchi

The trajectory generation problem for mobile robots consists in providing a set of trajectories that are 'smooth' and meet certain boundary conditions. The authors present a method to generate curvature continuous trajectories for which the curvature profile is a polynomial function of arc length. An algorithm based on the deformation of a curve by energy minimization allows one to solve general geometric constraints which was not possible by previous methods. Furthermore, it is able to take into account the limitation of radius of curvature of the robot by controlling the extrema of curvature along the path. >

ICRA Conference 1989 Conference Paper

Building and navigating maps of road scenes using an active sensor

• Martial Hebert

The author presents algorithms for building maps of road scenes using an active range and reflectance sensor and for using the maps to traverse a portion of the world already explored. He describes some advantages of an active sensor, namely, that it is independent of the illumination conditions, does not require complex calibration in order to transform observed features to the vehicle's reflectance frame, and provides 3-D terrain models as well as road models. Using this map built from sensor data facilitates navigation in two respects: the vehicle may navigate faster, since less perception processing is necessary, and the vehicle may follow a more accurate path, since the navigation system does not rely entirely on inaccurate visual data. The author presents a complete system that includes road-following, map-building, and map-based navigation using the ERIM laser rangefinder. Experimental results are presented. >

IROS Conference 1989 Conference Paper

Methods for Identifying Footfall Positions for a Legged Robot

• Claude Caillas
• Martial Hebert
• Eric Krotkov
• In-So Kweon
• Takeo Kanade

ICRA Conference 1989 Conference Paper

Terrain mapping for a roving planetary explorer

• Martial Hebert
• Claude Caillas
• Eric Krotkov
• In-So Kweon
• Takeo Kanade

The authors are prototyping a legged vehicle, the Ambler, for an exploratory mission on another planet, conceivably Mars, where it is to traverse uncharted areas and collect material samples. They describe how the rover can construct from range imagery a geometric terrain representation, i. e. , elevation map that includes uncertainty, unknown areas, and local features. First, they present an algorithm for constructing an elevation map from a single range image. By virtue of working in spherical-polar space, the algorithm is independent of the desired map resolution and the orientation of the sensor, unlike algorithms that work in Cartesian space. Secondly, the authors present a two-stage matching technique (feature matching followed by iconic matching) that identifies the transformation T corresponding to the vehicle displacement between two viewing positions. Thirdly, to support legged locomotion over rough terrain, they describe methods for evaluating regions of the constructed elevation maps as footholds. >

ICRA Conference 1987 Conference Paper

An architecture and two cases in range-based modeling and planning

• William Whittaker
• George Turkiyyah
• Martial Hebert

This paper presents a framework for autonomous robots that reason from range data. We argue for spatial reasoning as a basic cognition mode for robots operating in unpredictable work environments and present a three-level architecture for modeling and planning from range data. Two implementations, robotic excavation with sonar ranging and mine navigation with laser ranging, illustrate the techniques and provide two experiences to evaluate the architecture.

ICRA Conference 1986 Conference Paper

Outdoor scene analysis using range data

• Martial Hebert

This paper describes techniques for outdoor scene analysis using range data. The purpose of these techniques is to build a 3-D representation of the environment of an mobile robot equipped with a range sensor. Algorithms are presented for scene segmentation, object detection, map building, and object recognition. We present results obtained in an outdoor navigation environment in which a laser range finder is mounted on a vehicle. These results have been successfully applied to the problem of path planning through obstacles.