James M. Rehg

ICLR Conference 2025 Conference Paper

RelCon: Relative Contrastive Learning for a Motion Foundation Model for Wearable Data

• Maxwell A. Xu
• Jaya Narain
• Gregory Darnell
• Haraldur Tómas Hallgrímsson
• Hyewon Jeong
• Darren Forde
• Richard Andres Fineman
• Karthik Jayaraman Raghuram

We present RelCon, a novel self-supervised Relative Contrastive learning approach for training a motion foundation model from wearable accelerometry sensors. First, a learnable distance measure is trained to capture motif similarity and domain-specific semantic information such as rotation invariance. Then, the learned distance provides a measurement of semantic similarity between a pair of accelerometry time-series, which we use to train our foundation model to model relative relationships across time and across subjects. The foundation model is trained on 1 billion segments from 87,376 participants, and achieves strong performance across multiple downstream tasks, including human activity recognition and gait metric regression. To our knowledge, we are the first to show the generalizability of a foundation model with motion data from wearables across distinct evaluation tasks.

ICLR Conference 2024 Conference Paper

REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning

• Maxwell A. Xu
• Alexander Moreno
• Hui Wei
• Benjamin M. Marlin
• James M. Rehg

The success of self-supervised contrastive learning hinges on identifying positive data pairs, such that when they are pushed together in embedding space, the space encodes useful information for subsequent downstream tasks. Constructing positive pairs is non-trivial as the pairing must be similar enough to reflect a shared semantic meaning, but different enough to capture within-class variation. Classical approaches in vision use augmentations to exploit well-established invariances to construct positive pairs, but invariances in the time-series domain are much less obvious. In our work, we propose a novel method of using a learned measure for identifying positive pairs. Our Retrieval-Based Reconstruction (REBAR) measure measures the similarity between two sequences as the reconstruction error that results from reconstructing one sequence with retrieved information from the other. Then, if the two sequences have high REBAR similarity, we label them as a positive pair. Through validation experiments, we show that the REBAR error is a predictor of mutual class membership. Once integrated into a contrastive learning framework, our REBAR method learns an embedding that achieves state-of-the-art performance on downstream tasks across various modalities.

NeurIPS Conference 2023 Conference Paper

Low-shot Object Learning with Mutual Exclusivity Bias

• Anh Thai
• Ahmad Humayun
• Stefan Stojanov
• Zixuan Huang
• Bikram Boote
• James M. Rehg

This paper introduces Low-shot Object Learning with Mutual Exclusivity Bias (LSME), the first computational framing of mutual exclusivity bias, a phenomenon commonly observed in infants during word learning. We provide a novel dataset, comprehensive baselines, and a SOTA method to enable the ML community to tackle this challenging learning task. The goal of LSME is to analyze an RGB image of a scene containing multiple objects and correctly associate a previously-unknown object instance with a provided category label. This association is then used to perform low-shot learning to test category generalization. We provide a data generation pipeline for the LSME problem and conduct a thorough analysis of the factors that contribute to its difficulty. Additionally, we evaluate the performance of multiple baselines, including state-of-the-art foundation models. Finally, we present a baseline approach that outperforms state-of-the-art models in terms of low-shot accuracy. Code and data are available at https: //github. com/rehg-lab/LSME.

AAMAS Conference 2023 Conference Paper

Which way is 'right'? : Uncovering limitations of Vision-and-Language Navigation Models

• Meera Hahn
• Amit Raj
• James M. Rehg

The challenging task of Vision-and-Language Navigation (VLN) requires embodied agents to follow natural language instructions to reach a goal location or object (e. g. ‘walk down the hallway and turn left at the piano’). For agents to complete this task successfully, they must be able to ground objects referenced into the instruction (e. g. ‘piano’) into the visual scene as well as ground directional phrases (e. g. ‘turn left’) into actions. In this work we ask the following question – to what degree are spatial and directional language cues informing the navigation model’s decisions? We propose a series of simple masking experiments to inspect the model’s reliance on different parts of the instruction. Surprisingly we uncover that certain top performing models rely only on the noun tokens of the instructions.

NeurIPS Conference 2022 Conference Paper

Kernel Multimodal Continuous Attention

• Alexander Moreno
• Zhenke Wu
• Supriya Nagesh
• Walter Dempsey
• James M. Rehg

Attention mechanisms take an expectation of a data representation with respect to probability weights. Recently, (Martins et al. 2020, 2021) proposed continuous attention mechanisms, focusing on unimodal attention densities from the exponential and deformed exponential families: the latter has sparse support. (Farinhas et al 2021) extended this to to multimodality via Gaussian mixture attention densities. In this paper, we extend this to kernel exponential families (Canu and Smola 2006) and our new sparse counterpart, kernel deformed exponential families. Theoretically, we show new existence results for both kernel exponential and deformed exponential families, and that the deformed case has similar approximation capabilities to kernel exponential families. Lacking closed form expressions for the context vector, we use numerical integration: we show exponential convergence for both kernel exponential and deformed exponential families. Experiments show that kernel continuous attention often outperforms unimodal continuous attention, and the sparse variant tends to highlight peaks of time series.

NeurIPS Conference 2022 Conference Paper

Learning Dense Object Descriptors from Multiple Views for Low-shot Category Generalization

• Stefan Stojanov
• Anh Thai
• Zixuan Huang
• James M. Rehg

A hallmark of the deep learning era for computer vision is the successful use of large-scale labeled datasets to train feature representations. This has been done for tasks ranging from object recognition and semantic segmentation to optical flow estimation and novel view synthesis of 3D scenes. In this work, we aim to learn dense discriminative object representations for low-shot category recognition without requiring any category labels. To this end, we propose Deep Object Patch Encodings (DOPE), which can be trained from multiple views of object instances without any category or semantic object part labels. To train DOPE, we assume access to sparse depths, foreground masks and known cameras, to obtain pixel-level correspondences between views of an object, and use this to formulate a self-supervised learning task to learn discriminative object patches. We find that DOPE can directly be used for low-shot classification of novel categories using local-part matching, and is competitive with and outperforms supervised and self-supervised learning baselines.

NeurIPS Conference 2022 Conference Paper

PulseImpute: A Novel Benchmark Task for Pulsative Physiological Signal Imputation

• Maxwell Xu
• Alexander Moreno
• Supriya Nagesh
• Varol Aydemir
• David Wetter
• Santosh Kumar
• James M. Rehg

The promise of Mobile Health (mHealth) is the ability to use wearable sensors to monitor participant physiology at high frequencies during daily life to enable temporally-precise health interventions. However, a major challenge is frequent missing data. Despite a rich imputation literature, existing techniques are ineffective for the pulsative signals which comprise many mHealth applications, and a lack of available datasets has stymied progress. We address this gap with PulseImpute, the first large-scale pulsative signal imputation challenge which includes realistic mHealth missingness models, an extensive set of baselines, and clinically-relevant downstream tasks. Our baseline models include a novel transformer-based architecture designed to exploit the structure of pulsative signals. We hope that PulseImpute will enable the ML community to tackle this important and challenging task.

ICRA Conference 2021 Conference Paper

Approximate Inverse Reinforcement Learning from Vision-based Imitation Learning

• Keuntaek Lee
• Bogdan I. Vlahov
• Jason Gibson
• James M. Rehg
• Evangelos A. Theodorou

In this work, we present a method for obtaining an implicit objective function for vision-based navigation. The proposed methodology relies on Imitation Learning, Model Predictive Control (MPC), and an interpretation technique used in Deep Neural Networks. We use Imitation Learning as a means to do Inverse Reinforcement Learning in order to create an approximate cost function generator for a visual navigation challenge. The resulting cost function, the costmap, is used in conjunction with MPC for real-time control and outperforms other state-of-the-art costmap generators in novel environments. The proposed process allows for simple training and robustness to out-of-sample data. We apply our method to the task of vision-based autonomous driving in multiple real and simulated environments and show its generalizability. Supplementary video: https://youtu.be/WyJfT5lc0aQ

NeurIPS Conference 2021 Conference Paper

No RL, No Simulation: Learning to Navigate without Navigating

• Meera Hahn
• Devendra Singh Chaplot
• Shubham Tulsiani
• Mustafa Mukadam
• James M. Rehg
• Abhinav Gupta

Most prior methods for learning navigation policies require access to simulation environments, as they need online policy interaction and rely on ground-truth maps for rewards. However, building simulators is expensive (requires manual effort for each and every scene) and creates challenges in transferring learned policies to robotic platforms in the real-world, due to the sim-to-real domain gap. In this paper, we pose a simple question: Do we really need active interaction, ground-truth maps or even reinforcement-learning (RL) in order to solve the image-goal navigation task? We propose a self-supervised approach to learn to navigate from only passive videos of roaming. Our approach, No RL, No Simulator (NRNS), is simple and scalable, yet highly effective. NRNS outperforms RL-based formulations by a significant margin. We present NRNS as a strong baseline for any future image-based navigation tasks that use RL or Simulation.

NeurIPS Conference 2020 Conference Paper

A Robust Functional EM Algorithm for Incomplete Panel Count Data

• Alexander Moreno
• Zhenke Wu
• Jamie Roslyn Yap
• Cho Lam
• David Wetter
• Inbal Nahum-Shani
• Walter Dempsey
• James M. Rehg

Panel count data describes aggregated counts of recurrent events observed at discrete time points. To understand dynamics of health behaviors and predict future negative events, the field of quantitative behavioral research has evolved to increasingly rely upon panel count data collected via multiple self reports, for example, about frequencies of smoking using in-the-moment surveys on mobile devices. However, missing reports are common and present a major barrier to downstream statistical learning. As a first step, under a missing completely at random assumption (MCAR), we propose a simple yet widely applicable functional EM algorithm to estimate the counting process mean function, which is of central interest to behavioral scientists. The proposed approach wraps several popular panel count inference methods, seamlessly deals with incomplete counts and is robust to misspecification of the Poisson process assumption. Theoretical analysis of the proposed algorithm provides finite-sample guarantees by extending parametric EM theory to the general non-parametric setting. We illustrate the utility of the proposed algorithm through numerical experiments and an analysis of smoking cessation data. We also discuss useful extensions to address deviations from the MCAR assumption and covariate effects.

JBHI Journal 2020 Journal Article

Detecting Suspected Pump Thrombosis in Left Ventricular Assist Devices via Acoustic Analysis

• Beren Semiz
• Sinan Hersek
• Maziyar Baran Pouyan
• Cynthia Partida
• Leticia Blazquez-Arroyo
• Van Selby
• Georg Wieselthaler
• James M. Rehg

Objective: Left ventricular assist devices (LVADs) fail in up to 10% of patients due to the development of pump thrombosis. Remote monitoring of patients with LVADs can enable early detection and, subsequently, treatment and prevention of pump thrombosis. We assessed whether acoustical signals measured on the chest of patients with LVADs, combined with machine learning algorithms, can be used for detecting pump thrombosis. Methods: 13 centrifugal pump (HVAD) recipients were enrolled in the study. When hospitalized for suspected pump thrombosis, clinical data and acoustical recordings were obtained at admission, prior to and after administration of thrombolytic therapy, and every 24 hours until laboratory and pump parameters normalized. First, we selected the most important features among our feature set using LDH-based correlation analysis. Then using these features, we trained a logistic regression model and determined our decision threshold to differentiate between thrombosis and non-thrombosis episodes. Results: Accuracy, sensitivity and precision were calculated to be 88. 9%, 90. 9% and 83. 3%, respectively. When tested on the post-thrombolysis data, our algorithm suggested possible pump abnormalities that were not identified by the reference pump power or biomarker abnormalities. Significance: We showed that the acoustical signatures of LVADs can be an index of mechanical deterioration and, when combined with machine learning algorithms, provide clinical decision support regarding the presence of pump thrombosis.

ICRA Conference 2018 Conference Paper

Best Response Model Predictive Control for Agile Interactions Between Autonomous Ground Vehicles

• Grady Williams
• Brian Goldfain
• Paul Drews
• James M. Rehg
• Evangelos A. Theodorou

We introduce an algorithm for autonomous control of multiple fast ground vehicles operating in close proximity to each other. The algorithm is based on a combination of the game theoretic notion of iterated best response, and an information theoretic model predictive control algorithm designed for non-linear stochastic systems. We test the algorithm on two one-fifth scale AutoRally platforms traveling at speeds upwards of 8 meters per second, while maintaining a following distance of under two meters from bumper-to-bumper.

ICML Conference 2018 Conference Paper

Towards Black-box Iterative Machine Teaching

• Weiyang Liu
• Bo Dai 0001
• Xingguo Li
• Zhen Liu 0019
• James M. Rehg
• Le Song

In this paper, we make an important step towards the black-box machine teaching by considering the cross-space machine teaching, where the teacher and the learner use different feature representations and the teacher can not fully observe the learner’s model. In such scenario, we study how the teacher is still able to teach the learner to achieve faster convergence rate than the traditional passive learning. We propose an active teacher model that can actively query the learner (i. e. , make the learner take exams) for estimating the learner’s status and provably guide the learner to achieve faster convergence. The sample complexities for both teaching and query are provided. In the experiments, we compare the proposed active teacher with the omniscient teacher and verify the effectiveness of the active teacher model.

ICRA Conference 2017 Conference Paper

Information theoretic MPC for model-based reinforcement learning

• Grady Williams
• Nolan Wagener
• Brian Goldfain
• Paul Drews
• James M. Rehg
• Byron Boots
• Evangelos A. Theodorou

We introduce an information theoretic model predictive control (MPC) algorithm capable of handling complex cost criteria and general nonlinear dynamics. The generality of the approach makes it possible to use multi-layer neural networks as dynamics models, which we incorporate into our MPC algorithm in order to solve model-based reinforcement learning tasks. We test the algorithm in simulation on a cart-pole swing up and quadrotor navigation task, as well as on actual hardware in an aggressive driving task. Empirical results demonstrate that the algorithm is capable of achieving a high level of performance and does so only utilizing data collected from the system.

ICML Conference 2017 Conference Paper

iSurvive: An Interpretable, Event-time Prediction Model for mHealth

• Walter H. Dempsey
• Alexander Moreno
• Christy K. Scott
• Michael L. Dennis
• David H. Gustafson 0001
• Susan A. Murphy
• James M. Rehg

An important mobile health (mHealth) task is the use of multimodal data, such as sensor streams and self-report, to construct interpretable time-to-event predictions of, for example, lapse to alcohol or illicit drug use. Interpretability of the prediction model is important for acceptance and adoption by domain scientists, enabling model outputs and parameters to inform theory and guide intervention design. Temporal latent state models are therefore attractive, and so we adopt the continuous time hidden Markov model (CT-HMM) due to its ability to describe irregular arrival times of event data. Standard CT-HMMs, however, are not specialized for predicting the time to a future event, the key variable for mHealth interventions. Also, standard emission models lack a sufficiently rich structure to describe multimodal data and incorporate domain knowledge. We present iSurvive, an extension of classical survival analysis to a CT-HMM. We present a parameter learning method for GLM emissions and survival model fitting, and present promising results on both synthetic data and an mHealth drug use dataset.

ICML Conference 2017 Conference Paper

Iterative Machine Teaching

• Weiyang Liu
• Bo Dai 0001
• Ahmad Humayun
• Charlene Tay
• Chen Yu 0001
• Linda B. Smith
• James M. Rehg
• Le Song

In this paper, we consider the problem of machine teaching, the inverse problem of machine learning. Different from traditional machine teaching which views the learners as batch algorithms, we study a new paradigm where the learner uses an iterative algorithm and a teacher can feed examples sequentially and intelligently based on the current performance of the learner. We show that the teaching complexity in the iterative case is very different from that in the batch case. Instead of constructing a minimal training set for learners, our iterative machine teaching focuses on achieving fast convergence in the learner model. Depending on the level of information the teacher has from the learner model, we design teaching algorithms which can provably reduce the number of teaching examples and achieve faster convergence than learning without teachers. We also validate our theoretical findings with extensive experiments on different data distribution and real image datasets.

ICRA Conference 2016 Conference Paper

Aggressive driving with model predictive path integral control

• Grady Williams
• Paul Drews
• Brian Goldfain
• James M. Rehg
• Evangelos A. Theodorou

In this paper we present a model predictive control algorithm designed for optimizing non-linear systems subject to complex cost criteria. The algorithm is based on a stochastic optimal control framework using a fundamental relationship between the information theoretic notions of free energy and relative entropy. The optimal controls in this setting take the form of a path integral, which we approximate using an efficient importance sampling scheme. We experimentally verify the algorithm by implementing it on a Graphics Processing Unit (GPU) and apply it to the problem of controlling a fifth-scale Auto-Rally vehicle in an aggressive driving task.

IROS Conference 2015 Conference Paper

Combining tactile sensing and vision for rapid haptic mapping

• Tapomayukh Bhattacharjee
• Ashwin A. Shenoi
• Daehyung Park
• James M. Rehg
• Charles C. Kemp

We consider the problem of enabling a robot to efficiently obtain a dense haptic map of its visible surroundings using the complementary properties of vision and tactile sensing. Our approach assumes that visible surfaces that look similar to one another are likely to have similar haptic properties. We present an iterative algorithm that enables a robot to infer dense haptic labels across visible surfaces when given a color-plus-depth (RGB-D) image along with a sequence of sparse haptic labels representative of what could be obtained via tactile sensing. Our method uses a color-based similarity measure and connected components on color and depth data. We evaluated our method using several publicly available RGBD image datasets with indoor cluttered scenes pertinent to robot manipulation. We analyzed the effects of algorithm parameters and environment variation, specifically the level of clutter and the type of setting, like a shelf, table top, or sink area. In these trials, the visible surface for each object consisted of an average of 8602 pixels, and we provided the algorithm with a sequence of haptically-labeled pixels up to a maximum of 40 times the number of objects in the image. On average, our algorithm correctly assigned haptic labels to 76. 02% of all of the object pixels in the image given this full sequence of labels. We also performed experiments with the humanoid robot DARCI reaching in a cluttered foliage environment while using our algorithm to create a haptic map. Doing so enabled the robot to reach goal locations using a single plan after a single greedy reach, while our previous tactile-only mapping method required 5 or more plans to reach each goal.

ICRA Conference 2015 Conference Paper

Multi-scale perception and path planning on probabilistic obstacle maps

• Florian Hauer 0001
• Abhijit Kundu
• James M. Rehg
• Panagiotis Tsiotras

We present a path-planning algorithm that leverages a multi-scale representation of the environment. The algorithm works in n dimensions. The information of the environment is stored in a tree representing a recursive dyadic partitioning of the search space. The information used by the algorithm is the probability that a node of the tree corresponds to an obstacle in the search space. The complexity of the proposed algorithm is analyzed and its completeness is shown.

IROS Conference 2014 Conference Paper

Learning to reach into the unknown: Selecting initial conditions when reaching in clutter

• Daehyung Park
• Ariel Kapusta
• You Keun Kim
• James M. Rehg
• Charles C. Kemp

Often in highly-cluttered environments, a robot can observe the exterior of the environment with ease, but cannot directly view nor easily infer its detailed internal structure (e. g. , dense foliage or a full refrigerator shelf). We present a data-driven approach that greatly improves a robot's success at reaching to a goal location in the unknown interior of an environment based on observable external properties, such as the category of the clutter and the locations of openings into the clutter (i. e. , apertures). We focus on the problem of selecting a good initial configuration for a manipulator when reaching with a greedy controller. We use density estimation to model the probability of a successful reach given an initial condition and then perform constrained optimization to find an initial condition with the highest estimated probability of success. We evaluate our approach with two simulated robots reaching in clutter, and provide a demonstration with a real PR2 robot reaching to locations through random apertures. In our evaluations, our approach significantly outperformed two alternative approaches when making two consecutive reach attempts to goals in distinct categories of unknown clutter. Our approach only uses sparse readily-apparent features.

ICRA Conference 2013 Conference Paper

Decoupling behavior, perception, and control for autonomous learning of affordances

• Tucker Hermans
• James M. Rehg
• Aaron F. Bobick

A novel behavior representation is introduced that permits a robot to systematically explore the best methods by which to successfully execute an affordance-based behavior for a particular object. The approach decomposes affordance-based behaviors into three components. We first define controllers that specify how to achieve a desired change in object state through changes in the agent's state. For each controller we develop at least one behavior primitive that determines how the controller outputs translate to specific movements of the agent. Additionally we provide multiple perceptual proxies that define the representation of the object that is to be computed as input to the controller during execution. A variety of proxies may be selected for a given controller and a given proxy may provide input for more than one controller. When developing an appropriate affordance-based behavior strategy for a given object, the robot can systematically vary these elements as well as note the impact of additional task variables such as location in the workspace. We demonstrate the approach using a PR2 robot that explores different combinations of controller, behavior primitive, and proxy to perform a push or pull positioning behavior on a selection of household objects, learning which methods best work for each object.

IROS Conference 2012 Conference Paper

Guided pushing for object singulation

• Tucker Hermans
• James M. Rehg
• Aaron F. Bobick

We propose a novel method for a robot to separate and segment objects in a cluttered tabletop environment. The method leverages the fact that external object boundaries produce visible edges within an object cluster. We achieve this singulation of objects by using the robot arm to perform pushing actions specifically selected to test whether particular visible edges correspond to object boundaries. We verify the separation of objects after a push by examining the clusters formed by geometric segmentation of regions residing on the table surface. To avoid explicitly representing and tracking edges across push behaviors we aggregate over all edges in a given orientation by representing the push-history as an orientation histogram. By tracking the history of directions pushed for each object cluster we can build evidence that a cluster cannot be further separated. We present quantitative and qualitative experimental results performed in a real home environment by a mobile manipulator using input from an RGB-D camera mounted on the robot's head. We show that our pushing strategy can more reliably obtain singulation in fewer pushes than an approach, that does not explicitly reason about boundary information.

IROS Conference 2012 Conference Paper

Haptic classification and recognition of objects using a tactile sensing forearm

• Tapomayukh Bhattacharjee
• James M. Rehg
• Charles C. Kemp

In this paper, we demonstrate data-driven inference of mechanical properties of objects using a tactile sensor array (skin) covering a robot's forearm. We focus on the mobility (sliding vs. fixed), compliance (soft vs. hard), and identity of objects in the environment, as this information could be useful for efficient manipulation and search. By using the large surface area of the forearm, a robot could potentially search and map a cluttered volume more efficiently, and be informed by incidental contact during other manipulation tasks. Our approach tracks a contact region on the forearm over time in order to generate time series of select features, such as the maximum force, contact area, and contact motion. We then process and reduce the dimensionality of these time series to generate a feature vector to characterize the contact. Finally, we use the k-nearest neighbor algorithm (k-NN) to classify a new feature vector based on a set of previously collected feature vectors. Our results show a high cross-validation accuracy in both classification of mechanical properties and object recognition. In addition, we analyze the effect of taxel resolution, duration of observation, feature selection, and feature scaling on the classification accuracy.

JMLR Journal 2011 Journal Article

Efficient and Effective Visual Codebook Generation Using Additive Kernels

• Jianxin Wu
• Wei-Chian Tan
• James M. Rehg

Common visual codebook generation methods used in a bag of visual words model, for example, k-means or Gaussian Mixture Model, use the Euclidean distance to cluster features into visual code words. However, most popular visual descriptors are histograms of image measurements. It has been shown that with histogram features, the Histogram Intersection Kernel (HIK) is more effective than the Euclidean distance in supervised learning tasks. In this paper, we demonstrate that HIK can be used in an unsupervised manner to significantly improve the generation of visual codebooks. We propose a histogram kernel k-means algorithm which is easy to implement and runs almost as fast as the standard k-means. The HIK codebooks have consistently higher recognition accuracy over k-means codebooks by 2-4% in several benchmark object and scene recognition data sets. The algorithm is also generalized to arbitrary additive kernels. Its speed is thousands of times faster than a naive implementation of the kernel k-means algorithm. In addition, we propose a one-class SVM formulation to create more effective visual code words. Finally, we show that the standard k-median clustering method can be used for visual codebook generation and can act as a compromise between the HIK / additive kernel and the k-means approaches. [abs] [ pdf ][ bib ] &copy JMLR 2011. ( edit, beta )

ICRA Conference 2011 Conference Paper

Real-time human detection using contour cues

• Jianxin Wu 0001
• Christopher Geyer
• James M. Rehg

A real-time and accurate human detector, C 4, is proposed in this paper. C 4 achieves 20 fps speed and state-of-the-art detection accuracy, using only one processing thread without resorting to special hardwares like GPU. Real-time accurate human detection is made possible by two contributions. First, we show that contour is exactly what we should capture and signs of comparisons among neighboring pixels are the key information to capture contours. Second, we show that the CENTRIST visual descriptor is particularly suitable for human detection, because it encodes the sign information and can implicitly represent the global contour. When CENTRIST and linear classifier are used, we propose a computational method that does not need to explicitly generate feature vectors. It involves no image pre-processing or feature vector normalization, and only requires O(1) steps to test an image patch. C 4 is also friendly to further hardware acceleration. In a robot with embedded 1. 2GHz CPU, we also achieved accurate and 20 fps high speed human detection.

IROS Conference 2009 Conference Paper

Visual Place Categorization: Problem, dataset, and algorithm

• Jianxin Wu 0001
• Henrik I. Christensen
• James M. Rehg

In this paper we describe the problem of Visual Place Categorization (VPC) for mobile robotics, which involves predicting the semantic category of a place from image measurements acquired from an autonomous platform. For example, a robot in an unfamiliar home environment should be able to recognize the functionality of the rooms it visits, such as kitchen, living room, etc. We describe an approach to VPC based on sequential processing of images acquired with a conventional video camera. We identify two key challenges: Dealing with non-characteristic views and integrating restricted-FOV imagery into a holistic prediction. We present a solution to VPC based upon a recently-developed visual feature known as CENTRIST (CENsus TRansform hISTogram). We describe a new dataset for VPC which we have recently collected and are making publicly available. We believe this is the first significant, realistic dataset for the VPC problem. It contains the interiors of six different homes with ground truth labels. We use this dataset to validate our solution approach, achieving promising results.

IROS Conference 2007 Conference Paper

Traversability classification for UGV navigation: a comparison of patch and superpixel representations

• Dongshin Kim 0002
• Sang Min Oh
• James M. Rehg

Robot navigation in complex outdoor terrain can benefit from accurate traversability classification. Appearancebased traversability estimation can provide a long-range sensing capability which complements the traditional use of stereo or LIDAR ranging. In the standard approach to traversability classification, each image frame is decomposed into patches or pixels for further analysis. However, classification at the pixel level is prone to noise and complicates the task of identifying homogeneous regions for navigation. Fixed-sized patches aggregate pixel information, resulting in better noise properties, but they can span multiple distinct image regions, which can degrade the classification performance and make thin obstacles difficult to detect. We address the use of superpixels as the visual primitives for traversability estimation. Superpixels are obtained from an over-segmentation of the image and they aggregate visually homogeneous pixels while respecting natural terrain boundaries. We show that superpixels are superior to patches in classification accuracy and result in more effective navigation in complex terrain environments. Our experimental results include a study of the effect of patch and superpixel size on classification accuracy. We demonstrate that superpixels can be computed on-line on a real robot at a sufficient frame rate to support long-range sensing and planning.

ICRA Conference 2006 Conference Paper

Traversability Classification using Unsupervised on-line Visual Learning for Outdoor Robot Navigation

• Dongshin Kim 0002
• Jie Sun 0004
• Sang Min Oh
• James M. Rehg
• Aaron F. Bobick

Estimating the traversability of terrain in an unstructured outdoor environment is a core functionality for autonomous robot navigation. While general-purpose sensing can be used to identify the existence of terrain features such as vegetation and sloping ground, the traversability of these regions is a complex function of the terrain characteristics and vehicle capabilities, which makes it extremely difficult to characterize a priori. Moreover, it is difficult to find general rules which work for a wide variety of terrain types such as trees, rocks, tall grass, logs, and bushes. As a result, methods which provide traversability estimates based on predefined terrain properties such as height or shape will be unlikely to work reliably in unknown outdoor environments. Our approach is based on the observation that traversability in the most general sense is an affordance which is jointly determined by the vehicle and its environment. We describe a novel on-line learning method which can make accurate predictions of the traversability properties of complex terrain. Our method is based on autonomous training data collection which exploits the robot's experience in navigating its environment to train classifiers without human intervention. This is in contrast to other learning methods in which training data is collected manually. We have implemented and tested our traversability learning method on an unmanned ground vehicle (UGV) and evaluated its performance in several realistic outdoor environments. The experiments quantify the benefit of our on-line traversability learning approach

ICML Conference 2005 Conference Paper

Efficient discriminative learning of Bayesian network classifier via boosted augmented naive Bayes

• Yushi Jing
• Vladimir Pavlovic 0001
• James M. Rehg

ICML Conference 2005 Conference Paper

Linear Asymmetric Classifier for cascade detectors

• Jianxin Wu 0001
• Matthew D. Mullin
• James M. Rehg

ICRA Conference 1991 Conference Paper

Visual tracking with deformation models

• James M. Rehg
• Andrew P. Witkin

A novel solution to the 2-D tracking problem is presented. This solution has two major components: a deformation model that constrains the interpretation of motion, and a set of energy-based match criteria that specify image features to be used in tracking. The separation of the motion model from the match features is an advantage of this approach over previous tracking systems. An implementation of these ideas has been shown to exhibit fast and flexible operation over a wide class of image motions. Experimental results are given for two real-world image sequences. >