Dean Pomerleau

YNIMG Journal 2012 Journal Article

Tracking neural coding of perceptual and semantic features of concrete nouns

• Gustavo Sudre
• Dean Pomerleau
• Mark Palatucci
• Leila Wehbe
• Alona Fyshe
• Riitta Salmelin
• Tom Mitchell

We present a methodological approach employing magnetoencephalography (MEG) and machine learning techniques to investigate the flow of perceptual and semantic information decodable from neural activity in the half second during which the brain comprehends the meaning of a concrete noun. Important information about the cortical location of neural activity related to the representation of nouns in the human brain has been revealed by past studies using fMRI. However, the temporal sequence of processing from sensory input to concept comprehension remains unclear, in part because of the poor time resolution provided by fMRI. In this study, subjects answered 20 questions (e. g. is it alive?) about the properties of 60 different nouns prompted by simultaneous presentation of a pictured item and its written name. Our results show that the neural activity observed with MEG encodes a variety of perceptual and semantic features of stimuli at different times relative to stimulus onset, and in different cortical locations. By decoding these features, our MEG-based classifier was able to reliably distinguish between two different concrete nouns that it had never seen before. The results demonstrate that there are clear differences between the time course of the magnitude of MEG activity and that of decodable semantic information. Perceptual features were decoded from MEG activity earlier in time than semantic features, and features related to animacy, size, and manipulability were decoded consistently across subjects. We also observed that regions commonly associated with semantic processing in the fMRI literature may not show high decoding results in MEG. We believe that this type of approach and the accompanying machine learning methods can form the basis for further modeling of the flow of neural information during language processing and a variety of other cognitive processes.

NeurIPS Conference 2009 Conference Paper

Zero-shot Learning with Semantic Output Codes

• Mark Palatucci
• Dean Pomerleau
• Geoffrey Hinton
• Tom Mitchell

We consider the problem of zero-shot learning, where the goal is to learn a classifier $f: X \rightarrow Y$ that must predict novel values of $Y$ that were omitted from the training set. To achieve this, we define the notion of a semantic output code classifier (SOC) which utilizes a knowledge base of semantic properties of $Y$ to extrapolate to novel classes. We provide a formalism for this type of classifier and study its theoretical properties in a PAC framework, showing conditions under which the classifier can accurately predict novel classes. As a case study, we build a SOC classifier for a neural decoding task and show that it can often predict words that people are thinking about from functional magnetic resonance images (fMRI) of their neural activity, even without training examples for those words.

ICRA Conference 1998 Conference Paper

Model-Based Car Tracking Integrated with a Road-Follower

• Frank Dellaert
• Dean Pomerleau
• Charles E. Thorpe

This paper discusses how we integrated our 3D car tracking approach with the lane following module RALPH on the Navlab autonomous vehicles, obtaining a hybrid vision system that tracks both the road and cars better than those two systems in isolation. The tracking system brings precise and crisp measurements of the car in the image, and performs image stabilization. However, because it does not know, about the yaw or lateral offset of the ego-vehicle, its curvature estimate can be misguided. RALPH takes a more global image processing approach and can provide this missing information, as well as a good estimate of curvature, so that the combined curvature estimate is superior to both taken in isolation. The additional information provided by RALPH also improves tracking performance, and allows us to estimate properties of the tracked car that were previously unobservable, in particular its in-lane displacement. Better car tracking, and a better idea of where the road is, gives us a substantial foundation on which to base other capabilities needed to realize fully autonomous vehicles.

AIJ Journal 1997 Journal Article

Dynamic relevance: vision-based focus of attention using artificial neural networks

• Shumeet Baluja
• Dean Pomerleau

This paper presents a method for ascertaining the relevance of inputs in vision-based tasks by exploiting temporal coherence and predictability. In contrast to the tasks explored in many previous relevance experiments, the class of tasks examined in this study is one in which relevance is a time-varying function of the previous and current inputs. The method proposed in this paper dynamically allocates relevance to inputs by using expectations of their future values. As a model of the task is learned, the model is simultaneously extended to create task-specific predictions of the future values of inputs. Inputs that are not relevant, and therefore not accounted for in the model, will not be predicted accurately. These inputs can be de-emphasized, and, in turn, a new, improved, model of the task created. The techniques presented in this paper have been successfully applied to the vision-based autonomous control of a land vehicle, vision-based hand tracking in cluttered scenes, and the detection of faults in the plasma-etch step of semiconductor wafers.

IROS Conference 1995 Conference Paper

AURORA: a vision-based roadway departure warning system

• Mei Chen
• Todd Jochem
• Dean Pomerleau

AURORA is a vision-based system designed to warn a vehicle driver of possible impending roadway departure accidents. It employs a downward looking color video camera with a wide angle lens, a digitizer, and a portable Sun Sparc workstation. Using a binormalized adjustable template correlation algorithm, it reliably detects lane markers on structured roads at 60 Hz. A time-to-lane-crossing (TLC) measurement is calculated for each image based on the estimation of vehicle's lateral position and velocity. This measurement is used to trigger an alarm when the TLC falls below a preset threshold. Promising results have been achieved under a variety of weather and lighting conditions, on many road types.

ICML Conference 1995 Invited Paper

Learning for Automotive Collision Avoidance and Autonomous Control

• Dean Pomerleau

IROS Conference 1995 Conference Paper

Vision-based neural network road and intersection detection and traversal

• Todd Jochem
• Dean Pomerleau
• Charles E. Thorpe

The use of artificial neural networks in the domain of autonomous driving has produced promising results. ALVINN has shown that a neural system can drive a vehicle reliably and safely on many different types of roads, ranging from paved paths to interstate highways. The next step in the evolution of autonomous driving systems is to intelligently handle road junctions. In this paper the authors present an addition to the basic ALVINN driving system which makes autonomous detection of roads and traversal of simple intersections possible. The addition is based on geometrically modelling the world, accurately imaging interesting parts of the scene using this model, and monitoring ALVINN's response to the created image.

NeurIPS Conference 1994 Conference Paper

A Connectionist Technique for Accelerated Textual Input: Letting a Network Do the Typing

• Dean Pomerleau

Each year people spend a huge amount of time typing. The text people type typically contains a tremendous amount of redundancy due to predictable word usage patterns and the text's structure. This paper describes a neural network system call AutoTypist that monitors a person's typing and predicts what will be entered next. AutoTypist displays the most likely subsequent word to the typist, who can accept it with a single keystroke, instead of typing it in its entirety. The multi-layer perceptron at the heart of Auto'JYpist adapts its predictions of likely subsequent text to the user's word usage pattern, and to the characteristics of the text currently being typed. Increases in typing speed of 2-3% when typing English prose and 10-20% when typing C code have been demonstrated using the system, suggesting a potential time savings of more than 20 hours per user per year. In addition to increasing typing speed, AutoTypist reduces the number of keystrokes a user must type by a similar amount (2-3% for English, 10- 20% for computer programs). This keystroke savings has the potential to significantly reduce the frequency and severity of repeated stress injuries caused by typing, which are the most common injury suffered in today's office environment.

NeurIPS Conference 1994 Conference Paper

Using a Saliency Map for Active Spatial Selective Attention: Implementation & Initial Results

• Shumeet Baluja
• Dean Pomerleau

In many vision based tasks, the ability to focus attention on the important portions of a scene is crucial for good performance on the tasks. In this paper we present a simple method of achieving spatial selective attention through the use of a saliency map. The saliency map indicates which regions of the input retina are important for performing the task. The saliency map is cre(cid: 173) ated through predictive auto-encoding. The performance of this method is demonstrated on two simple tasks which have multiple very strong distract(cid: 173) ing features in the input retina. Architectural extensions and application directions for this model are presented.

NeurIPS Conference 1993 Conference Paper

Non-Intrusive Gaze Tracking Using Artificial Neural Networks

• Shumeet Baluja
• Dean Pomerleau

We have developed an artificial neural network based gaze tracking system which can be customized to individual users. Unlike other gaze trackers, which normally require the user to wear cumbersome headgear, or to use a chin rest to ensure head immobility, our system is entirely non-intrusive. Currently, the best intrusive gaze tracking systems are accurate to approxi(cid: 173) mately 0. 75 degrees. In our experiments, we have been able to achieve an accuracy of 1. 5 degrees, while allowing head mobility. In this paper we present an empirical analysis of the performance of a large number of artifi(cid: 173) cial neural network architectures for this task.

NeurIPS Conference 1992 Conference Paper

Input Reconstruction Reliability Estimation

• Dean Pomerleau

This paper describes a technique called Input Reconstruction Reliability Estimation (IRRE) for determining the response reliability of a restricted class of multi-layer perceptrons (MLPs). The technique uses a network's ability to accurately encode the input pattern in its internal representation as a measure of its reliability. The more accurately a network is able to reconstruct the input pattern from its internal representation, the more reliable the network is considered to be. IRRE is provides a good estimate of the reliability of MLPs trained for autonomous driving. Results are presented in which the reliability estimates provided by IRRE are used to select between networks trained for different driving situations.

NeurIPS Conference 1990 Conference Paper

Rapidly Adapting Artificial Neural Networks for Autonomous Navigation

• Dean Pomerleau

The ALVINN (Autonomous Land Vehicle In a Neural Network) project addresses the problem of training artificial neural networks in real time to perform difficult perception tasks. ALVINN, is a back-propagation network that uses inputs from a video camera and an imaging laser rangefinder to drive the CMU Navlab, a modified Chevy van. This paper describes training techniques which allow ALVINN to learn in under 5 minutes to autonomously control the Navlab by watching a human driver's response to new situations. Using these techniques, ALVINN has been trained to drive in a variety of circumstances including single-lane paved and unpaved roads, multilane lined and unlined roads, and obstacle-ridden on- and off-road environments, at speeds of up to 20 miles per hour.

NeurIPS Conference 1988 Conference Paper

ALVINN: An Autonomous Land Vehicle in a Neural Network

• Dean Pomerleau

ALVINN (Autonomous Land Vehicle In a Neural Network) is a 3-layer back-propagation network designed for the task of road following. Cur(cid: 173) rently ALVINN takes images from a camera and a laser range finder as input and produces as output the direction the vehicle should travel in order to follow the road. Training has been conducted using simulated road images. Successful tests on the Carnegie Mellon autonomous navigation test vehicle indicate that the network can effectively follow real roads under certain field conditions. The representation developed to perfOIm the task differs dra(cid: 173) matically when the networlc is trained under various conditions, suggesting the possibility of a novel adaptive autonomous navigation system capable of tailoring its processing to the conditions at hand.