Garrison W. Cottrell

AAAI Conference 2021 Conference Paper

Joint-Label Learning by Dual Augmentation for Time Series Classification

• Qianli Ma
• Zhenjing Zheng
• Jiawei Zheng
• Sen Li
• Wanqing Zhuang
• Garrison W. Cottrell

Recently, deep neural networks (DNNs) have achieved excellent performance on time series classification. However, DNNs require large amounts of labeled data for supervised training. Although data augmentation can alleviate this problem, the standard approach assigns the same label to all augmented samples from the same source. This leads to the expansion of the data distribution such that the classification boundaries may be even harder to determine. In this paper, we propose Joint-label learning by Dual Augmentation (JobDA), which can enrich the training samples without expanding the distribution of the original data. Instead, we apply simple transformations to the time series and give these modified time series new labels, so that the model has to distinguish between these and the original data, as well as separating the original classes. This approach sharpens the boundaries around the original time series, and results in superior classification performance. We use Time Series Warping for our transformations: We shrink and stretch different regions of the original time series, like a fun-house mirror. Experiments conducted on extensive time-series datasets show that JobDA can improve the model performance on small datasets. Moreover, we verify that JobDA has better generalization ability compared with conventional data augmentation, and the visualization analysis further demonstrates that JobDA can learn more compact clusters.

AAAI Conference 2021 Conference Paper

Learning Representations for Incomplete Time Series Clustering

• Qianli Ma
• Chuxin Chen
• Sen Li
• Garrison W. Cottrell

Time-series clustering is an essential unsupervised technique for data analysis, applied to many real-world fields, such as medical analysis and DNA microarray. Existing clustering methods are usually based on the assumption that the data is complete. However, time series in real-world applications often contain missing values. Traditional strategy (imputing first and then clustering) does not optimize the imputation and clustering process as a whole, which not only makes performance dependent on the combination of imputation and clustering methods but also fails to achieve satisfactory results. How to best improve the clustering performance on incomplete time series remains a challenge. This paper proposes a novel unsupervised temporal representation learning model, named Clustering Representation Learning on Incomplete time-series data (CRLI). CRLI jointly optimizes the imputation and clustering process to impute more discriminative values for clustering and make the learned representations possessed good clustering property. Also, to reduce the error propagation from imputation to clustering, we introduce a discriminator to make the distribution of imputation values close to the true one and train CRLI in an alternating training manner. An experiment conducted on eight real-world incomplete time-series datasets shows that CRLI outperforms existing methods. We demonstrate the effectiveness of the learned representations and the convergence of the model through visualization analysis. Moreover, we reveal that the joint training strategy can impute values close to the true ones in those important sub-sequences, and impute more discriminative values in those less important sub-sequences at the same time, making the imputed sequence cluster-friendly.

UAI Conference 2021 Conference Paper

ReZero is all you need: fast convergence at large depth

• Thomas Bachlechner
• Bodhisattwa Prasad Majumder
• Huanru Henry Mao
• Garrison W. Cottrell
• Julian J. McAuley

Deep networks often suffer from vanishing or exploding gradients due to inefficient signal propagation, leading to long training times or convergence difficulties. Various architecture designs, sophisticated residual-style networks, and initialization schemes have been shown to improve deep signal propagation. Recently, Pennington et al. [2017] used free probability theory to show that dynamical isometry plays an integral role in efficient deep learning. We show that the simplest architecture change of gating each residual connection using a single zero-initialized parameter satisfies initial dynamical isometry and outperforms more complex approaches. Although much simpler than its predecessors, this gate enables training thousands of fully connected layers with fast convergence and better test performance for ResNets trained on an image recognition task. We apply this technique to language modeling and find that we can easily train 120-layer Transformers. When applied to 12 layer Transformers, it converges 56% faster.

ICLR Conference 2020 Conference Paper

Detecting and Diagnosing Adversarial Images with Class-Conditional Capsule Reconstructions

• Yao Qin 0001
• Nicholas Frosst
• Sara Sabour
• Colin Raffel
• Garrison W. Cottrell
• Geoffrey E. Hinton

Adversarial examples raise questions about whether neural network models are sensitive to the same visual features as humans. In this paper, we first detect adversarial examples or otherwise corrupted images based on a class-conditional reconstruction of the input. To specifically attack our detection mechanism, we propose the Reconstructive Attack which seeks both to cause a misclassification and a low reconstruction error. This reconstructive attack produces undetected adversarial examples but with much smaller success rate. Among all these attacks, we find that CapsNets always perform better than convolutional networks. Then, we diagnose the adversarial examples for CapsNets and find that the success of the reconstructive attack is highly related to the visual similarity between the source and target class. Additionally, the resulting perturbations can cause the input image to appear visually more like the target class and hence become non-adversarial. This suggests that CapsNets use features that are more aligned with human perception and have the potential to address the central issue raised by adversarial examples.

ICML Conference 2019 Conference Paper

Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition

• Yao Qin 0001
• Nicholas Carlini
• Garrison W. Cottrell
• Ian J. Goodfellow
• Colin Raffel

Adversarial examples are inputs to machine learning models designed by an adversary to cause an incorrect output. So far, adversarial examples have been studied most extensively in the image domain. In this domain, adversarial examples can be constructed by imperceptibly modifying images to cause misclassification, and are practical in the physical world. In contrast, current targeted adversarial examples on speech recognition systems have neither of these properties: humans can easily identify the adversarial perturbations, and they are not effective when played over-the-air. This paper makes progress on both of these fronts. First, we develop effectively imperceptible audio adversarial examples (verified through a human study) by leveraging the psychoacoustic principle of auditory masking, while retaining 100% targeted success rate on arbitrary full-sentence targets. Then, we make progress towards physical-world audio adversarial examples by constructing perturbations which remain effective even after applying highly-realistic simulated environmental distortions.

IJCAI Conference 2017 Conference Paper

A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction

• Yao Qin
• Dongjin Song
• Haifeng Chen
• Wei Cheng
• Guofei Jiang
• Garrison W. Cottrell

The Nonlinear autoregressive exogenous (NARX) model, which predicts the current value of a time series based upon its previous values as well as the current and past values of multiple driving (exogenous) series, has been studied for decades. Despite the fact that various NARX models have been developed, few of them can capture the long-term temporal dependencies appropriately and select the relevant driving series to make predictions. In this paper, we propose a dual-stage attention-based recurrent neural network (DA-RNN) to address these two issues. In the first stage, we introduce an input attention mechanism to adaptively extract relevant driving series (a. k. a. , input features) at each time step by referring to the previous encoder hidden state. In the second stage, we use a temporal attention mechanism to select relevant encoder hidden states across all time steps. With this dual-stage attention scheme, our model can not only make predictions effectively, but can also be easily interpreted. Thorough empirical studies based upon the SML 2010 dataset and the NASDAQ 100 Stock dataset demonstrate that the DA-RNN can outperform state-of-the-art methods for time series prediction.

IROS Conference 2017 Conference Paper

Belief tree search for active object recognition

• Mohsen Malmir
• Garrison W. Cottrell

Active Object Recognition (AOR) has been approached as an unsupervised learning problem, in which optimal trajectories for object inspection are not known and to be discovered by reducing label uncertainty or training with reinforcement learning. Such approaches suffer from local optima and have no guarantees of the quality of their solution. In this paper, we treat AOR as a Partially Observable Markov Decision Process (POMDP) and find near-optimal values and corresponding action-values of training data using Belief Tree Search (BTS) on the AOR belief Markov Decision Process (MDP). AOR then reduces to the problem of knowledge transfer from these action-values to the test set. We train a Long Short Term Memory (LSTM) network on these values to predict the best next action on the training set rollouts and experimentally show that our method generalizes well to explore novel objects and novel views of familiar objects with high accuracy. We compare this supervised scheme against guided policy search, and show that the LSTM network reaches higher recognition accuracy compared to the guided policy search and guided Neurally Fitted Q-iteration. We further look into optimizing the observation function to increase the total collected reward during active recognition. In AOR, the observation function is known only approximately. We derive a gradient-based update for the observation function to increase the total expected reward. We show that by optimizing the observation function and retraining the supervised LSTM network, the AOR performance on the test set improves significantly.

IJCAI Conference 2017 Conference Paper

WALKING WALKing walking: Action Recognition from Action Echoes

• Qianli Ma
• Lifeng Shen
• Enhuan Chen
• Shuai Tian
• Jiabing Wang
• Garrison W. Cottrell

Recognizing human actions represented by 3D trajectories of skeleton joints is a challenging machine learning task. In this paper, the 3D skeleton sequences are regarded as multivariate time series, and their dynamics and multiscale features are efficiently learned from action echo states. Specifically, first the skeleton data from the limbs and trunk are projected into five high dimensional nonlinear spaces, that are randomly generated by five dynamic, training-free recurrent networks, i. e. , the reservoirs of echo state networks (ESNs). In this way, the history of the time series is represented as nonlinear echo states of actions. We then use a single multiscale convolutional layer to extract multiscale features from the echo states, and maintain multiscale temporal invariance by a max-over-time pooling layer. We propose two multi-step fusion strategies to integrate the spatial information over the five parts of the human physical structure. Finally, we learn the label distribution using softmax. With one training-free recurrent layer and only layer of convolution, our Convolutional Echo State Network (ConvESN) is a very efficient end-to-end model, and achieves state-of-the-art performance on four skeleton benchmark data sets.

ICRA Conference 2008 Conference Paper

Gamma-SLAM: Using stereo vision and variance grid maps for SLAM in unstructured environments

• Tim K. Marks
• Andrew Howard
• Max Bajracharya
• Garrison W. Cottrell
• Larry H. Matthies

We introduce a new method for stereo visual SLAM (simultaneous localization and mapping) that works in unstructured, outdoor environments. Unlike other grid-based SLAM algorithms, which use occupancy grid maps, our algorithm uses a new mapping technique that maintains a posterior distribution over the height variance in each cell. This idea was motivated by our experience with outdoor navigation tasks, which has shown height variance to be a useful measure of traversability. To obtain a joint posterior over poses and maps, we use a Rao-Blackwellized particle filter: the pose distribution is estimated using a particle filter, and each particle has its own map that is obtained through exact filtering conditioned on the particle’s pose. Visual odometry provides good proposal distributions for the particle pose. In the analytical (exact) filter for the map, we update the sufficient statistics of a gamma distribution over the precision (inverse variance) of heights in each grid cell. We verify the algorithm’s accuracy on two outdoor courses by comparing with ground truth data obtained using electronic surveying equipment. In addition, we solve for the optimal transformation from the SLAM map to georeferenced coordinates, based on a noisy GPS signal. We derive an online version of this alignment process, which can be used to maintain a running estimate of the robot’s global position that is much more accurate than the GPS readings.

ICRA Conference 2008 Conference Paper

Visual saliency model for robot cameras

• Nicholas J. Butko
• Lingyun Zhang
• Garrison W. Cottrell
• Javier R. Movellan

Recent years have seen an explosion of research on the computational modeling of human visual attention in task free conditions, i. e. , given an image predict where humans are likely to look. This area of research could potentially provide general purpose mechanisms for robots to orient their cameras. One difficulty is that most current models of visual saliency are computationally very expensive and not suited to real time implementations needed for robotic applications. Here we propose a fast approximation to a Bayesian model of visual saliency recently proposed in the literature. The approximation can run in real time on current computers at very little computational cost, leaving plenty of CPU cycles for other tasks. We empirically evaluate the saliency model in the domain of controlling saccades of a camera in social robotics situations. The goal was to orient a camera as quickly as possible toward human faces. We found that this simple general purpose saliency model doubled the success rate of the camera: it captured images of people 70% of the time, when compared to a 35% success rate when the camera was controlled using an open-loop scheme. After 3 saccades (camera movements), the robot was 96% likely to capture at least one person. The results suggest that visual saliency models may provide a useful front end for camera control in robotics applications.

ICML Conference 2003 Conference Paper

Principled Methods for Advising Reinforcement Learning Agents

• Eric Wiewiora
• Garrison W. Cottrell
• Charles Elkan

IJCAI Conference 1985 Conference Paper

Paraifelism in Inheritance Hierarchies with Exceptions

• Garrison W. Cottrell

In a recent paper, Etherington & Reiter formalized a simple version of semantic networks with exceptions in terms of Reiter's Default Logic. With this approach they were able to formally characterize the correctness of an inference algorithm in terms of Default Logic, and exhibited an algorithm that was correct in this sense. Finally, they concluded that massively parallel architectures for semantic networks, such as NETL apparently cannot implement this algorithm. In this paper, we present a different massively parallel architecture for the simplified semantic networks outlined in their paper which appears to avoid the objections to NETL. We also present some results of simulations in this framework of the examples presented in Etherington and Reiter.

AAAI Conference 1984 Conference Paper

A Model of Lexical Access of Ambiguous Words

• Garrison W. Cottrell

Recent psycholinguistic work in the study of lexical access has supported a modular view of the process. That is, lexical access proceeds indepedently of the sentential context. Herein we describe a connectionist model of the process which retains modularity, explains apparent anomalies in the results, and makes empirically verifiable predictions.

NMR Workshop 1984 Conference Paper

Re: Inheritance Hierarchies with Exceptions

• Garrison W. Cottrell