Deep Active Learning with Noise Stability

Uncertainty estimation for unlabeled data is crucial to active learning. With a deep neural network employed as the backbone model, the data selection process is highly challenging due to the potential over-confidence of the model inference. Existing methods resort to special learning fashions (e.g. adversarial) or auxiliary models to address this challenge. This tends to result in complex and inefficient pipelines, which would render the methods impractical. In this work, we propose a novel algorithm that leverages noise stability to estimate data uncertainty. The key idea is to measure the output derivation from the original observation when the model parameters are randomly perturbed by noise. We provide theoretical analyses by leveraging the small Gaussian noise theory and demonstrate that our method favors a subset with large and diverse gradients. Our method is generally applicable in various tasks, including computer vision, natural language processing, and structural data analysis. It achieves competitive performance compared against state-of-the-art active learning baselines.

Authors

• Xingjian Li Ray and Stephanie Lane Computational Biology Department, Carnegie Mellon University, USA
• Pengkun Yang Center for Statistical Science, Tsinghua University
• Yangcheng Gu School of Software, Tsinghua University
• Xueying Zhan Computational Biology Department, Carnegie Mellon University
• Tianyang Wang Department of Computer Science, University of Alabama at Birmingham
• Min Xu
• Chengzhong Xu University of Macau

Keywords

• ML: Active Learning
• ML: Deep Learning Algorithms