SC2Net: Sparse LSTMs for Sparse Coding

The iterative hard-thresholding algorithm (ISTA) is one of the most popular optimization solvers to achieve sparse codes. However, ISTA suffers from following problems: 1) ISTA employs non-adaptive updating strategy to learn the parameters on each dimension with a fixed learning rate. Such a strategy may lead to inferior performance due to the scarcity of diversity; 2) ISTA does not incorporate the historical information into the updating rules, and the historical information has been proven helpful to speed up the convergence. To address these challenging issues, we propose a novel formulation of ISTA (named as adaptive ISTA) by introducing a novel adaptive momentum vector. To efficiently solve the proposed adaptive ISTA, we recast it as a recurrent neural network unit and show its connection with the well-known long short term memory (LSTM) model. With a new proposed unit, we present a neural network (termed SC2Net) to achieve sparse codes in an end-to-end manner. To the best of our knowledge, this is one of the first works to bridge the 1-solver and LSTM, and may provide novel insights in understanding model-based optimization and LSTM. Extensive experiments show the effectiveness of our method on both unsupervised and supervised tasks.

Authors

• Joey Tianyi Zhou Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore
• Kai Di
• Jiawei Du Centre for Frontier AI Research (CFAR) & Institute of High Performance Computing (IHPC), A*STAR, Singapore
• Xi Peng University of Delaware
• Hao Yang Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
• Sinno Jialin Pan The Chinese University of Hong Kong
• Ivor Tsang Centre for Frontier AI Research, Agency for Science, Technology and Research Institute of High Performance Computing, Agency for Science, Technology and Research
• Yong Liu
• Zheng Qin Xi'an Jiaotong University
• Rick Siow Mong Goh Institute of High Performance Computing, A*STAR, Singapore

Keywords

No keywords are indexed for this paper.