Accelerating Sparse Convolution with Column Vector-Wise Sparsity

Weight sparsity is a promising approach to reducing the model size and computation cost of convolutional neural networks (CNNs). Nevertheless, non-zero weights often distribute randomly in sparse CNN models, introducing enormous difficulty in obtaining actual speedup on common hardware (e. g. , GPU) over their dense counterparts. Existing acceleration solutions either require hardware modifications for irregular memory access support or rely on a partially structured sparsity pattern. Neither of these methods is capable of achieving fruitful speedup on convolution layers. In this work, we propose an algorithm-software co-designed sparse convolution based on a novel out-vector-wise (OVW) sparse pattern. Building on the insight that vertical vector integrity can preserve continuous memory access in IM2COL, the OVW pattern treats a $V\times1$ vector as an entirety. To reduce the error caused by sparsity, we propose an equivalent transformation process, i. e. , clustering-based channel permutation, to gather similar rows together. Experimental evaluations demonstrate that our method achieves a $1. 7\times$ and $3. 2\times$ speedup over the SOTA solution and the dense convolution of ResNet50 on NVIDIA V100 at 75\% sparsity, respectively, with only negligible accuracy loss. Moreover, compared to the SOTA solution that achieves speedups only on data with 60\% sparsity or more, our method begins to obtain speedups on data with only 10\% sparsity.

Authors

• Yijun Tan
• Kai Han
• Kang Zhao Tongyi Lab
• Xianzhi Yu
• Zidong Du SKL of Processors, Institute of Computing Technology, CAS
• Yunji Chen SKL of Processors, Institute of Computing Technology, CAS University of Chinese Academy of Sciences
• Yunhe Wang Huawei Noah's Ark Lab
• Jun Yao

Keywords

No keywords are indexed for this paper.