AAAI Conference 2022 Short Paper
Gradient and Mangitude Based Pruning for Sparse Deep Neural Networks
- Kaleab Belay
Deep Neural Networks have memory and computational demands that often render them difficult to use in lowresource environments. Also, highly dense networks are over-parameterized and thus prone to overfitting. To address these problems, we introduce a novel algorithm that prunes (sparsifies) weights from the network by taking into account their magnitudes and gradients taken against a validation dataset. Unlike existing pruning methods, our method does not require the network model to be retrained once initial training is completed. On the CIFAR-10 dataset, our method reduced the number of para1 meters of MobileNet by a factor of 9X, from 14 million to 1. 5 million, with just a 3. 8% drop in accuracy.