Memory-Oriented Structural Pruning for Efficient Image Restoration

Deep learning (DL) based methods have significantly pushed forward the state-of-the-art for image restoration (IR) task. Nevertheless, DL-based IR models are highly computation- and memory-intensive. The surging demands for processing higher-resolution images and multi-task paralleling in practical mobile usage further add to their computation and memory burdens. In this paper, we reveal the overlooked memory redundancy of the IR models and propose a Memory-Oriented Structural Pruning (MOSP) method. To properly compress the long-range skip connections (a major source of the memory burden), we introduce a compactor module onto each skip connection to decouple the pruning of the skip connections and the main branch. MOSP progressively prunes the original model layers and the compactors to cut down the peak memory while maintaining high IR quality. Experiments on real image denoising, image super-resolution and low-light image enhancement show that MOSP can yield models with higher memory efficiency while better preserving performance compared with baseline pruning methods.

Authors

• Xiangsheng Shi Department of Electronic Engineering, Tsinghua University Shenzhen International Graduate School, Tsinghua University
• Xuefei Ning Tsinghua University
• Lidong Guo School of Materials Science and Engineering, Tsinghua University
• Tianchen Zhao Department of Electronic Engineering, Tsinghua University
• Enshu Liu Department of Electronic Engineering, Tsinghua University
• Yi Cai South China University of Technology
• Yuhan Dong Tsinghua University
• Huazhong Yang Tsinghua University
• Yu Wang

Keywords

• CV: Low Level & Physics-based Vision
• ML: Learning on the Edge & Model Compression