FSR: A General Frequency-Oriented Framework to Accelerate Image Super-resolution Networks

Deep neural networks (DNNs) have witnessed remarkable achievement in image super-resolution (SR), and plenty of DNN-based SR models with elaborated network designs have recently been proposed. However, existing methods usually require substantial computations by operating in spatial domain. To address this issue, we propose a general frequency-oriented framework (FSR) to accelerate SR networks by considering data characteristics in frequency domain. Our FSR mainly contains dual feature aggregation module (DFAM) to extract informative features in both spatial and transform domains, followed by a four-path SR-Module with different capacities to super-resolve in the frequency domain. Specifically, DFAM further consists of a transform attention block (TABlock) and a spatial context block (SCBlock) to extract global spectral information and local spatial information, respectively, while SR-Module is a parallel network container that contains four to-be-accelerated branches. Furthermore, we propose an adaptive weight strategy for a trade-off between image details recovery and visual quality. Extensive experiments show that our FSR can save FLOPs by almost 40% while reducing inference time by 50% for other SR methods (e.g., FSRCNN, CARN, SRResNet and RCAN). Code is available at https://github.com/THU-Kingmin/FSR.

Authors

• Jinmin Li Tsinghua University
• Tao Dai Shenzhen University
• Mingyan Zhu Tsinghua Shenzhen International Graduate School, Tsinghua University
• Bin Chen Harbin Institute of Technology, Shenzhen Research Center of Artificial Intelligence, Peng Cheng Laboratory
• Zhi Wang Tsinghua University
• Shu-Tao Xia Tsinghua Shenzhen International Graduate School Pengcheng Laboratory

Keywords

• CV: Applications
• CV: Computational Photography, Image & Video Synthesis
• CV: Low Level & Physics-based Vision