Multi-Modality Deep Network for JPEG Artifacts Reduction

In recent years, many convolutional neural network-based models are designed for JPEG artifacts reduction, and have achieved notable progress. However, few methods are suitable for extreme low-bitrate image compression artifacts reduction. The main challenge is that the highly compressed image loses too much information, resulting in reconstructing high-quality image difficultly. To address this issue, we propose a multimodal fusion learning method for text-guided JPEG artifacts reduction, in which the corresponding text description not only provides the potential prior information of the highly compressed image, but also serves as supplementary information to assist in image deblocking. We fuse image features and text semantic features from the global and local perspectives respectively, and design a contrastive loss built upon contrastive learning to produce visually pleasing results. Extensive experiments, including a user study, prove that our method can obtain better deblocking results compared to the state-of-the-art methods.

Authors

• Xuhao Jiang School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University
• Weimin Tan School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University
• Qing Lin
• Chenxi Ma Fudan University
• Bo Yan School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University
• Liquan Shen Shanghai University

Keywords

• Computer Vision: CV: Machine learning for vision
• Machine Learning: ML: Multi-modal learning