Multi-Modality Deep Network for Extreme Learned Image Compression

Image-based single-modality compression learning approaches have demonstrated exceptionally powerful encoding and decoding capabilities in the past few years, but suffer from blur and severe semantics loss at extremely low bitrates. To address this issue, we propose a multimodal machine learning method for text-guided image compression, in which the semantic information of text is used as prior information to guide image compression for better compression performance. We fully study the role of text description in different components of the codec, and demonstrate its effectiveness. In addition, we adopt the image-text attention module and image-request complement module to better fuse image and text features, and propose an improved multimodal semantic-consistent loss to produce semantically complete reconstructions. Extensive experiments, including a user study, prove that our method can obtain visually pleasing results at extremely low bitrates, and achieves a comparable or even better performance than state-of-the-art methods, even though these methods are at 2x to 4x bitrates of ours.

Authors

• Xuhao Jiang School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University
• Weimin Tan School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University
• Tian Tan Fudan University
• Bo Yan School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University
• Liquan Shen Shanghai University

Keywords

• CV: Low Level & Physics-based Vision
• CV: Multi-modal Vision