A Geometric Perspective on Optimizing Vector Quantized Latent Diffusion Model for Image Restoration

In this paper, we investigate the limitations of the Vector Quantized Latent Diffusion Model (VQ-LDM) in restoration tasks. We identify a performance gap between the Vector Quantization (VQ) and Diffusion Model components, manifested as a significant discrepancy between the reconstruction quality of ground truth images processed via VQ autoregression and degraded images restored by VQ-LDM. Through experiments, we attribute this gap primarily to the lack of robustness in the mapped points of VQ within the original VQ-LDM framework. To address this issue, we propose a geometric based optimization approach. First, we introduce a simple yet effective method, termed interpolation-based latent initial state optimization, which mitigates the performance gap by replacing the original mapped points with interpolated values, supported by theoretical analysis. Here, the latent initial state refers specifically to the input of the diffusion model. Building upon this, we further propose a Chebyshev center-based latent initial state optimization, an elegant theoretical solution from a geometric perspective, that further enhances restoration performance. Our improvements consistently achieve superior results across nine benchmark datasets.

Authors

• Chen Hang East China Normal University
• Haoming Chen East China Normal University
• Xuwei Fang Bestpay AI Lab
• Weisheng Xie Bestpay AI Lab, Shanghai, China
• Xiangxiang Gao Bestpay AI Lab, Shanghai, China
• Faming Fang School of Computer Science and Technology, East China Normal University Shanghai Key Laboratory of Multidimensional Information Processing
• Guixu Zhang School of Computer Science and Technology, East China Normal University
• Haichuan Song

Keywords

No keywords are indexed for this paper.