Temporal-Consistent Video Restoration with Pre-trained Diffusion Models

Video restoration (VR) aims to recover high-quality videos from degraded ones. Although recent zero-shot VR methods using pre-trained diffusion models (DMs) show good promise, they suffer from approximation errors during reverse diffusion and insufficient temporal consistency. Moreover, dealing with 3D video data, VR is inherently computationally intensive. In this paper, we advocate viewing the reverse process in DMs as a function and present a novel Maximum a Posterior (MAP) framework that directly parameterizes video frames in the seed space of DMs, eliminating approximation errors. We also introduce strategies to promote bilevel temporal consistency: semantic consistency by leveraging clustering structures in the seed space, and pixel-level consistency by progressive warping with optical flow refinements. Extensive experiments on multiple virtual reality tasks demonstrate superior visual quality and temporal consistency achieved by our method compared to the state-of-the-art.

Authors

• Hengkang Wang University of Minnesota, Twin Cities
• Yang Liu College of Computer Science and Software Engineering, Shenzhen University
• Huidong Liu Amazon
• Chien-Chih Wang Amazon
• Yanhui Guo Amazon
• Hongdong Li School of Computer Science and Engineering, Central South University, Changsha Hunan Provincial Key Lab on Bioinformatics, Central South University, Changsha
• Bryan Wang
• Ju Sun University of Minnesota, Twin Cities

Keywords

No keywords are indexed for this paper.