Unsupervised Region-Based Image Editing of Denoising Diffusion Models

Although diffusion models have achieved remarkable success in the field of image generation, their latent space remains under-explored. Current methods for identifying semantics within latent space often rely on external supervision, such as textual information and segmentation masks. In this paper, we propose a method to identify semantic attributes in the latent space of pre-trained diffusion models without any further training. By projecting the Jacobian of the targeted semantic region into a low-dimensional subspace which is orthogonal to the non-masked regions, our approach facilitates precise semantic discovery and control over local masked areas, eliminating the need for annotations. We conducted extensive experiments across multiple datasets and various architectures of diffusion models, achieving state-of-the-art performance. In particular, for some specific face attributes, the performance of our proposed method even surpasses that of supervised approaches, demonstrating its superior ability in editing local image properties.

Authors

• Zixiang Li Institute of Information Science, Beijing Jiaotong University Visual Intellgence +X International Cooperation Joint Laboratory of MOE
• Yue Song University of Trento, Italy
• Renshuai Tao Institute of Information Science, Beijing Jiaotong University Visual Intellgence +X International Cooperation Joint Laboratory of MOE
• Xiaohong Jia Lanzhou Jiaotong University
• Yao Zhao Institute of Information Science, Beijing Jiaotong University Beijing Key Laboratory of Advanced Information Science and Network Technology
• Wei Wang

Keywords

No keywords are indexed for this paper.