TiGAN: Text-Based Interactive Image Generation and Manipulation

Using natural-language feedback to guide image generation and manipulation can greatly lower the required efforts and skills. This topic has received increased attention in recent years through refinement of Generative Adversarial Networks (GANs); however, most existing works are limited to singleround interaction, which is not reflective of real world interactive image editing workflows. Furthermore, previous works dealing with multi-round scenarios are limited to predefined feedback sequences, which is also impractical. In this paper, we propose a novel framework for Text-based interactive image generation and manipulation (TiGAN) that responds to users’ natural-language feedback. TiGAN utilizes the powerful pre-trained CLIP model to understand users’ naturallanguage feedback and exploits contrastive learning for a better text-to-image mapping. To maintain the image consistency during interactions, TiGAN generates intermediate feature vectors aligned with the feedback and selectively feeds these vectors to our proposed generative model. Empirical results on several datasets show that TiGAN improves both interaction efficiency and image quality while better avoids undesirable image manipulation during interactions.

Authors

• Yufan Zhou Adobe Research
• Ruiyi Zhang Adobe Research
• Jiuxiang Gu Adobe Research
• Chris Tensmeyer
• Tong Yu Adobe Research
• Changyou Chen University at Buffalo
• Jinhui Xu Department of Computer Science and Engineering, State University of New York at Buffalo, NY, USA
• Tong Sun Adobe Research

Keywords

No keywords are indexed for this paper.