Universal Guidance for Diffusion Models

Typical diffusion models are trained to accept a particular form of conditioning, most commonly text, and cannot be conditioned on other modalities without retraining. In this work, we propose a universal guidance algorithm that enables diffusion models to be controlled by arbitrary guidance modalities without the need to retrain any use-specific components. We show that our algorithm successfully generates quality images with guidance functions including segmentation, face recognition, object detection, style guidance and classifier signals.

Authors

• Arpit Bansal
• Hong-Min Chu
• Avi Schwarzschild
• Soumyadip Sengupta
• Micah Goldblum
• Jonas Geiping
• Tom Goldstein

Keywords

• Generative Models
• Computer Vision
• Diffusion Models