Faster Diffusion Through Temporal Attention Decomposition

We explore the role of the attention mechanism during inference in text-conditional diffusion models. Empirical observations suggest that cross-attention outputs converge to a fixed point after several inference steps. The convergence time naturally divides the entire inference process into two phases: an initial phase for planning text-oriented visual semantics, which are then translated into images in a subsequent fidelity-improving phase. Cross-attention is essential in the initial phase but almost irrelevant thereafter. Self-attention, however, initially plays a minor role but becomes increasingly important in the second phase. These findings yield a simple and training-free method called TGATE which efficiently generates images by caching and reusing attention outputs at scheduled time steps. Experiments show TGATE’s broad applicability to various existing text-conditional diffusion models which it speeds up by 10-50%. The code of TGATE is available at https://github.com/HaozheLiu-ST/T-GATE.

Authors

• Haozhe Liu AI Initiative, King Abdullah University of Science and Technology Jarvis Lab, Tencent
• Wentian Zhang
• Jinheng Xie National University of Singapore
• Francesco Faccio The Swiss AI Lab IDSIA AI Initiative, KAUST
• Mengmeng Xu
• Tao Xiang Chongqing University
• Mike Zheng Shou National University of Singapore
• Juan-Manuel Perez-Rua
• Jürgen Schmidhuber

Keywords

No keywords are indexed for this paper.