TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings

This paper presents the Text Encoding Diffusion Model (TEncDM), a novel approach to diffusion modeling that operates in the space of pre-trained language model encodings. In contrast to traditionally used embeddings, encodings integrate contextual information. In our approach, we also employ a transformer-based decoder, specifically designed to incorporate context in the token prediction process. We conduct a comprehensive examination of the influence of the encoder, decoder, noise scheduler, and self-conditioning on zero-shot generation. Furthermore, we compare TEncDM with previous approaches on three conditional text generation tasks: QQP, XSum, and Wiki-Auto. The results show that TEncDM exhibits superior performance compared to existing non-autoregressive diffusion models.

Authors

• Alexander Shabalin Constructor University Higher School of Economics
• Viacheslav Meshchaninov Higher School of Economics
• Egor Chimbulatov Higher School of Economics
• Vladislav Lapikov Higher School of Economics
• Roman Kim Higher School of Economics
• Grigory Bartosh University of Amsterdam
• Dmitry Molchanov Independent Researcher
• Sergey Markov SberDevices
• Dmitry Vetrov Constructor University

Keywords

No keywords are indexed for this paper.