Document Informed Neural Autoregressive Topic Models with Distributional Prior

We address two challenges in topic models: (1) Context information around words helps in determining their actual meaning, e. g. , “networks” used in the contexts artificial neural networks vs. biological neuron networks. Generative topic models infer topic-word distributions, taking no or only little context into account. Here, we extend a neural autoregressive topic model to exploit the full context information around words in a document in a language modeling fashion. The proposed model is named as iDocNADE. (2) Due to the small number of word occurrences (i. e. , lack of context) in short text and data sparsity in a corpus of few documents, the application of topic models is challenging on such texts. Therefore, we propose a simple and efficient way of incorporating external knowledge into neural autoregressive topic models: we use embeddings as a distributional prior. The proposed variants are named as DocNADEe and iDocNADEe. We present novel neural autoregressive topic model variants that consistently outperform state-of-the-art generative topic models in terms of generalization, interpretability (topic coherence) and applicability (retrieval and classification) over 7 long-text and 8 short-text datasets from diverse domains.

Authors

• Pankaj Gupta Biometric Research Laboratory, Delhi Technological University
• Yatin Chaudhary
• Florian Buettner German Cancer Research Center (DKFZ) Heidelberg, Germany German Cancer Consortium (DKTK), Germany Goethe University Frankfurt, Germany
• Hinrich Schütze

Keywords

No keywords are indexed for this paper.