Variational Structured Semantic Inference for Diverse Image Captioning

Despite the exciting progress in image captioning, generating diverse captions for a given image remains as an open problem. Existing methods typically apply generative models such as Variational Auto-Encoder to diversify the captions, which however neglect two key factors of diverse expression, i. e. , the lexical diversity and the syntactic diversity. To model these two inherent diversities in image captioning, we propose a Variational Structured Semantic Inferring model (termed VSSI-cap) executed in a novel structured encoder-inferer-decoder schema. VSSI-cap mainly innovates in a novel structure, i. e. , Variational Multi-modal Inferring tree (termed VarMI-tree). In particular, conditioned on the visual-textual features from the encoder, the VarMI-tree models the lexical and syntactic diversities by inferring their latent variables (with variations) in an approximate posterior inference guided by a visual semantic prior. Then, a reconstruction loss and the posterior-prior KL-divergence are jointly estimated to optimize the VSSI-cap model. Finally, diverse captions are generated upon the visual features and the latent variables from this structured encoder-inferer-decoder model. Experiments on the benchmark dataset show that the proposed VSSI-cap achieves significant improvements over the state-of-the-arts.

Authors

• Fuhai Chen College of Computer and Data Science, Fuzhou University
• Rongrong Ji Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China
• Jiayi Ji Xiamen University National University of Singapore
• Xiaoshuai Sun Xiamen University
• Baochang Zhang Hangzhou Research Institute, School of Artificial Intelligence, Beihang University, China Nanchang Institute of Technology, Nanchang, China
• Xuri Ge Shandong University
• Yongjian Wu Tencent Technology (Shanghai) Co.,Ltd
• Feiyue Huang
• Yan Wang

Keywords

No keywords are indexed for this paper.