A Question Type Driven Framework to Diversify Visual Question Generation

Visual question generation aims at asking questions about an image automatically. Existing research works on this topic usually generate a single question for each given image without considering the issue of diversity. In this paper, we propose a question type driven framework to produce multiple questions for a given image with different focuses. In our framework, each question is constructed following the guidance of a sampled question type in a sequence-to-sequence fashion. To diversify the generated questions, a novel conditional variational auto-encoder is introduced to generate multiple questions with a specific question type. Moreover, we design a strategy to conduct the question type distribution learning for each image to select the final questions. Experimental results on three benchmark datasets show that our framework outperforms the state-of-the-art approaches in terms of both relevance and diversity.

Authors

• Zhihao Fan
• Zhongyu Wei Fudan University
• Piji Li Chen Frontier Lab for AI and Mental Health, Tianqiao and Chrissy Chen Institute, Shanghai, China
• Yanyan Lan
• Xuanjing Huang School of Computer Science, Fudan University, Shanghai, China Shanghai Collaborative Innovation Center of Intelligent Visual Computing, Shanghai, China

Keywords

• Machine Learning: Deep Learning
• Machine Learning: Neural Networks
• Natural Language Processing: Natural Language Generation