Explicit Knowledge-based Reasoning for Visual Question Answering

We describe a method for visual question answering which is capable of reasoning about an image on the basis of information extracted from a large-scale knowledge base. The method not only answers natural language questions using concepts not contained in the image, but can explain the reasoning by which it developed its answer. It is capable of answering far more complex questions than the predominant long short-term memory-based approach, and outperforms it significantly in testing. We also provide a dataset and a protocol by which to evaluate general visual question answering methods.

Authors

• Peng Wang
• Qi Wu City University of Hong Kong
• Chunhua Shen College of Computer Science and Technology, Zhejiang University
• Anthony Dick
• Anton van den Hengel Amazon

Keywords

• Knowledge Representation, Reasoning, and Logic: Common-Sense Reasoning
• Knowledge Representation, Reasoning, and Logic: Reasoning about Knowlege and Belief
• Natural Language Processing: Question Answering