On Analyzing the Role of Image for Visual-Enhanced Relation Extraction (Student Abstract)

Multimodal relation extraction is an essential task for knowledge graph construction. In this paper, we take an in-depth empirical analysis that indicates the inaccurate information in the visual scene graph leads to poor modal alignment weights, further degrading performance. Moreover, the visual shuffle experiments illustrate that the current approaches may not take full advantage of visual information. Based on the above observation, we further propose a strong baseline with an implicit fine-grained multimodal alignment based on Transformer for multimodal relation extraction. Experimental results demonstrate the better performance of our method. Codes are available at https://github.com/zjunlp/DeepKE/tree/main/example/re/multimodal.

Authors

• Lei Li Cleveland State University
• Xiang Chen Zhejiang University
• Shuofei Qiao Zhejiang University
• Feiyu Xiong Institute for Advanced Algorithms Research, Shanghai, China
• Huajun Chen College of Computer Science and Technology, Zhejiang University ZJU-Ant Group Joint Lab of Knowledge Graph Zhejiang Key Laboratory of Big Data Intelligent Computing
• Ningyu Zhang Zhejiang University Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph

Keywords

• Knowledge Graph
• Multimodal
• Relation Extraction