Video Event Extraction with Multi-View Interaction Knowledge Distillation

Video event extraction (VEE) aims to extract key events and generate the event arguments for their semantic roles from the video. Despite promising results have been achieved by existing methods, they still lack an elaborate learning strategy to adequately consider: (1) inter-object interaction, which reflects the relation between objects; (2) inter-modality interaction, which aligns the features from text and video modality. In this paper, we propose a Multi-view Interaction with knowledge Distillation (MID) framework to solve the above problems with the Knowledge Distillation (KD) mechanism. Specifically, we propose the self-Relational KD (self-RKD) to enhance the inter-object interaction, where the relation between objects is measured by distance metric, and the high-level relational knowledge from the deeper layer is taken as the guidance for boosting the shallow layer in the video encoder. Meanwhile, to improve the inter-modality interaction, the Layer-to-layer KD (LKD) is proposed, which integrates additional cross-modal supervisions (i.e., the results of cross-attention) with the textual supervising signal for training each transformer decoder layer. Extensive experiments show that without any additional parameters, MID achieves the state-of-the-art performance compared to other strong methods in VEE.

Authors

• Kaiwen Wei Aerospace Information Research Institute,Chinese Academy of Sciences; Key Laboratory of Network Information System Technology (NIST), Aerospace Information Research Institute School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences
• Runyan Du University of Chinese Academy of Sciences, Beijing, China
• Li Jin
• Jian Liu
• Jianhua Yin School of Computer Science and Technology, Shandong University, Qingdao, China
• Linhao Zhang Aerospace Information Research Institute, Chinese Academy of Sciences; Key Laboratory of Network Information System Technology (NIST), Aerospace Information Research Institute School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences
• Jintao Liu University of Chinese Academy of Sciences
• Nayu Liu School of Computer Science and Technology, Tiangong University
• Jingyuan Zhang Kuaishou Technology Inc., Beijing, China
• Zhi Guo Baidu Inc.

Keywords

• CV: Language and Vision
• DMKM: Mining of Visual, Multimedia & Multimodal Data
• ML: Multimodal Learning
• NLP: Information Extraction
• NLP: Language Grounding & Multi-modal NLP