Distilling Knowledge from Well-Informed Soft Labels for Neural Relation Extraction

Extracting relations from plain text is an important task with wide application. Most existing methods formulate it as a supervised problem and utilize one-hot hard labels as the sole target in training, neglecting the rich semantic information among relations. In this paper, we aim to explore the supervision with soft labels in relation extraction, which makes it possible to integrate prior knowledge. Specifically, a bipartite graph is first devised to discover type constraints between entities and relations based on the entire corpus. Then, we combine such type constraints with neural networks to achieve a knowledgeable model. Furthermore, this model is regarded as teacher to generate well-informed soft labels and guide the optimization of a student network via knowledge distillation. Besides, a multi-aspect attention mechanism is introduced to help student mine latent information from text. In this way, the enhanced student inherits the dark knowledge (e. g. , type constraints and relevance among relations) from teacher, and directly serves the testing scenarios without any extra constraints. We conduct extensive experiments on the TACRED and SemEval datasets, the experimental results justify the effectiveness of our approach.

Authors

• Zhenyu Zhang State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China School of Intelligence Science and Technology, Nanjing University, Suzhou, China
• Xiaobo Shu
• Bowen Yu Alibaba Group
• Tingwen Liu Institute of Information Engineering, Chinese Academy of Sciences School of Cyber Security, University of Chinese Academy of Sciences
• Jiapeng Zhao
• Quangang Li
• Li Guo School of Artificial Intelligence, Beijing University of Posts and Telecommunications

Keywords

No keywords are indexed for this paper.