Semantics-Aware BERT for Language Understanding

The latest work on language representations carefully integrates contextualized features into language model training, which enables a series of success especially in various machine reading comprehension and natural language inference tasks. However, the existing language representation models including ELMo, GPT and BERT only exploit plain context-sensitive features such as character or word embeddings. They rarely consider incorporating structured semantic information which can provide rich semantics for language representation. To promote natural language understanding, we propose to incorporate explicit contextual semantics from pre-trained semantic role labeling, and introduce an improved language representation model, Semanticsaware BERT (SemBERT), which is capable of explicitly absorbing contextual semantics over a BERT backbone. Sem- BERT keeps the convenient usability of its BERT precursor in a light fine-tuning way without substantial task-specific modi- fications. Compared with BERT, semantics-aware BERT is as simple in concept but more powerful. It obtains new state-ofthe-art or substantially improves results on ten reading comprehension and language inference tasks.

Authors

• Zhuosheng Zhang Shanghai Jiao Tong University
• Yuwei Wu Beijing Institute of Technology Shenzhen MSU-BIT University
• Hai Zhao Department of Computer Science and Engineering, Shanghai Jiao Tong University Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3
• Zuchao Li Wuhan University
• Shuailiang Zhang
• Xi Zhou Hainan University
• Xiang Zhou

Keywords

No keywords are indexed for this paper.