Multi-Scale Self-Attention for Text Classification

In this paper, we introduce the prior knowledge, multi-scale structure, into self-attention modules. We propose a Multi- Scale Transformer which uses multi-scale multi-head selfattention to capture features from different scales. Based on the linguistic perspective and the analysis of pre-trained Transformer (BERT) on a huge corpus, we further design a strategy to control the scale distribution for each layer. Results of three different kinds of tasks (21 datasets) show our Multi-Scale Transformer outperforms the standard Transformer consistently and significantly on small and moderate size datasets.

Authors

• Qipeng Guo
• Xipeng Qiu School of Computer Science, Fudan University
• Pengfei Liu
• Xiangyang Xue Fudan University
• Zheng Zhang

Keywords

No keywords are indexed for this paper.