Syntax-Directed Attention for Neural Machine Translation

Attention mechanism, including global attention and local attention, plays a key role in neural machine translation (NMT). Global attention attends to all source words for word prediction. In comparison, local attention selectively looks at fixed-window source words. However, alignment weights for the current target word often decrease to the left and right by linear distance centering on the aligned source position and neglect syntax distance constraints. In this paper, we extend the local attention with syntax-distance constraint, which focuses on syntactically related source words with the predicted target word to learning a more effective context vector for predicting translation. Moreover, we further propose a double context NMT architecture, which consists of a global context vector and a syntax-directed context vector from the global attention, to provide more translation performance for NMT from source representation. The experiments on the largescale Chinese-to-English and English-to-German translation tasks show that the proposed approach achieves a substantial and significant improvement over the baseline system.

Authors

• Kehai Chen Harbin Institute of Technology (Shenzhen)
• Rui Wang National Key Laboratory of Space Integrated Information System, Institute of Software Chinese Academy of Sciences
• Masao Utiyama National Institute of Information and Communications Technology
• Eiichiro Sumita National Institute of Information and Communications Technology
• Tiejun Zhao Harbin Institute of Technology

Keywords

No keywords are indexed for this paper.