Finding Sparse Structures for Domain Specific Neural Machine Translation

Neural machine translation often adopts the fine-tuning approach to adapt to specific domains. However, nonrestricted fine-tuning can easily degrade on the general domain and over-fit to the target domain. To mitigate the issue, we propose PRUNE-TUNE, a novel domain adaptation method via gradual pruning. It learns tiny domain-specific sub-networks during fine-tuning on new domains. PRUNE-TUNE alleviates the over-fitting and the degradation problem without model modification. Furthermore, PRUNE-TUNE is able to sequentially learn a single network with multiple disjoint domainspecific sub-networks for multiple domains. Empirical experiment results show that PRUNE-TUNE outperforms several strong competitors in the target domain test set without sacrificing the quality on the general domain in both single and multi-domain settings. The source code and data are available at https: //github. com/ohlionel/Prune-Tune.

Authors

• Jianze Liang
• Chengqi Zhao ByteDance AI Lab
• Mingxuan Wang Bytedance Seed, China
• Xipeng Qiu School of Computer Science, Fudan University
• Lei Li Cleveland State University

Keywords

No keywords are indexed for this paper.