LangGPS: Language Separability Guided Data Pre-Selection for Joint Multilingual Instruction Tuning

Joint multilingual instruction tuning is a widely adopted approach to improve the multilingual instruction-following ability and downstream performance of large language models (LLMs), but the resulting multilingual capability remains highly sensitive to the composition and selection of the training data. Existing selection methods, often based on features like text quality, diversity, or task relevance, typically overlook the intrinsic linguistic structure of multilingual data. In this paper, we propose LangGPS, a lightweight two-stage pre-selection framework guided by language separability—a signal that quantifies how well samples in different languages can be distinguished in the model’s representation space. LangGPS first filters training data based on separability scores and then refines the subset using existing selection methods. Extensive experiments across six benchmarks and 22 languages demonstrate that applying LangGPS on top of existing selection methods improves their effectiveness and generalizability in multilingual training, especially for understanding tasks and low-resource languages. Further analysis reveals that highly separable samples facilitate the formation of clearer language boundaries and support faster adaptation, while low-separability samples tend to function as bridges for cross-lingual alignment. Besides, we also find that language separability can serves as an effective signal for multilingual curriculum learning, where interleaving samples with diverse separability levels yields stable and generalizable gains. Together, we hope our work offers a new perspective on data utility in multilingual contexts and support the development of more linguistically informed LLMs.

Authors

• Yangfan Ye Harbin Institute of Technology
• Xiaocheng Feng Research Center for SCIR, Harbin Institute of Technology, Harbin, China
• Xiachong Feng University of Hong Kong
• Lei Huang Tianjin University
• Weitao Ma Harbin Institute of Technology
• Qichen Hong Huawei Technologies Ltd.
• Yunfei Lu Huawei Technologies Co., Ltd
• Duyu Tang Huawei Inc.
• Dandan Tu Huawei Inc.
• Bing Qin Harbin Institute of Technology

Keywords

No keywords are indexed for this paper.