Learning to Rank for In-Context Example Retrieval

Recent advances in retrieval-based in-context learning (ICL) train the retriever using a classification objective, which categorizes in-context examples (ICEs) into the most useful and the rest based on absolute scores. However, during inference, ICEs are retrieved by score ranking rather than classification — The classification training objective deviates from this test scenario. Hence, in this paper, we propose a novel algorithm that trains a retrieval model by ranking formulation, where the preference rankings between ICEs are given by comparing the likelihood of the LLM generating the correct answer conditioned on each exemplar. By learning to rank, we motivate the retriever to automatically learn diverse rationales why specific examples are more useful for ICL decisions. This addresses the issue that classification models poorly capture broader utility. Experimental results demonstrate the top-1 performance of our proposal across 9 NLP tasks, with ablation studies and case studies further validating the effectiveness of our design. The code can be found in: https: //github. com/2022neo/SeDPO_NIPS25

Authors

• Yuwen Ji Beihang University, China Amap, China
• Luodan Zhang
• Ambyer han
• Haoran Que Sino-French Engineer School, Beihang University, Beijing, China
• Lei Shi State Key Laboratory of Media Convergence and Communication, CUC, Beijing 100024, China State Key Laboratory of Intelligent Game, Yangtze River Delta Research Institute of NPU, Taicang 215400, China
• Wang Chao
• Yue Zhang Monash University

Keywords

No keywords are indexed for this paper.