RAC: Request-adaptive Configuration for Efficient Speculative Decoding

Speculative decoding enhances the inference efficiency of large language models by employing a lightweight draft model to generate candidate tokens, which are then verified in parallel by the target model. However, existing approaches typically use fixed speculative configurations—such as the draft model and speculative length—across similar requests, neglecting semantic and structural differences. This limits acceleration potential and reduces adaptability to diverse, dynamic real-world scenarios. To address this, we propose a reinforcement learning-based method called Request- AdaptiveConfigurationselection(RAC). Byformulatingspeculative configuration selection as a Markov decision process, RAC dynamically determines the optimal draft model and speculative length for each incoming request. It integrates static request features with historical execution feedback to enable fine-grained, request-level inference optimization. Experiments on various text generation benchmarks demonstrate that RAC achieves maximum speedups of 2. 02× and 1. 37× over autoregressive decoding and vanilla speculative decoding, respectively.

Authors

• Jinhao Sheng School of Intelligent Medicine, China Medical University
• Xiao Huang School of Artificial Intelligence, Shanghai Jiao Tong University
• Hongxin Wei Department of Statistics and Data Science, Southern University of Science and Technology
• Feng Zhou Center for Applied Statistics and School of Statistics, Renmin University of China

Keywords

• LargeLanguageModel
• SpeculativeDecoding
• ReinforcementLearning