TOP-RL: Task-Optimized Progressive Token Pruning with Reinforcement Learning for Vision Language Models

In recent years, Large Vision-Language Models (LVLMs) have significantly advanced multimodal tasks. However, their inference requires intensive processing of numerous visual tokens and incurs substantial computational overhead. Existing methods typically compress visual tokens either at the input stage or in early model layers, ignoring variations across tasks and depths. To address these limitations, we introduce TOP-RL, a Task-Optimized Progressive token pruning framework based on Reinforcement Learning. TOP-RL formulates visual token pruning as a multi-stage Markov Decision Process (MDP). It employs an agent trained with dense and fine-grained reward signals to progressively generate differentiable binary masks. This enables TOP-RL to adaptively select crucial visual tokens tailored to each task, effectively balancing accuracy and computational efficiency. Extensive experiments on leading multimodal datasets and advanced LVLMs validate that TOP-RL effectively learns task-optimized pruning policies, significantly boosting inference efficiency while preserving robust performance. For instance, LLaVA-NeXT equipped with TOP-RL achieves a 1.9x speedup in inference time and a 9.3x reduction in FLOPs, with 96% performance preserved.

Authors

• Hengyi Wang University of Science and Technology of China
• Weiying Xie Xidian University
• Hui Jiang Shanghai Institution of Pancreatic Disease
• Yaotao Wei Beijing Institute of Technology
• Kai Jiang Xidian University
• Mingxiang Cao Xidian University
• Chenhe Hao State Key Laboratory of Integrated Services Networks, Xidian University
• Leyuan Fang Hunan University

Keywords

No keywords are indexed for this paper.