SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs

Multimodal Large Language Models (MLLMs) typically process a large number of visual tokens, leading to considerable computational overhead, even though many of these tokens are redundant. Existing visual token pruning methods primarily focus on selecting the most salient tokens based on attention scores, resulting in the semantic incompleteness of the selected tokens. In this paper, we propose a novel visual token pruning strategy, called S aliency- C overage O riented token P runing for E fficient MLLMs (SCOPE), to jointly model both the saliency and coverage of the selected visual tokens to better preserve semantic completeness. Specifically, we introduce a set-coverage for a given set of selected tokens, computed based on the token relationships. We then define a token-coverage gain for each unselected token, quantifying how much additional coverage would be obtained by including it. By integrating the saliency score into the token-coverage gain, we propose our SCOPE score and iteratively select the token with the highest SCOPE score. We conduct extensive experiments on multiple vision-language understanding benchmarks using the LLaVA-1. 5 and LLaVA-Next models. Experimental results demonstrate that our method consistently outperforms prior approaches.

Authors

• Jinhong Deng
• Wen Li Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China School of Computer Science and Engineering, University of Electronic Science and Technology of China
• Joey Tianyi Zhou Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore
• Yang He

Keywords

No keywords are indexed for this paper.