Retrieval-Augmented Perception: High-resolution Image Perception Meets Visual RAG

High-resolution (HR) image perception remains a key challenge in multimodal large language models (MLLMs). To drive progress beyond the limits of heuristic methods, this paper advances HR perception capabilities of MLLMs by harnessing cutting-edge long-context techniques such as retrieval-augmented generation (RAG). Towards this end, this paper presents the first study exploring the use of RAG to address HR perception challenges. Specifically, we propose Retrieval-Augmented Perception (RAP), a training-free framework that retrieves and fuses relevant image crops while preserving spatial context using the proposed Spatial-Awareness Layout. To accommodate different tasks, the proposed Retrieved-Exploration Search (RE-Search) dynamically selects the optimal number of crops based on model confidence and retrieval scores. Experimental results on HR benchmarks demonstrate the significant effectiveness of RAP, with LLaVA-v1. 5-13B achieving a 43% improvement on $V^*$ Bench and 19% on HR-Bench. Code is available at https: //github. com/DreamMr/RAP.

Authors

• Wenbin Wang
• Yongcheng Jing
• Liang Ding 0006
• Yingjie Wang
• Li Shen 0008
• Yong Luo 0002
• Bo Du 0001
• Dacheng Tao

Keywords

• Multimodal Large Language Models
• High-resolution Image Perception