Retrieval-driven Reasoning for Deliberative Visual Classification

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in visual classification tasks. Existing methods for enhancing VLMs on this task often rely heavily on direct category-to-image matching, which limits generalization and results in suboptimal performance. In addition, these methods provide no understanding of why a specific category is chosen. To address these limitations, we introduce a new deliberative visual classification task that decomposes the classification process into multiple deliberative steps and leverages Large Language Models (LLMs) to perform explicit reasoning before the final decision. Specifically, we propose a Retrieval-driven Reasoning model (RdR) with two components, i.e., retrieval database construction and deliberative category prediction. The first component leverages LLMs to extract category-relevant descriptors and constructs a retrieval database for effective image–descriptor matching. The second component facilitates multiple deliberative steps and performs explicit reasoning based on the retrieved descriptors to augment the category prediction. Extensive experiments on multiple datasets demonstrate that RdR consistently outperforms strong baselines, highlighting its robustness and generalization ability.

Authors

• Jianye Xie College of Computer Science and Technology, China University of Petroleum (East China), China Shandong Key Laboratory of Intelligent Oil and Gas Industrial Software, China
• Lianyong Qi China University of Petroleum Nanjing University
• Fan Wang Zhejiang University
• Anqi Wang College of Computer Science and Technology, China University of Petroleum (East China), China Shandong Key Laboratory of Intelligent Oil and Gas Industrial Software, China
• Wenjuan Gong College of Computer Science and Technology, China University of Petroleum (East China), China Shandong Key Laboratory of Intelligent Oil and Gas Industrial Software, China
• Danxin Wang College of Computer Science and Technology, China University of Petroleum (East China), China Shandong Key Laboratory of Intelligent Oil and Gas Industrial Software, China
• Wanchun Dou Nanjing University, China
• Yang Cao University of Science and Technology of China
• Shichao Pei University of Massachusetts Boston
• Xiaokang Zhou Faculty of Business Data Science, Kansai University, Japan RIKEN Center for Advanced Intelligence Project, Japan

Keywords

No keywords are indexed for this paper.