Boosting Knowledge Utilization in Multimodal Large Language Models via Adaptive Logits Fusion and Attention Reallocation

Despite their recent progress, Multimodal Large Language Models (MLLMs) often struggle in knowledge-intensive tasks due to the limited and outdated parametric knowledge acquired during training. Multimodal Retrieval Augmented Generation addresses this issue by retrieving contextual knowledge from external databases, thereby enhancing MLLMs with expanded knowledge sources. However, existing MLLMs often fail to fully leverage the retrieved contextual knowledge for response generation. We examine representative MLLMs and identify two major causes, namely, attention bias toward different tokens and knowledge conflicts between parametric and contextual knowledge. To this end, we design Adaptive Logits Fusion and Attention Reallocation (ALFAR), a training-free and plug-and-play approach that improves MLLM responses by maximizing the utility of the retrieved knowledge. Specifically, ALFAR tackles the challenges from two perspectives. First, it alleviates attention bias by adaptively shifting attention from visual tokens to relevant context tokens according to query-context relevance. Second, it decouples and weights parametric and contextual knowledge at output logits, mitigating conflicts between the two types of knowledge. As a plug-and-play method, ALFAR achieves superior performance across diverse datasets without requiring additional training or external tools. Extensive experiments over multiple MLLMs and benchmarks show that ALFAR consistently outperforms the state-of-the-art by large margins. Our code and data are available at https: //github. com/Lackel/ALFAR.

Authors

• Wenbin An Xi'an Jiaotong University National Engineering Laboratory for Big Data Analytics
• Jiahao Nie Nanyang Technological University
• Feng Tian Institute of Software, Chinese Academy of Sciences University of Chinese Academy of Sciences
• Haonan Lin School of Computer Science and Technology, MOEKLINNS, Xi’an Jiaotong University
• Mingxiang Cai
• Yaqiang Wu Lenovo Research
• Qianying Wang Lenovo Research
• Xiaoqin Zhang Wenzhou University
• Shijian Lu Nanyang Technological University, Singapore

Keywords

No keywords are indexed for this paper.