InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-language models is challenging due to the rich input distributions and task diversity resulting from the additional visual input. Although vision-language pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models. We gather 26 publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e. g. , 90. 7% accuracy on ScienceQA questions with image contexts). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-source.

Authors

• Wenliang Dai Hong Kong University of Science and Technology
• Junnan Li Salesforce Research
• Dongxu Li
• Anthony Tiong
• Junqi Zhao
• Weisheng Wang
• Boyang Li School of Computer Science and Engineering, Nanyang Technological University, Singapore
• Pascale N Fung
• Steven Hoi

Keywords

No keywords are indexed for this paper.