Foundation Model is Efficient Multimodal Multitask Model Selector

This paper investigates an under-explored but important problem: given a collection of pre-trained neural networks, predicting their performance on each multi-modal task without fine-tuning them, such as image recognition, referring, captioning, visual question answering, and text question answering. A brute-force approach is to finetune all models on all target datasets, bringing high computational costs. Although recent-advanced approaches employed lightweight metrics to measure models’ transferability, they often depend heavily on the prior knowledge of a single task, making them inapplicable in a multi-modal multi-task scenario. To tackle this issue, we propose an efficient multi-task model selector (EMMS), which employs large-scale foundation models to transform diverse label formats such as categories, texts, and bounding boxes of different downstream tasks into a unified noisy label embedding. EMMS can estimate a model’s transferability through a simple weighted linear regression, which can be efficiently solved by an alternating minimization algorithm with a convergence guarantee. Extensive experiments on 5 downstream tasks with 24 datasets show that EMMS is fast, effective, and generic enough to assess the transferability of pre-trained models, making it the first model selection method in the multi-task scenario. For instance, compared with the state- of-the-art method LogME enhanced by our label embeddings, EMMS achieves 9. 0%, 26. 3%, 20. 1%, 54. 8%, 12. 2% performance gain on image recognition, referring, captioning, visual question answering, and text question answering, while bringing 5. 13×, 6. 29×, 3. 59×, 6. 19×, and 5. 66× speedup in wall-clock time, respectively. The code is available at https: //github. com/OpenGVLab/Multitask-Model-Selector.

Authors

• Fanqing Meng
• Wenqi Shao The Chinese University of Hong Kong
• zhanglin peng
• Chonghe Jiang
• Kaipeng Zhang Shanghai AI Laboratory
• Yu Qiao Shanghai Artificial Intelligence Laboratory
• Ping Luo The University of Hong Kong

Keywords

No keywords are indexed for this paper.