A Multi-task Learning Approach for Image Captioning

In this paper, we propose a Multi-task Learning Approach for Image Captioning (MLAIC ), motivated by the fact that humans have no difficulty performing such task because they possess capabilities of multiple domains. Specifically, MLAIC consists of three key components: (i) A multi-object classification model that learns rich category-aware image representations using a CNN image encoder; (ii) A syntax generation model that learns better syntax-aware LSTM based decoder; (iii) An image captioning model that generates image descriptions in text, sharing its CNN encoder and LSTM decoder with the object classification task and the syntax generation task, respectively. In particular, the image captioning model can benefit from the additional object categorization and syntax knowledge. To verify the effectiveness of our approach, we conduct extensive experiments on MS-COCO dataset. The experimental results demonstrate that our model achieves impressive results compared to other strong competitors.

Authors

• Wei Zhao
• Benyou Wang The Chinese University of Hong Kong, Shenzhen
• Jianbo Ye
• Min Yang Shenzhen Institutes of Advanced Technology Chinese Academy of Sciences, Shenzhen, China
• Zhou Zhao Zhejiang University
• Ruotian Luo
• Yu Qiao Shanghai Artificial Intelligence Laboratory

Keywords

• Computer Vision: Language and Vision
• Humans and AI: Cognitive Modeling
• Machine Learning: Transfer, Adaptation, Multi-task Learning