Exploring Stochastic Autoregressive Image Modeling for Visual Representation

Autoregressive language modeling (ALM) has been successfully used in self-supervised pre-training in Natural language processing (NLP). However, this paradigm has not achieved comparable results with other self-supervised approaches in computer vision (e.g., contrastive learning, masked image modeling). In this paper, we try to find the reason why autoregressive modeling does not work well on vision tasks. To tackle this problem, we fully analyze the limitation of visual autoregressive methods and proposed a novel stochastic autoregressive image modeling (named SAIM) by the two simple designs. First, we serialize the image into patches. Second, we employ the stochastic permutation strategy to generate an effective and robust image context which is critical for vision tasks. To realize this task, we create a parallel encoder-decoder training process in which the encoder serves a similar role to the standard vision transformer focusing on learning the whole contextual information, and meanwhile the decoder predicts the content of the current position so that the encoder and decoder can reinforce each other. Our method significantly improves the performance of autoregressive image modeling and achieves the best accuracy (83.9%) on the vanilla ViT-Base model among methods using only ImageNet-1K data. Transfer performance in downstream tasks also shows that our model achieves competitive performance. Code is available at https://github.com/qiy20/SAIM.

Authors

• Yu Qi MOE Frontier Science Center for Brain Science and Brain-machine Integration, Zhejiang University Affiliated Mental Health Center & Hangzhou Seventh People’s Hospital, Zhejiang University The College of Computer Science and Technology, Zhejiang University State Key Lab of Brain-Machine Intelligence, Zhejiang University
• Fan Yang Nanyang Technological University
• Yousong Zhu Institute of Automation, Chinese Academy of Sciences
• Yufei Liu School of Informatics, Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan, Ministry of Culture and Tourism, Xiamen University Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University
• Liwei Wu SenseTime Research
• Rui Zhao SenseTime Research
• Wei Li

Keywords

• CV: Representation Learning for Vision