Dynamic Grained Encoder for Vision Transformers

Transformers, the de-facto standard for language modeling, have been recently applied for vision tasks. This paper introduces sparse queries for vision transformers to exploit the intrinsic spatial redundancy of natural images and save computational costs. Specifically, we propose a Dynamic Grained Encoder for vision transformers, which can adaptively assign a suitable number of queries to each spatial region. Thus it achieves a fine-grained representation in discriminative regions while keeping high efficiency. Besides, the dynamic grained encoder is compatible with most vision transformer frameworks. Without bells and whistles, our encoder allows the state-of-the-art vision transformers to reduce computational complexity by 40%-60% while maintaining comparable performance on image classification. Extensive experiments on object detection and segmentation further demonstrate the generalizability of our approach. Code is available at https: //github. com/StevenGrove/vtpack.

Authors

• Lin Song
• Songyang Zhang University of Louisiana at Lafeyette
• Songtao Liu Megvii(Face++) Inc
• Zeming Li MEGVII Technology
• Xuming He ShanghaiTech University Shanghai Engineering Research Center of Intelligent Vision and Imaging
• Hongbin Sun
• Jian Sun Xi'an Jiaotong University Pazhou Laboratory (Huangpu), China Peng Cheng Laboratory, China
• Nanning Zheng National Key Laboratory of Human-Machine Hybrid Augmented Intelligence National Engineering Research Center of Visual Information and Applications Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University

Keywords

No keywords are indexed for this paper.