Fully Convolutional Transformer with Local-Global Attention

In an attempt to imitate the success of transformers in the field of natural language processing into computer vision tasks, vision transformers (ViTs) have recently gained attention. Performance breakthroughs have been achieved in coarse-grained tasks like classification. However, dense prediction tasks, such as detection, segmentation, and depth estimation, require additional modifications and have been tackled only in an ad-hoc manner, by replacing the convolutional neural network encoder backbone of an existing architecture with a ViT. This study proposes a fully convolutional transformer that can perform both coarse and dense prediction tasks. The proposed architecture is, to the best of our knowledge, the first architecture composed of attention layers, even in the decoder part of the network. This is because our newly proposed local-global attention (LGA) can flexibly perform both downsampling and upsampling of spatial features, which are key operations required for dense prediction. Against existing ViTs on classification tasks, our architecture shows a reasonable trade-off between performance and efficiency. In the depth estimation task, our architecture achieves performance comparable to that of state-of-the-art transformer-based methods.

Authors

• Sihaeng Lee
• Eojindl Yi
• Janghyeon Lee 0001
• Jinsu Yoo
• Honglak Lee
• Seung Hwan Kim

Keywords

• Knowledge engineering
• Architecture
• Memory management
• Estimation
• Computer architecture
• Transformers
• Feature extraction
• Convolutional Neural Network
• Computer Vision
• Classification Task
• Spatial Features
• Depth Estimation
• Vision Transformer
• Field Transformation
• Dense Prediction
• Root Mean Square Error
• Spatial Resolution
• Learning Rate
• Large Amount Of Data
• Local Features
• Spatial Dimensions
• ImageNet
• Design Choices
• Architectural Design
• Channel Dimension
• Translation Invariance
• Global Attention
• Local Attention
• Positional Encoding
• Inductive Bias
• Number Of Heads
• KITTI Dataset
• Transformer Architecture
• Long-range Dependencies
• Random Rotation
• Feature Maps