ClusterFomer: Clustering As A Universal Visual Learner

This paper presents ClusterFormer, a universal vision model that is based on the Clustering paradigm with TransFormer. It comprises two novel designs: 1) recurrent cross-attention clustering, which reformulates the cross-attention mechanism in Transformer and enables recursive updates of cluster centers to facilitate strong representation learning; and 2) feature dispatching, which uses the updated cluster centers to redistribute image features through similarity-based metrics, resulting in a transparent pipeline. This elegant design streamlines an explainable and transferable workflow, capable of tackling heterogeneous vision tasks (i. e. , image classification, object detection, and image segmentation) with varying levels of clustering granularity (i. e. , image-, box-, and pixel-level). Empirical results demonstrate that ClusterFormer outperforms various well-known specialized architectures, achieving 83. 41% top-1 acc. over ImageNet-1K for image classification, 54. 2% and 47. 0% mAP over MS COCO for object detection and instance segmentation, 52. 4% mIoU over ADE20K for semantic segmentation, and 55. 8% PQ over COCO Panoptic for panoptic segmentation. This work aims to initiate a paradigm shift in universal visual understanding and to benefit the broader field.

Authors

• James Liang
• Yiming Cui
• Qifan Wang Meta AI
• Tong Geng
• Wenguan Wang Zhejiang University
• Dongfang Liu

Keywords

No keywords are indexed for this paper.