GloTok: Global Perspective Tokenizer for Image Reconstruction and Generation

Existing state-of-the-art image tokenization methods leverage diverse semantic features from pre-trained vision models for additional supervision, to expand the distribution of latent representations and thereby improve the quality of image reconstruction and generation. These methods employ a locally supervised approach for semantic supervision, which limits the uniformity of semantic distribution. However, VA-VAE proves that a more uniform feature distribution yields better generation performance. In this work, we introduce a Global Perspective Tokenizer (GloTok), which utilizes global relational information to model a more uniform semantic distribution of tokenized features. Specifically, a codebook-wise histogram relation learning method is proposed to transfer the semantics, which are modeled by pre-trained models on the entire dataset, to the semantic codebook. Then, we design a residual learning module which recovers the fine-grained details to minimize the reconstruction error caused by quantization. Through the above design, GloTok delivers more uniformly distributed semantic latent representations, which facilitates the training of autoregressive (AR) models for generating high-quality images without requiring direct access to pre-trained models during the training process. Experiments on the standard ImageNet-1k benchmark clearly show that our proposed method achieves state-of-the-art reconstruction performance and generation quality.

Authors

• Xuan Zhao
• Zhongyu Zhang Tencent Youtu Lab
• Yuge Huang Youtu Lab, Tencent
• Yuxi Mi Fudan University
• Guodong Mu Youtu Lab, Tencent
• Shouhong Ding Youtu Lab, Tencent
• Jun Wang
• Rizen Guo Tencent WeChat Pay Lab33
• Shuigeng Zhou Fudan University

Keywords

No keywords are indexed for this paper.