Perception Encoder: The best visual embeddings are not at the output of the network

We introduce Perception Encoder (PE), a family of state-of-the-art vision encoders for image and video understanding. Traditionally, vision encoders have relied on a variety of pretraining objectives, each excelling at different downstream tasks. Surprisingly, after scaling a carefully tuned image pretraining recipe and refining with a robust video data engine, we find that contrastive vision-language training alone can produce strong, general embeddings for all of these downstream tasks. There is only one caveat: these embeddings are hidden within the intermediate layers of the network. To draw them out, we introduce two alignment methods: language alignment for multimodal language modeling, and spatial alignment for dense prediction. Together, our PE family of models achieves state-of-the-art results on a wide variety of tasks, including zero-shot image and video classification and retrieval; document, image, and video Q&A; and spatial tasks such as detection, tracking, and depth estimation. We release our models, code, and novel dataset of synthetically and human-annotated videos: https: //github. com/facebookresearch/perception_models

Authors

• Daniel Bolya
• Po-Yao Huang
• Peize Sun
• Jang Hyun Cho
• Andrea Madotto
• Chen Wei Xiaomi AI Lab
• Tengyu Ma Dalian University of Technology
• Jiale Zhi
• Jathushan Rajasegaran
• Hanoona Bangalath
• Junke Wang
• Marco Monteiro
• Hu Xu
• Shiyu Dong
• Nikhila Ravi
• Shang-Wen Li
• Piotr Dollar
• Christoph Feichtenhofer

Keywords

No keywords are indexed for this paper.