Author name cluster

Haoqi Fan

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

4 papers

1 author row

NeurIPS Conference 2024 Conference Paper

Classification Done Right for Vision-Language Pre-Training

Zilong Huang
Qinghao Ye
Bingyi Kang
Jiashi Feng
Haoqi Fan

We introduce SuperClass, a super simple classification method for vision-language pre-training on image-text data. Unlike its contrastive counterpart CLIP who contrast with a text encoder, SuperClass directly utilizes tokenized raw text as supervised classification labels, without the need for additional text filtering or selection. Due to the absence of the text encoding as contrastive target, SuperClass does not require a text encoder and does not need to maintain a large batch size as CLIP does. SuperClass demonstrated superior performance on various downstream tasks, including classic computer vision benchmarks and vision language downstream tasks. We further explored the scaling behavior of SuperClass on model size, training length, or data size, and reported encouraging results and comparisons to CLIP. https: //github. com/x-cls/superclass

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

MAViL: Masked Audio-Video Learners

Po-Yao Huang
Vasu Sharma
Hu Xu
Chaitanya Ryali
Haoqi Fan
Yanghao Li
Shang-Wen Li
Gargi Ghosh

We present Masked Audio-Video Learners (MAViL) to learn audio-visual representations with three complementary forms of self-supervision: (1) reconstructing masked raw audio and video inputs, (2) intra-modal and inter-modal contrastive learning with masking, and (3) self-training to predict aligned and contextualized audio-video representations learned from the first two objectives. Empirically, MAViL achieves state-of-the-art audio-video classification performance on AudioSet (53. 3 mAP) and VGGSound (67. 1\% accuracy), surpassing recent self-supervised models and supervised models that utilize external labeled data. Notably, pre-training with MAViL not only enhances performance in multimodal classification and retrieval tasks, but it also improves the representations of each modality in isolation, without relying on information from the other modality during uni-modal fine-tuning or inference. The code and models are available at https: //github. com/facebookresearch/MAViL.

PDF Details

NeurIPS Conference 2022 Conference Paper

Masked Autoencoders As Spatiotemporal Learners

Christoph Feichtenhofer
Haoqi Fan
Yanghao Li
Kaiming He

This paper studies a conceptually simple extension of Masked Autoencoders (MAE) to spatiotemporal representation learning from videos. We randomly mask out spacetime patches in videos and learn an autoencoder to reconstruct them in pixels. Interestingly, we show that our MAE method can learn strong representations with almost no inductive bias on spacetime (only except for patch and positional embeddings), and spacetime-agnostic random masking performs the best. We observe that the optimal masking ratio is as high as 90% (vs. 75% on images), supporting the hypothesis that this ratio is related to information redundancy of the data. A high masking ratio leads to a large speedup, e. g. , > 4x in wall-clock time or even more. We report competitive results on several challenging video datasets using vanilla Vision Transformers. We observe that MAE can outperform supervised pre-training by large margins. We further report encouraging results of training on real-world, uncurated Instagram data. Our study suggests that the general framework of masked autoencoding (BERT, MAE, etc. ) can be a unified methodology for representation learning with minimal domain knowledge.

PDF Details

AAAI Conference 2018 Conference Paper

Efficient K-Shot Learning With Regularized Deep Networks

Donghyun Yoo
Haoqi Fan
Vishnu Boddeti
Kris Kitani

Feature representations from pre-trained deep neural networks have been known to exhibit excellent generalization and utility across a variety of related tasks. Fine-tuning is by far the simplest and most widely used approach that seeks to exploit and adapt these feature representations to novel tasks with limited data. Despite the effectiveness of ﬁne-tuning, it is often sub-optimal and requires very careful optimization to prevent severe over-ﬁtting to small datasets. The problem of sub-optimality and over-ﬁtting, is due in part to the large number of parameters (≈ 106 ) used in a typical deep convolutional neural network. To address these problems, we propose a simple yet effective regularization method for ﬁne-tuning pre-trained deep networks for the task of k-shot learning. To prevent overﬁtting, our key strategy is to cluster the model parameters while ensuring intra-cluster similarity and intercluster diversity of the parameters, effectively regularizing the dimensionality of the parameter search space. In particular, we identify groups of neurons within each layer of a deep network that share similar activation patterns. When the network is to be ﬁne-tuned for a classiﬁcation task using only k examples, we propagate a single gradient to all of the neuron parameters that belong to the same group. The grouping of neurons is non-trivial as neuron activations depend on the distribution of the input data. To efﬁciently search for optimal groupings conditioned on the input data, we propose a reinforcement learning search strategy using recurrent networks to learn the optimal group assignments for each network layer. Experimental results show that our method can be easily applied to several popular convolutional neural networks and improve upon other state-of-the-art ﬁne-tuning based k-shot learning strategies by more than 10% of accuracy.

PDF Details