USB: A Unified Semi-supervised Learning Benchmark for Classification

Semi-supervised learning (SSL) improves model generalization by leveraging massive unlabeled data to augment limited labeled samples. However, currently, popular SSL evaluation protocols are often constrained to computer vision (CV) tasks. In addition, previous work typically trains deep neural networks from scratch, which is time-consuming and environmentally unfriendly. To address the above issues, we construct a Unified SSL Benchmark (USB) for classification by selecting 15 diverse, challenging, and comprehensive tasks from CV, natural language processing (NLP), and audio processing (Audio), on which we systematically evaluate the dominant SSL methods, and also open-source a modular and extensible codebase for fair evaluation of these SSL methods. We further provide the pre-trained versions of the state-of-the-art neural models for CV tasks to make the cost affordable for further tuning. USB enables the evaluation of a single SSL algorithm on more tasks from multiple domains but with less cost. Specifically, on a single NVIDIA V100, only 39 GPU days are required to evaluate FixMatch on 15 tasks in USB while 335 GPU days (279 GPU days on 4 CV datasets except for ImageNet) are needed on 5 CV tasks with TorchSSL.

Authors

• Yidong Wang
• Hao Chen The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, CASIA & School of Artificial Intelligence, UCAS
• Yue Fan
• Wang Sun
• Ran Tao Beijing Institute of Technology
• Wenxin Hou
• Renjie Wang
• Linyi Yang
• Zhi Zhou Nanjing University
• Lan-Zhe Guo Nanjing University
• Heli Qi
• Zhen Wu
• Yu-Feng Li Nanjing University
• Satoshi Nakamura
• Wei Ye Tongji University
• Marios Savvides
• Bhiksha Raj Carnegie Mellon University
• Takahiro Shinozaki
• Bernt Schiele Max Planck Institute for Informatics, Saarland Informatics Campus
• Jindong Wang
• Xing Xie Microsoft Research Asia
• Yue Zhang Monash University

Keywords

No keywords are indexed for this paper.