UniADS: Universal Architecture-Distiller Search for Distillation Gap

In this paper, we present UniADS, the first Universal Architecture-Distiller Search framework for co-optimizing student architecture and distillation policies. Teacher-student distillation gap limits the distillation gains. Previous approaches seek to discover the ideal student architecture while ignoring distillation settings. In UniADS, we construct a comprehensive search space encompassing an architectural search for student models, knowledge transformations in distillation strategies, distance functions, loss weights, and other vital settings. To efficiently explore the search space, we utilize the NSGA-II genetic algorithm for better crossover and mutation configurations and employ the Successive Halving algorithm for search space pruning, resulting in improved search efficiency and promising results. Extensive experiments are performed on different teacher-student pairs using CIFAR-100 and ImageNet datasets. The experimental results consistently demonstrate the superiority of our method over existing approaches. Furthermore, we provide a detailed analysis of the search results, examining the impact of each variable and extracting valuable insights and practical guidance for distillation design and implementation.

Authors

• Liming Lu School of Cyber Science and Engineering, Nanjing University of Science and Technology
• Zhenghan Chen Microsoft
• Xiaoyu Lu School of Cyber Science and Engineering, Nanjing University of Science and Technology
• Yihang Rao School of Cyber Science and Engineering, Nanjing University of Science and Technology
• Lujun Li The Hong Kong University of Science and Technology
• Shuchao Pang School of Cyber Science and Engineering, Nanjing University of Science and Technology School of Computing, Macquarie University

Keywords

• CV: Applications
• DMKM: Graph Mining, Social Network Analysis & Community
• ML: Deep Learning Algorithms
• ML: Deep Learning Theory
• ML: Semi-Supervised Learning
• ML: Transfer, Domain Adaptation, Multi-Task Learning
• ML: Transparent, Interpretable, Explainable ML
• ML: Unsupervised & Self-Supervised Learning