Arrow Research search

Author name cluster

Yunhe Wang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

66 papers
1 author row

Possible papers

66

AAAI Conference 2026 Conference Paper

GenVidBench: A 6-Million Benchmark for AI-Generated Video Detection

  • Zhenliang Ni
  • Qiangyu Yan
  • Mouxiao Huang
  • Tianning Yuan
  • Yehui Tang
  • Hailin Hu
  • Xinghao Chen
  • Yunhe Wang

The rapid advancement of video generation models has made it increasingly challenging to distinguish AI-generated videos from real ones. This issue underscores the urgent need for effective AI-generated video detectors to prevent the dissemination of false information via such videos. However, the development of high-performance AI-generated video detectors is currently impeded by the lack of large-scale, high-quality datasets specifically designed for generative video detection. To this end, we introduce GenVidBench, a challenging AI-generated video detection dataset with several key advantages: 1) Large-scale video collection: The dataset contains 6.78 million videos and is currently the largest dataset for AI-generated video detection. 2) Cross-Source and Cross-Generator: The cross-source generation reduces the interference of video content on the detection. The cross-generator ensures diversity in video attributes between the training and test sets, preventing them from being overly similar. 3) State-of-the-Art Video Generators: The dataset includes videos from 11 state-of-the-art AI video generators, ensuring that it covers the latest advancements in the field of video generation. These generators ensure that the datasets are not only large in scale but also diverse, aiding in the development of generalized and effective detection models. Additionally, we present extensive experimental results with advanced video classification models. With GenVidBench, researchers can efficiently develop and evaluate AI-generated video detection models.

PDF Details DOI

JBHI Journal 2026 Journal Article

Hierarchical Multi-View Graph Diffusion Weighted Model for Cancer Subtype Identification

  • Yunhe Wang
  • Hang Zhang
  • Zhengyu Du
  • Yanchi Su
  • Xiangtao Li

Accurate cancer subtype identification is crucial for personalized medicine, as it enables precise diagnosis based on molecular characteristics. With the advent of large-scale multi-omics data from various resources, researchers now have unprecedented opportunities to explore cancer subtypes comprehensively. However, the inherent complexity, high dimensionality, and heterogeneity of these datasets present significant statistical and computational challenges, often leading to suboptimal clustering performance when inter-omics heterogeneity is overlooked. To address these challenges, we propose a novel method called the Hierarchical Multi-view Graph Diffusion Weighted (HMGDW) model for cancer subtype identification. Our approach begins with the generation of multiple base clusterings through random feature sampling, effectively mitigating the impact of high dimensionality. These base clusterings are subsequently integrated via a late integration strategy to yield the consensus clustering result. Then, we introduce a graph diffusion weighted mechanism that prioritizes views with the most significant contributions to the unified graph representation. Lastly, we conducted extensive experiments on both generic multi-view datasets and multi-omics cancer multi-omics datasets. The experimental results demonstrate that HMGDW consistently outperforms several state-of-the-art methods, achieving robust and accurate clustering. Additionally, a case study on the acute myeloid leukemia (AML) dataset validates the practical efficacy of our model in identifying clinically relevant subtypes.

Details DOI

EAAI Journal 2026 Journal Article

Learning Region-aware Patch Embedding with mask-text prompted contrastive supervision for vision-language tracking

  • Kang Liu
  • Long Liu
  • Yunhe Wang
  • Pingyan Hu

In recent years, vision-language tracking (VLT) has drawn emerging attention for its capacity to achieve more robust tracking by jointly exploiting various modality references. Nevertheless, most existing methods rely on global cross-modal alignment while overlooking regional semantic variations within the image, diluting target semantics and weakening effective cross-modal interaction. To alleviate this issue, we propose a Region-aware Embedding Vision-Language Tracking framework, termed RaEVLT, which employs mask-text prompted contrastive supervision to achieve region-level alignment and embedded representation. Specifically, for regional-level modeling, we introduce a Region-aware Patch Embedding (RaPE) that partitions the visual input into region-level representations using mask-wise prompt. To enable region-level cross-modal alignment, we develop Mask-text Prompted Contrastive Learning (MTPCL), which constructs multi-level contrastive constraints between textual descriptions and different regions, enhancing regional semantic alignment for learning target-aware representation. In addition, we incorporate a Low-Rank Adaptation (LoRA)-tuned Modal Interaction that adaptively enhances and fuses visual and textual features, thereby facilitating efficient cross-modal interaction for accurate vision-language tracking. Through extensive experiments, we achieve favorable performance on five vision-language tracking benchmarks and verify the effectiveness of our method.

Details DOI

AAAI Conference 2025 Conference Paper

Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts

  • Miao Rang
  • Zhenni Bi
  • Chuanjian Liu
  • Yehui Tang
  • Kai Han
  • Yunhe Wang

Multimodal vision language models (VLMs) have made significant progress with the support of continuously increasing model sizes and data volumes. Running VLMs on edge devices has become a challenge for their widespread application. There are several efficient VLM efforts, but they often sacrifice linguistic capabilities to enhance multimodal abilities, or require extensive training. To address this quandary, we introduce the innovative framework of Efficient Vision Language Models with Elastic Visual Experts (Eve). By strategically incorporating adaptable visual expertise at multiple stages of training, Eve strikes a balance between preserving linguistic abilities and augmenting multimodal capabilities. This balanced approach results in a versatile model with only 1.8B parameters that delivers significant improvements in both multimodal and linguistic tasks. Notably, in configurations below 3B parameters, Eve distinctly outperforms in language benchmarks and achieves state-of-the-art results in VLM Benchmarks. Additionally, its multimodal accuracy outstrips that of the larger 7B LLaVA-1.5 model.

PDF Details DOI

JBHI Journal 2025 Journal Article

Evolving Dual-Directional Multiobjective Feature Selection for High-Dimensional Gene Expression Data

  • Yunhe Wang
  • Zhengyu Du
  • Xiaomin Li
  • Wenyuan Xiao
  • Hongpu Liu
  • Liang Yang

High-dimensional gene expression data has gained considerable attention in diverse medical fields such as disease diagnosis, with the challenges of the dimensionality curse and exponentially growing computation. To analyze the data, feature selection is an essential step by reducing the dimensionality. However, most feature selection algorithms for high-dimensional gene expression data still suffer from low classification and poor generalization ability. An evolutionary algorithm is an effective paradigm for enhancing global search capability in feature selection. Inspired by the evolutionary algorithm Competitive Swarm Optimization, we propose a Multiobjective Dual-directional Competitive Swarm Optimization (MODCSO) method for feature selection from high-dimensional gene expression data. First, we design a competitive swarm optimization algorithm framework based on multi-objective optimization to evolve three objective functions simultaneously. Then, we introduce a dual-directional learning strategy that trains particles within the loser group using two distinct learning strategies. To assess the effectiveness and efficiency of the suggested algorithm, we evaluate MODCSO through extensive experiments on twenty high-dimensional gene expression datasets and three real-world biological datasets. Compared to various leading feature selection algorithms, our proposed algorithm MODCSO exhibits superior competitiveness for the high-dimensional feature selection task. Moreover, we provide other extensive analyses to demonstrate further the robustness and biological interpretability of MODCSO in handling high-dimensional gene expression data.

Details DOI

AAAI Conference 2025 Conference Paper

TinySAM: Pushing the Envelope for Efficient Segment Anything Model

  • Han Shu
  • Wenshuo Li
  • Yehui Tang
  • Yiman Zhang
  • Yihao Chen
  • Houqiang Li
  • Yunhe Wang
  • Xinghao Chen

Recently segment anything model (SAM) has shown powerful segmentation capability and has drawn great attention in computer vision fields. Massive following works have developed various applications based on the pre-trained SAM and achieved impressive performance on downstream vision tasks. However, SAM consists of heavy architectures and requires massive computational capacity, which hinders the further application of SAM on computation constrained edge devices. To this end, in this paper we propose a framework to obtain a tiny segment anything model (TinySAM) while maintaining the strong zero-shot performance. We first propose a full-stage knowledge distillation method with hard prompt sampling and hard mask weighting strategy to distill a lightweight student model. We also adapt the post-training quantization to the prompt-based segmentation task and further reduce the computational cost. Moreover, a hierarchical segmenting everything strategy is proposed to accelerate the everything inference by 2× with almost no performance degradation. With all these proposed methods, our TinySAM leads to orders of magnitude computational reduction and pushes the envelope for efficient segment anything task. Extensive experiments on various zero-shot transfer tasks demonstrate the significantly advantageous performance of our TinySAM against counterpart methods.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

U-REPA: Aligning Diffusion U-Nets to ViTs

  • Yuchuan Tian
  • Hanting Chen
  • Mengyu Zheng
  • Yuchen Liang
  • Chao Xu
  • Yunhe Wang

Representation Alignment (REPA) that aligns Diffusion Transformer (DiT) hidden-states with ViT visual encoders has proven highly effective in DiT training, demonstrating superior convergence properties, but it has not been validated on the canonical diffusion U-Net architecture that shows faster convergence compared to DiTs. However, adapting REPA to U-Net architectures presents unique challenges: (1) different block functionalities necessitate revised alignment strategies; (2) spatial-dimension inconsistencies emerge from U-Net's spatial downsampling operations; (3) space gaps between U-Net and ViT hinder the effectiveness of tokenwise alignment. To encounter these challenges, we propose U-REPA, a representation alignment paradigm that bridges U-Net hidden states and ViT features as follows: Firstly, we propose via observation that due to skip connection, the middle stage of U-Net is the best alignment option. Secondly, we propose upsampling of U-Net features after passing them through MLPs. Thirdly, we observe difficulty when performing tokenwise similarity alignment, and further introduces a manifold loss that regularizes the relative similarity between samples. Experiments indicate that the resulting U-REPA could achieve excellent generation quality and greatly accelerates the convergence speed. With CFG guidance interval, U-REPA could reach FID<1. 5 in 200 epochs or 1M iterations on ImageNet 256 $\times$ 256, and needs only half the total epochs to perform better than REPA under \textit{sd-vae-ft-ema}.

PDF Details

NeurIPS Conference 2024 Conference Paper

Enhancing Large Language Models through Adaptive Tokenizers

  • Mengyu Zheng
  • Hanting Chen
  • Tianyu Guo
  • Chong Zhu
  • Binfan Zheng
  • Chang Xu
  • Yunhe Wang

Tokenizers serve as crucial interfaces between models and linguistic data, substantially influencing the efficacy and precision of large language models (LLMs). Traditional tokenization methods often rely on static frequency-based statistics and are not inherently synchronized with LLM architectures, which may limit model performance. In this study, we propose a simple but effective method to learn tokenizers specifically engineered for seamless integration with LLMs. Initiating with a broad initial vocabulary, we refine our tokenizer by monitoring changes in the model’s perplexity during training, allowing for the selection of a tokenizer that is closely aligned with the model’s evolving dynamics. Through iterative refinement, we develop an optimized tokenizer. Our empirical evaluations demonstrate that this adaptive approach significantly enhances accuracy compared to conventional methods, maintaining comparable vocabulary sizes and affirming its potential to improve LLM functionality.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exiting

  • Fangcheng Liu
  • Yehui Tang
  • Zhenhua Liu
  • Yunsheng Ni
  • Duyu Tang
  • Kai Han
  • Yunhe Wang

Speculative decoding has demonstrated its effectiveness in accelerating the inference of large language models (LLMs) while maintaining an identical sampling distribution. However, the conventional approach of training separate draft model to achieve a satisfactory token acceptance rate can be costly and impractical. In this paper, we propose a novel self-speculative decoding framework \emph{Kangaroo} with \emph{double} early exiting strategy, which leverages the shallow sub-network and the \texttt{LM Head} of the well-trained target LLM to construct a self-drafting model. Then, the self-verification stage only requires computing the remaining layers over the \emph{early-exited} hidden states in parallel. To bridge the representation gap between the sub-network and the full model, we train a lightweight and efficient adapter module on top of the sub-network. One significant challenge that comes with the proposed method is that the inference latency of the self-draft model may no longer be negligible compared to the big model. To boost the token acceptance rate while minimizing the latency of the self-drafting model, we introduce an additional \emph{early exiting} mechanism for both single-sequence and the tree decoding scenarios. Specifically, we dynamically halt the small model's subsequent prediction during the drafting phase once the confidence level for the current step falls below a certain threshold. This approach reduces unnecessary computations and improves overall efficiency. Extensive experiments on multiple benchmarks demonstrate our effectiveness, where Kangaroo achieves walltime speedups up to 2. 04$\times$, outperforming Medusa-1 with 88. 7\% fewer additional parameters. The code for Kangaroo is available at https: //github. com/Equationliu/Kangaroo.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

MemoryFormer : Minimize Transformer Computation by Removing Fully-Connected Layers

  • Ning Ding
  • Yehui Tang
  • Haochen Qin
  • Zhenli Zhou
  • Chao Xu
  • Lin Li
  • Kai Han
  • Heng Liao

In order to reduce the computational complexity of large language models, great efforts have been made to to improve the efficiency of transformer models such as linear attention and flash-attention. However, the model size and corresponding computational complexity are constantly scaled up in pursuit of higher performance. In this work, we present MemoryFormer, a novel transformer architecture which significantly reduces the computational complexity (FLOPs) from a new perspective. We eliminate nearly all the computations of the transformer model except for the necessary computation required by the multi-head attention operation. This is made possible by utilizing an alternative method for feature transformation to replace the linear projection of fully-connected layers. Specifically, we first construct a group of in-memory lookup tables that store a large amount of discrete vectors to replace the weight matrix used in linear projection. We then use a hash algorithm to retrieve a correlated subset of vectors dynamically based on the input embedding. The retrieved vectors combined together will form the output embedding, which provides an estimation of the result of matrix multiplication operation in a fully-connected layer. Compared to conducting matrix multiplication, retrieving data blocks from memory is a much cheaper operation which requires little computations. We train MemoryFormer from scratch and conduct extensive experiments on various benchmarks to demonstrate the effectiveness of the proposed model.

PDF Details DOI

EAAI Journal 2024 Journal Article

Solution of wide and micro background bias in contrastive action representation learning

  • Shuai Liu
  • Zhe Luo
  • Yating Li
  • Yunhe Wang
  • Weina Fu
  • Weiping Ding

In recent years, contrastive learning has made great progress in the field of computer vision, which shows great potential in action representation learning. Current contrastive learning methods usually employ contrastive loss function to learn video motion representation, which attend to capturing the similar background appearance but ignore similar motion information. This problem is called background bias, which restricts the model from exploring motion patterns. The background bias can be divided into wide background bias and micro background bias. The wide background bias refers to the statistically significant background bias, while the micro background bias refers to the background bias directly interacting with the moving object. To tackle these problems, this paper first proposes a semi-negative pair merging foreground–background, which mainly decouples the dynamic factor with obvious motion and the static factor with stable invariants in the video frame sequence. Then the dynamic factor of the original video is fused with other static factors to obtain a random background image where the foreground is more similar than the background to solve the wide background bias. Secondly, a pixel-level motion aware representation decomposition module calculates the pixel-level intensity variations from the feature space of adjacent frames, and these variations are further accumulated to obtain the salient map that focuses on the boundaries of moving objects, so the model attends to the motion pattern rather than the background to solve the micro background bias. Furthermore, a new dual loss function is proposed based on joint wide and micro background bias to better capture both static and dynamic features. The heat map shows that the proposed method effectively can solve background bias and achieve better performance in public datasets compared with other existing methods.

Details DOI

NeurIPS Conference 2024 Conference Paper

Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning

  • Hang Zhou
  • Yehui Tang
  • Haochen Qin
  • Yujie Yang
  • Renren Jin
  • Deyi Xiong
  • Kai Han
  • Yunhe Wang

The efficacy of large language models (LLMs) on downstream tasks usually hinges on instruction tuning, which relies critically on the quality of training data. Unfortunately, collecting high-quality and diverse data is both expensive and time-consuming. To mitigate this issue, we propose a novel Star-Agents framework, which automates the enhancement of data quality across datasets through multi-agent collaboration and assessment. The framework adopts a three-pronged strategy. It initially generates diverse instruction data with multiple LLM agents through a bespoke sampling method. Subsequently, the generated data undergo a rigorous evaluation using a dual-model method that assesses both difficulty and quality. Finaly, the above process evolves in a dynamic refinement phase, where more effective LLMs are prioritized, enhancing the overall data quality. Our empirical studies, including instruction tuning experiments with models such as Pythia and LLaMA, demonstrate the effectiveness of the proposed framework. Optimized datasets have achieved substantial improvements, with an average increase of 12\% and notable gains in specific metrics, such as a 40\% improvement in Fermi, as evidenced by benchmarks like MT-bench, Vicuna bench, and WizardLM testset. Codes will be released soon.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformers

  • Yuchuan Tian
  • Zhijun Tu
  • Hanting Chen
  • Jie Hu
  • Chao Xu
  • Yunhe Wang

Diffusion Transformers (DiTs) introduce the transformer architecture to diffusion tasks for latent-space image generation. With an isotropic architecture that chains a series of transformer blocks, DiTs demonstrate competitive performance and good scalability; but meanwhile, the abandonment of U-Net by DiTs and their following improvements is worth rethinking. To this end, we conduct a simple toy experiment by comparing a U-Net architectured DiT with an isotropic one. It turns out that the U-Net architecture only gain a slight advantage amid the U-Net inductive bias, indicating potential redundancies within the U-Net-style DiT. Inspired by the discovery that U-Net backbone features are low-frequency-dominated, we perform token downsampling on the query-key-value tuple for self-attention and bring further improvements despite a considerable amount of reduction in computation. Based on self-attention with downsampled tokens, we propose a series of U-shaped DiTs (U-DiTs) in the paper and conduct extensive experiments to demonstrate the extraordinary performance of U-DiT models. The proposed U-DiT could outperform DiT-XL with only 1/6 of its computation cost. Codes are available at https: //github. com/YuchuanTian/U-DiT.

PDF Details DOI

TMLR Journal 2023 Journal Article

Complementary Sparsity: Accelerating Sparse CNNs with High Accuracy on General-Purpose Computing Platforms

  • Kang Zhao
  • Yijun Tan
  • Kai Han
  • Ting Hu
  • Hanting Chen
  • Tao Yuan
  • Yunhe Wang
  • Jun Yao

Model sparsity is a promising approach to reducing parameters or FLOPs of convolutional neural networks (CNNs). Compared to unstructured or coarse-grained structured sparsity, fine-grained structured sparsity, e.g., N:M sparse pattern, can achieve a better balance between accuracy and efficiency on general computing platforms like CPUs and GPUs. In particular, the 2:4 sparsity can accelerate CNN inference by 2$\times$ speed and with negligible accuracy drop. However, N:M sparsity needs to be supported by GPU within specific hardware circuits and hardly achieves significant speedups on common GPUs. To accelerate CNNs with general-purposed computing resources and simultaneously retain the model accuracy as much as possible, this paper proposes complementary sparsity (CS). CS denotes that only one weight can be retained for weights spaced at the same distance. On the one hand, CS features high mask flexibility, which is naturally favorable to high model accuracy. Moreover, we propose a CS-specific sparse training method to improve CS-based CNNs' accuracy under high parameter sparsities ($>$75\%). On the other hand, CS itself is memory-access balanced and robust to pattern hyperparameters, which can be utilized to speedup CS-based convolution computation on CPUs and common GPUs. We thus propose a CS convolution parallel computing algorithm that adapts to common GPUs without sparse tensor cores. Experimental results show that compared to other sparsity patterns, the proposed CS can achieve the optimal trade-off in terms of accuracy and latency for CPUs and common GPUs, respectively. Codes will be available at https://gitee.com/mindspore/models/tree/master/research/cv/CS.

PDF Details

TMLR Journal 2023 Journal Article

Deep Plug-and-Play Clustering with Unknown Number of Clusters

  • An Xiao
  • Hanting Chen
  • Tianyu Guo
  • Qinghua Zhang
  • Yunhe Wang

Clustering is an essential task for the purpose that data points can be classified in an unsupervised manner. Most deep clustering algorithms are very effective when given the number of clusters K. However, when K is unknown, finding the appropriate K for these algorithms can be computationally expensive via model-selection criteria, and applying algorithms with an inaccurate K can hardly achieve the state-of-the-art performance. This paper proposes a plug-and-play clustering module to automatically adjust the number of clusters, which can be easily embedded into existing deep parametric clustering methods. By analyzing the goal of clustering, a split-and-merge framework is introduced to reduce the intra-class diversity and increase the inter-class difference, which leverages the entropy between different clusters. Specifically, given an initial clustering number, clusters can be split into sub-clusters or merged into super-clusters and converge to a stable number of K clusters at the end of training. Experiments on benchmark datasets demonstrate that the proposed method can achieve comparable performance with the state-of-the-art works without requiring the number of clusters.

PDF Details

NeurIPS Conference 2023 Conference Paper

GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image

  • Mingjian Zhu
  • Hanting Chen
  • Qiangyu Yan
  • Xudong Huang
  • Guanyu Lin
  • Wei Li
  • Zhijun Tu
  • Hailin Hu

The extraordinary ability of generative models to generate photographic images has intensified concerns about the spread of disinformation, thereby leading to the demand for detectors capable of distinguishing between AI-generated fake images and real images. However, the lack of large datasets containing images from the most advanced image generators poses an obstacle to the development of such detectors. In this paper, we introduce the GenImage dataset, which has the following advantages: 1) Plenty of Images, including over one million pairs of AI-generated fake images and collected real images. 2) Rich Image Content, encompassing a broad range of image classes. 3) State-of-the-art Generators, synthesizing images with advanced diffusion models and GANs. The aforementioned advantages allow the detectors trained on GenImage to undergo a thorough evaluation and demonstrate strong applicability to diverse images. We conduct a comprehensive analysis of the dataset and propose two tasks for evaluating the detection method in resembling real-world scenarios. The cross-generator image classification task measures the performance of a detector trained on one generator when tested on the others. The degraded image classification task assesses the capability of the detectors in handling degraded images such as low-resolution, blurred, and compressed images. With the GenImage dataset, researchers can effectively expedite the development and evaluation of superior AI-generated image detectors in comparison to prevailing methodologies.

PDF Details

NeurIPS Conference 2023 Conference Paper

Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanism

  • Chengcheng Wang
  • Wei He
  • Ying Nie
  • Jianyuan Guo
  • Chuanjian Liu
  • Yunhe Wang
  • Kai Han

In the past years, YOLO-series models have emerged as the leading approaches in the area of real-time object detection. Many studies pushed up the baseline to a higher level by modifying the architecture, augmenting data and designing new losses. However, we find previous models still suffer from information fusion problem, although Feature Pyramid Network (FPN) and Path Aggregation Network (PANet) have alleviated this. Therefore, this study provides an advanced Gatherand-Distribute mechanism (GD) mechanism, which is realized with convolution and self-attention operations. This new designed model named as Gold-YOLO, which boosts the multi-scale feature fusion capabilities and achieves an ideal balance between latency and accuracy across all model scales. Additionally, we implement MAE-style pretraining in the YOLO-series for the first time, allowing YOLOseries models could be to benefit from unsupervised pretraining. Gold-YOLO-N attains an outstanding 39. 9% AP on the COCO val2017 datasets and 1030 FPS on a T4 GPU, which outperforms the previous SOTA model YOLOv6-3. 0-N with similar FPS by +2. 4%. The PyTorch code is available at https: //github. com/huawei-noah/Efficient-Computing/tree/master/Detection/Gold-YOLO, and the MindSpore code is available at https: //gitee. com/mindspore/models/tree/master/research/cv/Gold_YOLO.

PDF Details

NeurIPS Conference 2023 Conference Paper

One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation

  • Zhiwei Hao
  • Jianyuan Guo
  • Kai Han
  • Yehui Tang
  • Han Hu
  • Yunhe Wang
  • Chang Xu

Knowledge distillation (KD) has proven to be a highly effective approach for enhancing model performance through a teacher-student training scheme. However, most existing distillation methods are designed under the assumption that the teacher and student models belong to the same model family, particularly the hint-based approaches. By using centered kernel alignment (CKA) to compare the learned features between heterogeneous teacher and student models, we observe significant feature divergence. This divergence illustrates the ineffectiveness of previous hint-based methods in cross-architecture distillation. To tackle the challenge in distilling heterogeneous models, we propose a simple yet effective one-for-all KD framework called OFA-KD, which significantly improves the distillation performance between heterogeneous architectures. Specifically, we project intermediate features into an aligned latent space such as the logits space, where architecture-specific information is discarded. Additionally, we introduce an adaptive target enhancement scheme to prevent the student from being disturbed by irrelevant information. Extensive experiments with various architectures, including CNN, Transformer, and MLP, demonstrate the superiority of our OFA-KD framework in enabling distillation between heterogeneous architectures. Specifically, when equipped with our OFA-KD, the student models achieve notable performance improvements, with a maximum gain of 8. 0% on the CIFAR-100 dataset and 0. 7% on the ImageNet-1K dataset. PyTorch code and checkpoints can be found at https: //github. com/Hao840/OFAKD.

PDF Details

NeurIPS Conference 2023 Conference Paper

PUe: Biased Positive-Unlabeled Learning Enhancement by Causal Inference

  • Xutao Wang
  • Hanting Chen
  • Tianyu Guo
  • Yunhe Wang

Positive-Unlabeled (PU) learning aims to achieve high-accuracy binary classification with limited labeled positive examples and numerous unlabeled ones. Existing cost-sensitive-based methods often rely on strong assumptions that examples with an observed positive label were selected entirely at random. In fact, the uneven distribution of labels is prevalent in real-world PU problems, indicating that most actual positive and unlabeled data are subject to selection bias. In this paper, we propose a PU learning enhancement (PUe) algorithm based on causal inference theory, which employs normalized propensity scores and normalized inverse probability weighting (NIPW) techniques to reconstruct the loss function, thus obtaining a consistent, unbiased estimate of the classifier and enhancing the model's performance. Moreover, we investigate and propose a method for estimating propensity scores in deep learning using regularization techniques when the labeling mechanism is unknown. Our experiments on three benchmark datasets demonstrate the proposed PUe algorithm significantly improves the accuracy of classifiers on non-uniform label distribution datasets compared to advanced cost-sensitive PU methods. Codes are available at https: //github. com/huawei-noah/Noah-research/tree/master/PUe and https: //gitee. com/mindspore/models/tree/master/research/cv/PUe.

PDF Details

NeurIPS Conference 2023 Conference Paper

Revisit the Power of Vanilla Knowledge Distillation: from Small Scale to Large Scale

  • Zhiwei Hao
  • Jianyuan Guo
  • Kai Han
  • Han Hu
  • Chang Xu
  • Yunhe Wang

The tremendous success of large models trained on extensive datasets demonstrates that scale is a key ingredient in achieving superior results. Therefore, the reflection on the rationality of designing knowledge distillation (KD) approaches for limited-capacity architectures solely based on small-scale datasets is now deemed imperative. In this paper, we identify the small data pitfall that presents in previous KD methods, which results in the underestimation of the power of vanilla KD framework on large-scale datasets such as ImageNet-1K. Specifically, we show that employing stronger data augmentation techniques and using larger datasets can directly decrease the gap between vanilla KD and other meticulously designed KD variants. This highlights the necessity of designing and evaluating KD approaches in the context of practical scenarios, casting off the limitations of small-scale datasets. Our investigation of the vanilla KD and its variants in more complex schemes, including stronger training strategies and different model capacities, demonstrates that vanilla KD is elegantly simple but astonishingly effective in large-scale scenarios. Without bells and whistles, we obtain state-of-the-art ResNet-50, ViT-S, and ConvNeXtV2-T models for ImageNet, which achieve 83. 1%, 84. 3%, and 85. 0% top-1 accuracy, respectively. PyTorch code and checkpoints can be found at https: //github. com/Hao840/vanillaKD.

PDF Details

NeurIPS Conference 2023 Conference Paper

Species196: A One-Million Semi-supervised Dataset for Fine-grained Species Recognition

  • Wei He
  • Kai Han
  • Ying Nie
  • Chengcheng Wang
  • Yunhe Wang

The development of foundation vision models has pushed the general visual recognition to a high level, but cannot well address the fine-grained recognition in specialized domain such as invasive species classification. Identifying and managing invasive species has strong social and ecological value. Currently, most invasive species datasets are limited in scale and cover a narrow range of species, which restricts the development of deep-learning based invasion biometrics systems. To fill the gap of this area, we introduced Species196, a large-scale semi-supervised dataset of 196-category invasive species. It collects over 19K images with expert-level accurate annotations (Species196-L), and 1. 2M unlabeled images of invasive species (Species196-U). The dataset provides four experimental settings for benchmarking the existing models and algorithms, namely, supervised learning, semi-supervised learning and self-supervised pretraining. To facilitate future research on these four learning paradigms, we conduct an empirical study of the representative methods on the introduced dataset. The dataset will be made publicly available at https: //species-dataset. github. io/.

PDF Details

NeurIPS Conference 2023 Conference Paper

Towards Higher Ranks via Adversarial Weight Pruning

  • Yuchuan Tian
  • Hanting Chen
  • Tianyu Guo
  • Chao Xu
  • Yunhe Wang

Convolutional Neural Networks (CNNs) are hard to deploy on edge devices due to its high computation and storage complexities. As a common practice for model compression, network pruning consists of two major categories: unstructured and structured pruning, where unstructured pruning constantly performs better. However, unstructured pruning presents a structured pattern at high pruning rates, which limits its performance. To this end, we propose a Rank-based PruninG (RPG) method to maintain the ranks of sparse weights in an adversarial manner. In each step, we minimize the low-rank approximation error for the weight matrices using singular value decomposition, and maximize their distance by pushing the weight matrices away from its low rank approximation. This rank-based optimization objective guides sparse weights towards a high-rank topology. The proposed method is conducted in a gradual pruning fashion to stabilize the change of rank during training. Experimental results on various datasets and different tasks demonstrate the effectiveness of our algorithm in high sparsity. The proposed RPG outperforms the state-of-the-art performance by 1. 13\% top-1 accuracy on ImageNet in ResNet-50 with 98\% sparsity. The codes are available at https: //github. com/huawei-noah/Efficient-Computing/tree/master/Pruning/RPG and https: //gitee. com/mindspore/models/tree/master/research/cv/RPG.

PDF Details

NeurIPS Conference 2023 Conference Paper

VanillaNet: the Power of Minimalism in Deep Learning

  • Hanting Chen
  • Yunhe Wang
  • Jianyuan Guo
  • Dacheng Tao

At the heart of foundation models is the philosophy of "more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the challenges of optimization and inherent complexity of transformer models call for a paradigm shift towards simplicity. In this study, we introduce VanillaNet, a neural network architecture that embraces elegance in design. By avoiding high depth, shortcuts, and intricate operations like self-attention, VanillaNet is refreshingly concise yet remarkably powerful. Each layer is carefully crafted to be compact and straightforward, with nonlinear activation functions pruned after training to restore the original architecture. VanillaNet overcomes the challenges of inherent complexity, making it ideal for resource-constrained environments. Its easy-to-understand and highly simplified architecture opens new possibilities for efficient deployment. Extensive experimentation demonstrates that VanillaNet delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning. This visionary journey of VanillaNet has significant potential to redefine the landscape and challenge the status quo of foundation model, setting a new path for elegant and effective model design. Pre-trained models and codes are available at https: //github. com/huawei-noah/VanillaNet and https: //gitee. com/mindspore/models/tree/master/research/cv/vanillanet

PDF Details

NeurIPS Conference 2023 Conference Paper

When Visual Prompt Tuning Meets Source-Free Domain Adaptive Semantic Segmentation

  • Xinhong Ma
  • Yiming Wang
  • Hao Liu
  • Tianyu Guo
  • Yunhe Wang

Source-free domain adaptive semantic segmentation aims to adapt a pre-trained source model to the unlabeled target domain without accessing the private source data. Previous methods usually fine-tune the entire network, which suffers from expensive parameter tuning. To avoid this problem, we propose to utilize visual prompt tuning for parameter-efficient adaptation. However, the existing visual prompt tuning methods are unsuitable for source-free domain adaptive semantic segmentation due to the following two reasons: (1) Commonly used visual prompts like input tokens or pixel-level perturbations cannot reliably learn informative knowledge beneficial for semantic segmentation. (2) Visual prompts require sufficient labeled data to fill the gap between the pre-trained model and downstream tasks. To alleviate these problems, we propose a universal unsupervised visual prompt tuning (Uni-UVPT) framework, which is applicable to various transformer-based backbones. Specifically, we first divide the source pre-trained backbone with frozen parameters into multiple stages, and propose a lightweight prompt adapter for progressively encoding informative knowledge into prompts and enhancing the generalization of target features between adjacent backbone stages. Cooperatively, a novel adaptive pseudo-label correction strategy with a multiscale consistency loss is designed to alleviate the negative effect of target samples with noisy pseudo labels and raise the capacity of visual prompts to spatial perturbations. Extensive experiments demonstrate that Uni-UVPT achieves state-of-the-art performance on GTA5 $\to$ Cityscapes and SYNTHIA $\to$ Cityscapes tasks and can serve as a universal and parameter-efficient framework for large-model unsupervised knowledge transfer. Code will be available at https: //gitee. com/mindspore/models/tree/master/research/cv/uni-uvpt and https: //github. com/huawei-noah/noah-research/tree/master/uni-uvpt.

PDF Details

NeurIPS Conference 2022 Conference Paper

A Transformer-Based Object Detector with Coarse-Fine Crossing Representations

  • Zhishan Li
  • Ying Nie
  • Kai Han
  • Jianyuan Guo
  • Lei Xie
  • Yunhe Wang

Transformer-based object detectors have shown competitive performance recently. Compared with convolutional neural networks limited by the relatively small receptive fields, the advantage of transformer for visual tasks is the capacity to perceive long-range dependencies among all image patches, while the deficiency is that the local fine-grained information is not fully excavated. In this paper, we introduce the Coarse-grained and Fine-grained crossing representations to build an efficient Detection Transformer (CFDT). Specifically, we propose a local-global cross fusion module to establish the connection between local fine-grained features and global coarse-grained features. Besides, we propose a coarse-fine aware neck which enables detection tokens to interact with both coarse-grained and fine-grained features. Furthermore, an efficient feature integration module is presented for fusing multi-scale representations from different stages. Experimental results on the COCO dataset demonstrate the effectiveness of the proposed method. For instance, our CFDT achieves 48. 1 AP with 173G FLOPs, which possesses higher accuracy and less computation compared with the state-of-the-art transformer-based detector ViDT. Code will be available at https: //gitee. com/mindspore/models/tree/master/research/cv/CFDT.

PDF Details

NeurIPS Conference 2022 Conference Paper

Accelerating Sparse Convolution with Column Vector-Wise Sparsity

  • Yijun Tan
  • Kai Han
  • Kang Zhao
  • Xianzhi Yu
  • Zidong Du
  • Yunji Chen
  • Yunhe Wang
  • Jun Yao

Weight sparsity is a promising approach to reducing the model size and computation cost of convolutional neural networks (CNNs). Nevertheless, non-zero weights often distribute randomly in sparse CNN models, introducing enormous difficulty in obtaining actual speedup on common hardware (e. g. , GPU) over their dense counterparts. Existing acceleration solutions either require hardware modifications for irregular memory access support or rely on a partially structured sparsity pattern. Neither of these methods is capable of achieving fruitful speedup on convolution layers. In this work, we propose an algorithm-software co-designed sparse convolution based on a novel out-vector-wise (OVW) sparse pattern. Building on the insight that vertical vector integrity can preserve continuous memory access in IM2COL, the OVW pattern treats a $V\times1$ vector as an entirety. To reduce the error caused by sparsity, we propose an equivalent transformation process, i. e. , clustering-based channel permutation, to gather similar rows together. Experimental evaluations demonstrate that our method achieves a $1. 7\times$ and $3. 2\times$ speedup over the SOTA solution and the dense convolution of ResNet50 on NVIDIA V100 at 75\% sparsity, respectively, with only negligible accuracy loss. Moreover, compared to the SOTA solution that achieves speedups only on data with 60\% sparsity or more, our method begins to obtain speedups on data with only 10\% sparsity.

PDF Details

NeurIPS Conference 2022 Conference Paper

BiMLP: Compact Binary Architectures for Vision Multi-Layer Perceptrons

  • Yixing Xu
  • Xinghao Chen
  • Yunhe Wang

This paper studies the problem of designing compact binary architectures for vision multi-layer perceptrons (MLPs). We provide extensive analysis on the difficulty of binarizing vision MLPs and find that previous binarization methods perform poorly due to limited capacity of binary MLPs. In contrast with the traditional CNNs that utilizing convolutional operations with large kernel size, fully-connected (FC) layers in MLPs can be treated as convolutional layers with kernel size $1\times1$. Thus, the representation ability of the FC layers will be limited when being binarized, and places restrictions on the capability of spatial mixing and channel mixing on the intermediate features. To this end, we propose to improve the performance of binary MLP (BiMLP) model by enriching the representation ability of binary FC layers. We design a novel binary block that contains multiple branches to merge a series of outputs from the same stage, and also a universal shortcut connection that encourages the information flow from the previous stage. The downsampling layers are also carefully designed to reduce the computational complexity while maintaining the classification performance. Experimental results on benchmark dataset ImageNet-1k demonstrate the effectiveness of the proposed BiMLP models, which achieve state-of-the-art accuracy compared to prior binary CNNs. The MindSpore code is available at \url{https: //gitee. com/mindspore/models/tree/master/research/cv/BiMLP}.

PDF Details

NeurIPS Conference 2022 Conference Paper

Bridge the Gap Between Architecture Spaces via A Cross-Domain Predictor

  • Yuqiao Liu
  • Yehui Tang
  • Zeqiong Lv
  • Yunhe Wang
  • Yanan Sun

Neural Architecture Search (NAS) can automatically design promising neural architectures without artificial experience. Though it achieves great success, prohibitively high search cost is required to find a high-performance architecture, which blocks its practical implementation. Neural predictor can directly evaluate the performance of neural networks based on their architectures and thereby save much budget. However, existing neural predictors require substantial annotated architectures trained from scratch, which still consume many computational resources. To solve this issue, we propose a Cross-Domain Predictor (CDP), which is trained based on the existing NAS benchmark datasets (e. g. , NAS-Bench-101), but can be used to find high-performance architectures in large-scale search spaces. Particularly, we propose a progressive subspace adaptation strategy to address the domain discrepancy between the source architecture space and the target space. Considering the large difference between two architecture spaces, an assistant space is developed to smooth the transfer process. Compared with existing NAS methods, the proposed CDP is much more efficient. For example, CDP only requires the search cost of 0. 1 GPU Days to find architectures with 76. 9% top-1 accuracy on ImageNet and 97. 51% on CIFAR-10.

PDF Details

NeurIPS Conference 2022 Conference Paper

GhostNetV2: Enhance Cheap Operation with Long-Range Attention

  • Yehui Tang
  • Kai Han
  • Jianyuan Guo
  • Chang Xu
  • Chao Xu
  • Yunhe Wang

Light-weight convolutional neural networks (CNNs) are specially designed for applications on mobile devices with faster inference speed. The convolutional operation can only capture local information in a window region, which prevents performance from being further improved. Introducing self-attention into convolution can capture global information well, but it will largely encumber the actual speed. In this paper, we propose a hardware-friendly attention mechanism (dubbed DFC attention) and then present a new GhostNetV2 architecture for mobile applications. The proposed DFC attention is constructed based on fully-connected layers, which can not only execute fast on common hardware but also capture the dependence between long-range pixels. We further revisit the expressiveness bottleneck in previous GhostNet and propose to enhance expanded features produced by cheap operations with DFC attention, so that a GhostNetV2 block can aggregate local and long-range information simultaneously. Extensive experiments demonstrate the superiority of GhostNetV2 over existing architectures. For example, it achieves 75. 3% top-1 accuracy on ImageNet with 167M FLOPs, significantly suppressing GhostNetV1 (74. 5%) with a similar computational cost. The source code will be available at https: //github. com/huawei-noah/Efficient-AI-Backbones/tree/master/ghostnetv2_pytorch and https: //gitee. com/mindspore/models/tree/master/research/cv/ghostnetv2.

PDF Details

TMLR Journal 2022 Journal Article

GhostSR: Learning Ghost Features for Efficient Image Super-Resolution

  • Ying Nie
  • Kai Han
  • Zhenhua Liu
  • Chuanjian Liu
  • Yunhe Wang

Modern single image super-resolution (SISR) systems based on convolutional neural networks (CNNs) have achieved impressive performance but require huge computational costs. The problem on feature redundancy has been well studied in visual recognition task, but rarely discussed in SISR. Based on the observation that many features in SISR models are also similar to each other, we propose to use shift operation for generating the redundant features (i.e. ghost features). Compared with depth-wise convolution which is time-consuming on GPU-like devices, shift operation can bring a real inference acceleration for CNNs on common hardware. We analyze the benefits of shift operation in SISR and make the shift orientation learnable based on the Gumbel-Softmax trick. Besides, a clustering procedure is explored based on pre-trained models to identify the intrinsic filters for generating corresponding intrinsic features. The ghost features will be generated by moving these intrinsic features along a certain orientation. Finally, the complete output features are constructed by concatenating the intrinsic and ghost features together. Extensive experiments on several benchmark models and datasets demonstrate that both the non-compact and lightweight SISR CNN models embedded with the proposed method can achieve a comparable performance to the baseline models with a large reduction of parameters, FLOPs and GPU inference latency. For example, we reduce the parameters by 46%, FLOPs by 46% and GPU inference latency by 42% of x2 EDSR model with almost lossless performance. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/GhostSR.

PDF Details

NeurIPS Conference 2022 Conference Paper

Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation

  • Zhiwei Hao
  • Jianyuan Guo
  • Ding Jia
  • Kai Han
  • Yehui Tang
  • Chao Zhang
  • Han Hu
  • Yunhe Wang

In the past few years, transformers have achieved promising performance on various computer vision tasks. Unfortunately, the immense inference overhead of most existing vision transformers withholds them from being deployed on edge devices such as cell phones and smart watches. Knowledge distillation is a widely used paradigm for compressing cumbersome architectures into compact students via transferring information. However, most of them are designed for convolutional neural networks (CNNs), which do not fully investigate the character of vision transformers. In this paper, we fully utilize the patch-level information and propose a fine-grained manifold distillation method for transformer-based networks. Specifically, we train a tiny student model to match a pre-trained teacher model in the patch-level manifold space. Then, we decouple the manifold matching loss into three terms with careful design to further reduce the computational costs for the patch relationship. Equipped with the proposed method, a DeiT-Tiny model containing 5M parameters achieves 76. 5\% top-1 accuracy on ImageNet-1k, which is +2. 0\% higher than previous distillation approaches. Transfer learning results on other classification benchmarks and downstream vision tasks also demonstrate the superiority of our method over the state-of-the-art algorithms.

PDF Details

NeurIPS Conference 2022 Conference Paper

Random Normalization Aggregation for Adversarial Defense

  • Minjing Dong
  • Xinghao Chen
  • Yunhe Wang
  • Chang Xu

The vulnerability of deep neural networks has been widely found in various models as well as tasks where slight perturbations on the inputs could lead to incorrect predictions. These perturbed inputs are known as adversarial examples and one of the intriguing properties of them is Adversarial Transfersability, i. e. the capability of adversarial examples to fool other models. Traditionally, this transferability is always regarded as a critical threat to the defense against adversarial attacks, however, we argue that the network robustness can be significantly boosted by utilizing adversarial transferability from a new perspective. In this work, we first discuss the influence of different popular normalization layers on the adversarial transferability, and then provide both empirical evidence and theoretical analysis to shed light on the relationship between normalization types and transferability. Based on our theoretical analysis, we propose a simple yet effective module named Random Normalization Aggregation (RNA) which replaces the batch normalization layers in the networks and aggregates different selected normalization types to form a huge random space. Specifically, a random path is sampled during each inference procedure so that the network itself can be treated as an ensemble of a wide range of different models. Since the entire random space is designed with low adversarial transferability, it is difficult to perform effective attacks even when the network parameters are accessible. We conduct extensive experiments on various models and datasets, and demonstrate the strong superiority of proposed algorithm. The PyTorch code is available at https: //github. com/UniSerj/Random-Norm-Aggregation and the MindSpore code is available at https: //gitee. com/mindspore/models/tree/master/research/cv/RNA.

PDF Details

NeurIPS Conference 2022 Conference Paper

Redistribution of Weights and Activations for AdderNet Quantization

  • Ying Nie
  • Kai Han
  • Haikang Diao
  • Chuanjian Liu
  • Enhua Wu
  • Yunhe Wang

Adder Neural Network (AdderNet) provides a new way for developing energy-efficient neural networks by replacing the expensive multiplications in convolution with cheaper additions (i. e. , L1-norm). To achieve higher hardware efficiency, it is necessary to further study the low-bit quantization of AdderNet. Due to the limitation that the commutative law in multiplication does not hold in L1-norm, the well-established quantization methods on convolutional networks cannot be applied on AdderNets. Thus, the existing AdderNet quantization techniques propose to use only one shared scale to quantize both the weights and activations simultaneously. Admittedly, such an approach can keep the commutative law in the L1-norm quantization process, while the accuracy drop after low-bit quantization cannot be ignored. To this end, we first thoroughly analyze the difference on distributions of weights and activations in AdderNet and then propose a new quantization algorithm by redistributing the weights and the activations. Specifically, the pre-trained full-precision weights in different kernels are clustered into different groups, then the intra-group sharing and inter-group independent scales can be adopted. To further compensate the accuracy drop caused by the distribution difference, we then develop a lossless range clamp scheme for weights and a simple yet effective outliers clamp strategy for activations. Thus, the functionality of full-precision weights and the representation ability of full-precision activations can be fully preserved. The effectiveness of the proposed quantization method for AdderNet is well verified on several benchmarks, e. g. , our 4-bit post-training quantized adder ResNet-18 achieves an 66. 5% top-1 accuracy on the ImageNet with comparable energy efficiency, which is about 8. 5% higher than that of the previous AdderNet quantization methods. Code will be available at https: //gitee. com/mindspore/models/tree/master/research/cv/AdderQuant.

PDF Details

NeurIPS Conference 2022 Conference Paper

Vision GNN: An Image is Worth Graph of Nodes

  • Kai Han
  • Yunhe Wang
  • Jianyuan Guo
  • Yehui Tang
  • Enhua Wu

Network architecture plays a key role in the deep learning-based computer vision system. The widely-used convolutional neural network and transformer treat the image as a grid or sequence structure, which is not flexible to capture irregular and complex objects. In this paper, we propose to represent the image as a graph structure and introduce a new \emph{Vision GNN} (ViG) architecture to extract graph-level feature for visual tasks. We first split the image to a number of patches which are viewed as nodes, and construct a graph by connecting the nearest neighbors. Based on the graph representation of images, we build our ViG model to transform and exchange information among all the nodes. ViG consists of two basic modules: Grapher module with graph convolution for aggregating and updating graph information, and FFN module with two linear layers for node feature transformation. Both isotropic and pyramid architectures of ViG are built with different model sizes. Extensive experiments on image recognition and object detection tasks demonstrate the superiority of our ViG architecture. We hope this pioneering study of GNN on general visual tasks will provide useful inspiration and experience for future research. The PyTorch code is available at \url{https: //github. com/huawei-noah/Efficient-AI-Backbones} and the MindSpore code is available at \url{https: //gitee. com/mindspore/models}.

PDF Details

AAAI Conference 2022 Conference Paper

ZINB-Based Graph Embedding Autoencoder for Single-Cell RNA-Seq Interpretations

  • Zhuohan Yu
  • Yifu Lu
  • Yunhe Wang
  • Fan Tang
  • Ka-Chun Wong
  • Xiangtao Li

Single-cell RNA sequencing (scRNA-seq) provides high-throughput information about the genome-wide gene expression levels at the single-cell resolution, bringing a precise understanding on the transcriptome of individual cells. Unfortunately, the rapidly growing scRNA-seq data and the prevalence of dropout events pose substantial challenges for cell type annotation. Here, we propose a single-cell model-based deep graph embedding clustering (scTAG) method, which simultaneously learns cell–cell topology representations and identifies cell clusters based on deep graph convolutional network. scTAG integrates the zero-inflated negative binomial (ZINB) model into a topology adaptive graph convolutional autoencoder to learn the lowdimensional latent representation and adopts Kullback–Leibler (KL) divergence for the clustering tasks. By simultaneously optimizing the clustering loss, ZINB loss, and the cell graph reconstruction loss, scTAG jointly optimizes cluster label assignment and feature learning with the topological structures preserved in an end-to-end manner. Extensive experiments on 16 single-cell RNA-seq datasets from diverse yet representative single-cell sequencing platforms demonstrate the superiority of scTAG over various state-of-the-art clustering methods.

PDF Details

NeurIPS Conference 2021 Conference Paper

Adder Attention for Vision Transformer

  • Han Shu
  • Jiahao Wang
  • Hanting Chen
  • Lin Li
  • Yujiu Yang
  • Yunhe Wang

Transformer is a new kind of calculation paradigm for deep learning which has shown strong performance on a large variety of computer vision tasks. However, compared with conventional deep models (e. g. , convolutional neural networks), vision transformers require more computational resources which cannot be easily deployed on mobile devices. To this end, we present to reduce the energy consumptions using adder neural network (AdderNet). We first theoretically analyze the mechanism of self-attention and the difficulty for applying adder operation into this module. Specifically, the feature diversity, i. e. , the rank of attention map using only additions cannot be well preserved. Thus, we develop an adder attention layer that includes an additional identity mapping. With the new operation, vision transformers constructed using additions can also provide powerful feature representations. Experimental results on several benchmarks demonstrate that the proposed approach can achieve highly competitive performance to that of the baselines while achieving an about 2~3× reduction on the energy consumption.

PDF Details

AAAI Conference 2021 Conference Paper

Adversarial Robustness through Disentangled Representations

  • Shuo Yang
  • Tianyu Guo
  • Yunhe Wang
  • Chang Xu

Despite the remarkable empirical performance of deep learning models, their vulnerability to adversarial examples has been revealed in many studies. They are prone to make a susceptible prediction to the input with imperceptible adversarial perturbation. Although recent works have remarkably improved the model’s robustness under the adversarial training strategy, an evident gap between the natural accuracy and adversarial robustness inevitably exists. In order to mitigate this problem, in this paper, we assume that the robust and non-robust representations are two basic ingredients entangled in the integral representation. For achieving adversarial robustness, the robust representations of natural and adversarial examples should be disentangled from the non-robust part and the alignment of the robust representations can bridge the gap between accuracy and robustness. Inspired by this motivation, we propose a novel defence method called Deep Robust Representation Disentanglement Network (DRRDN). Specifically, DRRDN employs a disentangler to extract and align the robust representations from both adversarial and natural examples. Theoretical analysis guarantees the mitigation of the trade-off between robustness and accuracy with good disentanglement and alignment performance. Experimental results on benchmark datasets finally demonstrate the empirical superiority of our method.

PDF Details

NeurIPS Conference 2021 Conference Paper

An Empirical Study of Adder Neural Networks for Object Detection

  • Xinghao Chen
  • Chang Xu
  • Minjing Dong
  • Chunjing Xu
  • Yunhe Wang

Adder neural networks (AdderNets) have shown impressive performance on image classification with only addition operations, which are more energy efficient than traditional convolutional neural networks built with multiplications. Compared with classification, there is a strong demand on reducing the energy consumption of modern object detectors via AdderNets for real-world applications such as autonomous driving and face detection. In this paper, we present an empirical study of AdderNets for object detection. We first reveal that the batch normalization statistics in the pre-trained adder backbone should not be frozen, since the relatively large feature variance of AdderNets. Moreover, we insert more shortcut connections in the neck part and design a new feature fusion architecture for avoiding the sparse features of adder layers. We present extensive ablation studies to explore several design choices of adder detectors. Comparisons with state-of-the-arts are conducted on COCO and PASCAL VOC benchmarks. Specifically, the proposed Adder FCOS achieves a 37. 8% AP on the COCO val set, demonstrating comparable performance to that of the convolutional counterpart with an about $1. 4\times$ energy reduction.

PDF Details

NeurIPS Conference 2021 Conference Paper

Augmented Shortcuts for Vision Transformers

  • Yehui Tang
  • Kai Han
  • Chang Xu
  • An Xiao
  • Yiping Deng
  • Chao Xu
  • Yunhe Wang

Transformer models have achieved great progress on computer vision tasks recently. The rapid development of vision transformers is mainly contributed by their high representation ability for extracting informative features from input images. However, the mainstream transformer models are designed with deep architectures, and the feature diversity will be continuously reduced as the depth increases, \ie, feature collapse. In this paper, we theoretically analyze the feature collapse phenomenon and study the relationship between shortcuts and feature diversity in these transformer models. Then, we present an augmented shortcut scheme, which inserts additional paths with learnable parameters in parallel on the original shortcuts. To save the computational costs, we further explore an efficient approach that uses the block-circulant projection to implement augmented shortcuts. Extensive experiments conducted on benchmark datasets demonstrate the effectiveness of the proposed method, which brings about 1% accuracy increase of the state-of-the-art visual transformers without obviously increasing their parameters and FLOPs.

PDF Details

NeurIPS Conference 2021 Conference Paper

Dynamic Resolution Network

  • Mingjian Zhu
  • Kai Han
  • Enhua Wu
  • Qiulin Zhang
  • Ying Nie
  • Zhenzhong Lan
  • Yunhe Wang

Deep convolutional neural networks (CNNs) are often of sophisticated design with numerous learnable parameters for the accuracy reason. To alleviate the expensive costs of deploying them on mobile devices, recent works have made huge efforts for excavating redundancy in pre-defined architectures. Nevertheless, the redundancy on the input resolution of modern CNNs has not been fully investigated, i. e. , the resolution of input image is fixed. In this paper, we observe that the smallest resolution for accurately predicting the given image is different using the same neural network. To this end, we propose a novel dynamic-resolution network (DRNet) in which the input resolution is determined dynamically based on each input sample. Wherein, a resolution predictor with negligible computational costs is explored and optimized jointly with the desired network. Specifically, the predictor learns the smallest resolution that can retain and even exceed the original recognition accuracy for each image. During the inference, each input image will be resized to its predicted resolution for minimizing the overall computation burden. We then conduct extensive experiments on several benchmark networks and datasets. The results show that our DRNet can be embedded in any off-the-shelf network architecture to obtain a considerable reduction in computational complexity. For instance, DR-ResNet-50 achieves similar performance with an about 34% computation reduction, while gaining 1. 4% accuracy increase with 10% computation reduction compared to the original ResNet-50 on ImageNet. Code will be available at https: //gitee. com/mindspore/models/tree/master/research/cv/DRNet.

PDF Details

NeurIPS Conference 2021 Conference Paper

Handling Long-tailed Feature Distribution in AdderNets

  • Minjing Dong
  • Yunhe Wang
  • Xinghao Chen
  • Chang Xu

Adder neural networks (ANNs) are designed for low energy cost which replace expensive multiplications in convolutional neural networks (CNNs) with cheaper additions to yield energy-efficient neural networks and hardware accelerations. Although ANNs achieve satisfactory efficiency, there exist gaps between ANNs and CNNs where the accuracy of ANNs can hardly be compared to CNNs without the assistance of other training tricks, such as knowledge distillation. The inherent discrepancy lies in the similarity measurement between filters and features, however how to alleviate this difference remains unexplored. To locate the potential problem of ANNs, we focus on the property difference due to similarity measurement. We demonstrate that unordered heavy tails in ANNs could be the key component which prevents ANNs from achieving superior classification performance since fatter tails tend to overlap in feature space. Through pre-defining Multivariate Skew Laplace distributions and embedding feature distributions into the loss function, ANN features can be fully controlled and designed for various properties. We further present a novel method for tackling existing heavy tails in ANNs with only a modification of classifier where ANN features are clustered with their tails well-formulated through proposed angle-based constraint on the distribution parameters to encourage high diversity of tails. Experiments conducted on several benchmarks and comparison with other distributions demonstrate the effectiveness of proposed approach for boosting the performance of ANNs.

PDF Details

NeurIPS Conference 2021 Conference Paper

Learning Frequency Domain Approximation for Binary Neural Networks

  • Yixing Xu
  • Kai Han
  • Chang Xu
  • Yehui Tang
  • Chunjing Xu
  • Yunhe Wang

Binary neural networks (BNNs) represent original full-precision weights and activations into 1-bit with sign function. Since the gradient of the conventional sign function is almost zero everywhere which cannot be used for back-propagation, several attempts have been proposed to alleviate the optimization difficulty by using approximate gradient. However, those approximations corrupt the main direction of factual gradient. To this end, we propose to estimate the gradient of sign function in the Fourier frequency domain using the combination of sine functions for training BNNs, namely frequency domain approximation (FDA). The proposed approach does not affect the low-frequency information of the original sign function which occupies most of the overall energy, and high-frequency coefficients will be ignored to avoid the huge computational overhead. In addition, we embed a noise adaptation module into the training phase to compensate the approximation error. The experiments on several benchmark datasets and neural architectures illustrate that the binary network learned using our method achieves the state-of-the-art accuracy. Code will be available at https: //gitee. com/mindspore/models/tree/master/research/cv/FDA-BNN.

PDF Details

NeurIPS Conference 2021 Conference Paper

Neural Architecture Dilation for Adversarial Robustness

  • Yanxi Li
  • Zhaohui Yang
  • Yunhe Wang
  • Chang Xu

With the tremendous advances in the architecture and scale of convolutional neural networks (CNNs) over the past few decades, they can easily reach or even exceed the performance of humans in certain tasks. However, a recently discovered shortcoming of CNNs is that they are vulnerable to adversarial attacks. Although the adversarial robustness of CNNs can be improved by adversarial training, there is a trade-off between standard accuracy and adversarial robustness. From the neural architecture perspective, this paper aims to improve the adversarial robustness of the backbone CNNs that have a satisfactory accuracy. Under a minimal computational overhead, the introduction of a dilation architecture is expected to be friendly with the standard performance of the backbone CNN while pursuing adversarial robustness. Theoretical analyses on the standard and adversarial error bounds naturally motivate the proposed neural architecture dilation algorithm. Experimental results on real-world datasets and benchmark neural networks demonstrate the effectiveness of the proposed algorithm to balance the accuracy and adversarial robustness.

PDF Details

AAAI Conference 2021 Conference Paper

One-shot Graph Neural Architecture Search with Dynamic Search Space

  • Yanxi Li
  • Zean Wen
  • Yunhe Wang
  • Chang Xu

Relying on the diverse graph convolution operations that have emerged in recent years, graph neural networks (GNNs) are shown to be powerful to deal with high-dimensional non- Euclidean domains, such as social networks or citation networks. Despite the tremendous human efforts been taken to explore new graph convolution operations, there are a few attempts to automatically search operations in GNNs. The search space of GNNs is significantly larger than that of CNNs, because of diverse components in the messagepassing of GNNs. This, therefore, prevents the straightforward application of classical NAS methods for GNNs. In this work, we propose a novel dynamic one-shot search space for multi-branch neural architectures of GNNs. The dynamic search space maintains a subset of the large search space along with a set of importance weights for operation candidates in the subset as the architecture parameters. After each iteration, the subset is pruned by removing candidates with low importance weights and is expanded with new operations. The dynamic subsets of operation candidates are not uniform but is individual for each edge in the computation graph of the neural architecture, which can ensure the diversity of operations in the final architecture is as competitive as direct search in the large search space. Our experiments of semisupervised and supervised node classification on citation networks, including Cora, Citeseer, and Pubmed, demonstrate that our method outperforms the current state-of-the-art manually designed architectures and reaches competitive performance to existing GNN NAS approaches with up to 10 times of speedup.

PDF Details

NeurIPS Conference 2021 Conference Paper

Post-Training Quantization for Vision Transformer

  • Zhenhua Liu
  • Yunhe Wang
  • Kai Han
  • Wei Zhang
  • Siwei Ma
  • Wen Gao

Recently, transformer has achieved remarkable performance on a variety of computer vision applications. Compared with mainstream convolutional neural networks, vision transformers are often of sophisticated architectures for extracting powerful feature representations, which are more difficult to be developed on mobile devices. In this paper, we present an effective post-training quantization algorithm for reducing the memory storage and computational costs of vision transformers. Basically, the quantization task can be regarded as finding the optimal low-bit quantization intervals for weights and inputs, respectively. To preserve the functionality of the attention mechanism, we introduce a ranking loss into the conventional quantization objective that aims to keep the relative order of the self-attention results after quantization. Moreover, we thoroughly analyze the relationship between quantization loss of different layers and the feature diversity, and explore a mixed-precision quantization scheme by exploiting the nuclear norm of each attention map and output feature. The effectiveness of the proposed method is verified on several benchmark models and datasets, which outperforms the state-of-the-art post-training quantization algorithms. For instance, we can obtain an 81. 29% top-1 accuracy using DeiT-B model on ImageNet dataset with about 8-bit quantization. Code will be available at https: //gitee. com/mindspore/models/tree/master/research/cv/VT-PTQ.

PDF Details

NeurIPS Conference 2021 Conference Paper

Towards Stable and Robust AdderNets

  • Minjing Dong
  • Yunhe Wang
  • Xinghao Chen
  • Chang Xu

Adder neural network (AdderNet) replaces the original convolutions with massive multiplications by cheap additions while achieving comparable performance thus yields a series of energy-efficient neural networks. Compared with convolutional neural networks (CNNs), the training of AdderNets is much more sophisticated including several techniques for adjusting gradient and batch normalization. In addition, variances of both weights and activations in resulting adder networks are very enormous which limits its performance and the potential for applying to other tasks. To enhance the stability and robustness of AdderNets, we first thoroughly analyze the variance estimation of weight parameters and output features of an arbitrary adder layer. Then, we develop a weight normalization scheme for adaptively optimizing the weight distribution of AdderNets during the training procedure, which can reduce the perturbation on running mean and variance in batch normalization layers. Meanwhile, the proposed weight normalization can also be utilized to enhance the adversarial robustness of resulting networks. Experiments conducted on several benchmarks demonstrate the superiority of the proposed approach for generating AdderNets with higher performance.

PDF Details

NeurIPS Conference 2021 Conference Paper

Transformer in Transformer

  • Kai Han
  • An Xiao
  • Enhua Wu
  • Jianyuan Guo
  • Chunjing Xu
  • Yunhe Wang

Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (\eg, 16$\times$16) as “visual sentences” and present to further divide them into smaller patches (\eg, 4$\times$4) as “visual words”. The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, \eg, we achieve an 81. 5\% top-1 accuracy on the ImageNet, which is about 1. 7\% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at \url{https: //github. com/huawei-noah/CV-Backbones}, and the MindSpore code is available at \url{https: //gitee. com/mindspore/models/tree/master/research/cv/TNT}.

PDF Details

NeurIPS Conference 2020 Conference Paper

Adapting Neural Architectures Between Domains

  • Yanxi Li
  • Zhaohui Yang
  • Yunhe Wang
  • Chang Xu

Neural architecture search (NAS) has demonstrated impressive performance in automatically designing high-performance neural networks. The power of deep neural networks is to be unleashed for analyzing a large volume of data (e. g. ImageNet), but the architecture search is often executed on another smaller dataset (e. g. CIFAR-10) to finish it in a feasible time. However, it is hard to guarantee that the optimal architecture derived on the proxy task could maintain its advantages on another more challenging dataset. This paper aims to improve the generalization of neural architectures via domain adaptation. We analyze the generalization bounds of the derived architecture and suggest its close relations with the validation error and the data distribution distance on both domains. These theoretical analyses lead to AdaptNAS, a novel and principled approach to adapt neural architectures between domains in NAS. Our experimental evaluation shows that only a small part of ImageNet will be sufficient for AdaptNAS to extend its architecture success to the entire ImageNet and outperform state-of-the-art comparison algorithms.

PDF Details

AAAI Conference 2020 Conference Paper

Beyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks

  • Yehui Tang
  • Yunhe Wang
  • Yixing Xu
  • Boxin Shi
  • Chao Xu
  • Chunjing Xu
  • Chang Xu

Deep neural networks often consist of a great number of trainable parameters for extracting powerful features from given datasets. One one hand, massive trainable parameters significantly enhance the performance of these deep networks. One the other hand, they bring the problem of over-fitting. To this end, dropout based methods disable some elements in the output feature maps during the training phase for reducing the co-adaptation of neurons. Although the generalization ability of the resulting models can be enhanced by these approaches, the conventional binary dropout is not the optimal solution. Therefore, we investigate the empirical Rademacher complexity related to intermediate layers of deep neural networks and propose a feature distortion method for addressing the aforementioned problem. In the training period, randomly selected elements in the feature maps will be replaced with specific values by exploiting the generalization error bound. The superiority of the proposed feature map distortion for producing deep neural network with higher testing performance is analyzed and demonstrated on several benchmark image datasets.

PDF Details

AAAI Conference 2020 Conference Paper

Distilling Portable Generative Adversarial Networks for Image Translation

  • Hanting Chen
  • Yunhe Wang
  • Han Shu
  • Changyuan Wen
  • Chunjing Xu
  • Boxin Shi
  • Chao Xu
  • Chang Xu

Despite Generative Adversarial Networks (GANs) have been widely used in various image-to-image translation tasks, they can be hardly applied on mobile devices due to their heavy computation and storage cost. Traditional network compression methods focus on visually recognition tasks, but never deal with generation tasks. Inspired by knowledge distillation, a student generator of fewer parameters is trained by inheriting the low-level and high-level information from the original heavy teacher generator. To promote the capability of student generator, we include a student discriminator to measure the distances between real images, and images generated by student and teacher generators. An adversarial learning process is therefore established to optimize student generator and student discriminator. Qualitative and quantitative analysis by conducting experiments on benchmark datasets demonstrate that the proposed method can learn portable generative models with strong performance.

PDF Details

IJCAI Conference 2020 Conference Paper

DropNAS: Grouped Operation Dropout for Differentiable Architecture Search

  • Weijun Hong
  • Guilin Li
  • Weinan Zhang
  • Ruiming Tang
  • Yunhe Wang
  • Zhenguo Li
  • Yong Yu

Neural architecture search (NAS) has shown encouraging results in automating the architecture design. Recently, DARTS relaxes the search process with a differentiable formulation that leverages weight-sharing and SGD for cost reduction of NAS. In DARTS, all candidate operations are trained simultaneously during the network weight training step. Our empirical results show that this training procedure leads to the co-adaption problem and Matthew Effect: operations with fewer parameters would be trained maturely earlier. This causes two problems: firstly, the operations with more parameters may never have the chance to express the desired function since those with less have already done the job; secondly, the system will punish those underperforming operations by lowering their architecture parameter and backward smaller loss gradients, this causes the Matthew Effect. In this paper, we systematically study these problems and propose a novel grouped operation dropout algorithm named DropNAS to fix the problems with DARTS. Extensive experiments demonstrate that DropNAS solves the above issues and achieves promising performance. Specifically, DropNAS achieves 2. 26% test error on CIFAR-10, 16. 39% on CIFAR-100 and 23. 4% on ImageNet (with the same training hyperparameters as DARTS for a fair comparison). It is also observed that DropNAS is robust across variants of the DARTS search space. Code is available at https: //github. com/huawei-noah.

PDF Details DOI

AAAI Conference 2020 Conference Paper

Efficient Residual Dense Block Search for Image Super-Resolution

  • Dehua Song
  • Chang Xu
  • Xu Jia
  • Yiyi Chen
  • Chunjing Xu
  • Yunhe Wang

Although remarkable progress has been made on single image super-resolution due to the revival of deep convolutional neural networks, deep learning methods are confronted with the challenges of computation and memory consumption in practice, especially for mobile devices. Focusing on this issue, we propose an efficient residual dense block search algorithm with multiple objectives to hunt for fast, lightweight and accurate networks for image super-resolution. Firstly, to accelerate super-resolution network, we exploit the variation of feature scale adequately with the proposed efficient residual dense blocks. In the proposed evolutionary algorithm, the locations of pooling and upsampling operator are searched automatically. Secondly, network architecture is evolved with the guidance of block credits to acquire accurate superresolution network. The block credit reflects the effect of current block and is earned during model evaluation process. It guides the evolution by weighing the sampling probability of mutation to favor admirable blocks. Extensive experimental results demonstrate the effectiveness of the proposed searching method and the found efficient super-resolution models achieve better performance than the state-of-the-art methods with limited number of parameters and FLOPs.

PDF Details

NeurIPS Conference 2020 Conference Paper

Kernel Based Progressive Distillation for Adder Neural Networks

  • Yixing Xu
  • Chang Xu
  • Xinghao Chen
  • Wei Zhang
  • Chunjing Xu
  • Yunhe Wang

Adder Neural Networks (ANNs) which only contain additions bring us a new way of developing deep neural networks with low energy consumption. Unfortunately, there is an accuracy drop when replacing all convolution filters by adder filters. The main reason here is the optimization difficulty of ANNs using $\ell_1$-norm, in which the estimation of gradient in back propagation is inaccurate. In this paper, we present a novel method for further improving the performance of ANNs without increasing the trainable parameters via a progressive kernel based knowledge distillation (PKKD) method. A convolutional neural network (CNN) with the same architecture is simultaneously initialized and trained as a teacher network, features and weights of ANN and CNN will be transformed to a new space to eliminate the accuracy drop. The similarity is conducted in a higher-dimensional space to disentangle the difference of their distributions using a kernel based method. Finally, the desired ANN is learned based on the information from both the ground-truth and teacher, progressively. The effectiveness of the proposed method for learning ANN with higher performance is then well-verified on several benchmarks. For instance, the ANN-50 trained using the proposed PKKD method obtains a 76. 8\% top-1 accuracy on ImageNet dataset, which is 0. 6\% higher than that of the ResNet-50.

PDF Details

NeurIPS Conference 2020 Conference Paper

Model Rubik’s Cube: Twisting Resolution, Depth and Width for TinyNets

  • Kai Han
  • Yunhe Wang
  • Qiulin Zhang
  • Wei Zhang
  • Chunjing Xu
  • Tong Zhang

To obtain excellent deep neural architectures, a series of techniques are carefully designed in EfficientNets. The giant formula for simultaneously enlarging the resolution, depth and width provides us a Rubik’s cube for neural networks. So that we can find networks with high efficiency and excellent performance by twisting the three dimensions. This paper aims to explore the twisting rules for obtaining deep neural networks with minimum model sizes and computational costs. Different from the network enlarging, we observe that resolution and depth are more important than width for tiny networks. Therefore, the original method, \ie the compound scaling in EfficientNet is no longer suitable. To this end, we summarize a tiny formula for downsizing neural architectures through a series of smaller models derived from the EfficientNet-B0 with the FLOPs constraint. Experimental results on the ImageNet benchmark illustrate that our TinyNet performs much better than the smaller version of EfficientNets using the inversed giant formula. For instance, our TinyNet-E achieves a 59. 9\% Top-1 accuracy with only 24M FLOPs, which is about 1. 9\% higher than that of the previous best MobileNetV3 with similar computational cost. Code will be available at \url{https: //github. com/huawei-noah/CV-Backbones/tree/master/tinynet}, and \url{https: //gitee. com/mindspore/mindspore/tree/master/model_zoo/research/cv/tinynet}.

PDF Details

NeurIPS Conference 2020 Conference Paper

Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts

  • Guilin Li
  • Junlei Zhang
  • Yunhe Wang
  • Chuanjian Liu
  • Matthias Tan
  • Yunfeng Lin
  • Wei Zhang
  • Jiashi Feng

By transferring both features and gradients between different layers, shortcut connections explored by ResNets allow us to effectively train very deep neural networks up to hundreds of layers. However, the additional computation costs induced by those shortcuts are often overlooked. For example, during online inference, the shortcuts in ResNet-50 account for about 40 percent of the entire memory usage on feature maps, because the features in the preceding layers cannot be released until the subsequent calculation is completed. In this work, for the first time, we consider training the CNN models with shortcuts and deploying them without. In particular, we propose a novel joint-training framework to train plain CNN by leveraging the gradients of the ResNet counterpart. During forward step, the feature maps of the early stages of plain CNN are passed through later stages of both itself and the ResNet counterpart to calculate the loss. During backpropagation, gradients calculated from a mixture of these two parts are used to update the plainCNN network to solve the gradient vanishing problem. Extensive experiments on ImageNet/CIFAR10/CIFAR100 demonstrate that the plainCNN network without shortcuts generated by our approach can achieve the same level of accuracy as that of the ResNet baseline while achieving about $1. 4\times $ speed-up and $1. 25\times$ memory reduction. We also verified the feature transferability of our ImageNet pretrained plain-CNN network by fine-tuning it on MIT 67 and Caltech 101. Our results show that the performance of the plain-CNN is slightly higher than that of its baseline ResNet-50 on these two datasets. The codes are in: \href{https: //github. com/leoozy/JointRD_Neurips2020}{https: //github. com/leoozy/JointRD\_Neurips2020}

PDF Details

NeurIPS Conference 2020 Conference Paper

SCOP: Scientific Control for Reliable Neural Network Pruning

  • Yehui Tang
  • Yunhe Wang
  • Yixing Xu
  • Dacheng Tao
  • Chunjing Xu
  • Chao Xu
  • Chang Xu

This paper proposes a reliable neural network pruning algorithm by setting up a scientific control. Existing pruning methods have developed various hypotheses to approximate the importance of filters to the network and then execute filter pruning accordingly. To increase the reliability of the results, we prefer to have a more rigorous research design by including a scientific control group as an essential part to minimize the effect of all factors except the association between the filter and expected network output. Acting as a control group, knockoff feature is generated to mimic the feature map produced by the network filter, but they are conditionally independent of the example label given the real feature map. We theoretically suggest that the knockoff condition can be approximately preserved given the information propagation of network layers. Besides the real feature map on an intermediate layer, the corresponding knockoff feature is brought in as another auxiliary input signal for the subsequent layers. Redundant filters can be discovered in the adversarial process of different features. Through experiments, we demonstrate the superiority of the proposed algorithm over state-of-the-art methods. For example, our method can reduce 57. 8% parameters and 60. 2% FLOPs of ResNet-101 with only 0. 01% top-1 accuracy loss on ImageNet.

PDF Details

NeurIPS Conference 2020 Conference Paper

Searching for Low-Bit Weights in Quantized Neural Networks

  • Zhaohui Yang
  • Yunhe Wang
  • Kai Han
  • Chunjing Xu
  • Chao Xu
  • Dacheng Tao
  • Chang Xu

Quantized neural networks with low-bit weights and activations are attractive for developing AI accelerators. However, the quantization functions used in most conventional quantization methods are non-differentiable, which increases the optimization difficulty of quantized networks. Compared with full-precision parameters (\emph{i. e. }, 32-bit floating numbers), low-bit values are selected from a much smaller set. For example, there are only 16 possibilities in 4-bit space. Thus, we present to regard the discrete weights in an arbitrary quantized neural network as searchable variables, and utilize a differential method to search them accurately. In particular, each weight is represented as a probability distribution over the discrete value set. The probabilities are optimized during training and the values with the highest probability are selected to establish the desired quantized network. Experimental results on benchmarks demonstrate that the proposed method is able to produce quantized neural networks with higher performance over the state-of-the-arts on both image classification and super-resolution tasks.

PDF Details

IJCAI Conference 2019 Conference Paper

Attribute Aware Pooling for Pedestrian Attribute Recognition

  • Kai Han
  • Yunhe Wang
  • Han Shu
  • Chuanjian Liu
  • Chunjing Xu
  • Chang Xu

This paper expands the strength of deep convolutional neural networks (CNNs) to the pedestrian attribute recognition problem by devising a novel attribute aware pooling algorithm. Existing vanilla CNNs cannot be straightforwardly applied to handle multi-attribute data because of the larger label space as well as the attribute entanglement and correlations. We tackle these challenges that hampers the development of CNNs for multi-attribute classification by fully exploiting the correlation between different attributes. The multi-branch architecture is adopted for fucusing on attributes at different regions. Besides the prediction based on each branch itself, context information of each branch are employed for decision as well. The attribute aware pooling is developed to integrate both kinds of information. Therefore, attributes which are indistinct or tangled with others can be accurately recognized by exploiting the context information. Experiments on benchmark datasets demonstrate that the proposed pooling method appropriately explores and exploits the correlations between attributes for the pedestrian attribute recognition.

PDF Details

IJCAI Conference 2019 Conference Paper

Crafting Efficient Neural Graph of Large Entropy

  • Minjing Dong
  • Hanting Chen
  • Yunhe Wang
  • Chang Xu

Network pruning is widely applied to deep CNN models due to their heavy computation costs and achieves high performance by keeping important weights while removing the redundancy. Pruning redundant weights directly may hurt global information flow, which suggests that an efficient sparse network should take graph properties into account. Thus, instead of paying more attention to preserving important weight, we focus on the pruned architecture itself. We propose to use graph entropy as the measurement, which shows useful properties to craft high-quality neural graphs and enables us to propose efficient algorithm to construct them as the initial network architecture. Our algorithm can be easily implemented and deployed to different popular CNN models and achieve better trade-offs.

PDF Details

IJCAI Conference 2019 Conference Paper

Learning Instance-wise Sparsity for Accelerating Deep Models

  • Chuanjian Liu
  • Yunhe Wang
  • Kai Han
  • Chunjing Xu
  • Chang Xu

Exploring deep convolutional neural networks of high efficiency and low memory usage is very essential for a wide variety of machine learning tasks. Most of existing approaches used to accelerate deep models by manipulating parameters or filters without data, e. g. , pruning and decomposition. In contrast, we study this problem from a different perspective by respecting the difference between data. An instance-wise feature pruning is developed by identifying informative features for different instances. Specifically, by investigating a feature decay regularization, we expect intermediate feature maps of each instance in deep neural networks to be sparse while preserving the overall network performance. During online inference, subtle features of input images extracted by intermediate layers of a well-trained neural network can be eliminated to accelerate the subsequent calculations. We further take coefficient of variation as a measure to select the layers that are appropriate for acceleration. Extensive experiments conducted on benchmark datasets and networks demonstrate the effectiveness of the proposed method.

PDF Details

NeurIPS Conference 2019 Conference Paper

Positive-Unlabeled Compression on the Cloud

  • Yixing Xu
  • Yunhe Wang
  • Hanting Chen
  • Kai Han
  • Chunjing Xu
  • Dacheng Tao
  • Chang Xu

Many attempts have been done to extend the great success of convolutional neural networks (CNNs) achieved on high-end GPU servers to portable devices such as smart phones. Providing compression and acceleration service of deep learning models on the cloud is therefore of significance and is attractive for end users. However, existing network compression and acceleration approaches usually fine-tuning the svelte model by requesting the entire original training data (e. g. ImageNet), which could be more cumbersome than the network itself and cannot be easily uploaded to the cloud. In this paper, we present a novel positive-unlabeled (PU) setting for addressing this problem. In practice, only a small portion of the original training set is required as positive examples and more useful training examples can be obtained from the massive unlabeled data on the cloud through a PU classifier with an attention based multi-scale feature extractor. We further introduce a robust knowledge distillation (RKD) scheme to deal with the class imbalance problem of these newly augmented training examples. The superiority of the proposed method is verified through experiments conducted on the benchmark models and datasets. We can use only 8% of uniformly selected data from the ImageNet to obtain an efficient model with comparable performance to the baseline ResNet-34.

PDF Details

AAAI Conference 2018 Conference Paper

Adversarial Learning of Portable Student Networks

  • Yunhe Wang
  • Chang Xu
  • Chao Xu
  • Dacheng Tao

Effective methods for learning deep neural networks with fewer parameters are urgently required, since storage and computations of heavy neural networks have largely prevented their widespread use on mobile devices. Compared with algorithms which directly remove weights or filters for obtaining considerable compression and speed-up ratios, training thin deep networks exploiting the student-teacher learning paradigm is more flexible. However, it is very hard to determine which formulation is optimal to measure the information inherited from teacher networks. To overcome this challenge, we utilize the generative adversarial network (GAN) to learn the student network. In practice, the generator is exactly the student network with extremely less parameters and the discriminator is used as a teaching assistant for distinguishing features extracted from student and teacher networks. By simultaneously optimizing the generator and the discriminator, the resulting student network can produce features of input data with the similar distribution as that of features of the teacher network. Extensive experimental results on benchmark datasets demonstrate that the proposed method is capable of learning well-performed portable networks, which is superior to the state-of-the-art methods.

PDF Details

NeurIPS Conference 2018 Conference Paper

Learning Versatile Filters for Efficient Convolutional Neural Networks

  • Yunhe Wang
  • Chang Xu
  • Chunjing Xu
  • Chao Xu
  • Dacheng Tao

This paper introduces versatile filters to construct efficient convolutional neural network. Considering the demands of efficient deep learning techniques running on cost-effective hardware, a number of methods have been developed to learn compact neural networks. Most of these works aim to slim down filters in different ways, e. g. , investigating small, sparse or binarized filters. In contrast, we treat filters from an additive perspective. A series of secondary filters can be derived from a primary filter. These secondary filters all inherit in the primary filter without occupying more storage, but once been unfolded in computation they could significantly enhance the capability of the filter by integrating information extracted from different receptive fields. Besides spatial versatile filters, we additionally investigate versatile filters from the channel perspective. The new techniques are general to upgrade filters in existing CNNs. Experimental results on benchmark datasets and neural networks demonstrate that CNNs constructed with our versatile filters are able to achieve comparable accuracy as that of original filters, but require less memory and FLOPs.

PDF Details

AAAI Conference 2017 Conference Paper

Beyond RPCA: Flattening Complex Noise in the Frequency Domain

  • Yunhe Wang
  • Chang Xu
  • Chao Xu
  • Dacheng Tao

Discovering robust low-rank data representations is important in many real-world problems. Traditional robust principal component analysis (RPCA) assumes that the observed data are corrupted by some sparse noise (e. g. , Laplacian noise) and utilizes the 1-norm to separate out the noisy component. Nevertheless, as well as simple Gaussian or Laplacian noise, noise in real-world data is often more complex, and thus the 1 and 2-norms are insufficient for noise characterization. This paper presents a more flexible approach to modeling complex noise by investigating their properties in the frequency domain. Although elements of a noise matrix are chaotic in the spatial domain, the absolute values of its alternative coefficients in the frequency domain are constant w. r. t. their variance. Based on this observation, a new robust PCA algorithm is formulated by simultaneously discovering the low-rank and noisy components. Extensive experiments on synthetic data and video background subtraction demonstrate that FRPCA is effective for handles complex noise.

PDF Details

IJCAI Conference 2017 Conference Paper

Privileged Multi-label Learning

  • Shan You
  • Chang Xu
  • Yunhe Wang
  • Chao Xu
  • Dacheng Tao

This paper presents privileged multi-label learning (PrML) to explore and exploit the relationship between labels in multi-label learning problems. We suggest that for each individual label, it cannot only be implicitly connected with other labels via the low-rank constraint over label predictors, but also its performance on examples can receive the explicit comments from other labels together acting as an Oracle teacher. We generate privileged label feature for each example and its individual label, and then integrate it into the framework of low-rank based multi-label learning. The proposed algorithm can therefore comprehensively explore and exploit label relationships by inheriting all the merits of privileged information and low-rank constraints. We show that PrML can be efficiently solved by dual coordinate descent algorithm using iterative optimization strategy with cheap updates. Experiments on benchmark datasets show that through privileged label features, the performance can be significantly improved and PrML is superior to several competing methods in most cases.

PDF Details

NeurIPS Conference 2016 Conference Paper

CNNpack: Packing Convolutional Neural Networks in the Frequency Domain

  • Yunhe Wang
  • Chang Xu
  • Shan You
  • Dacheng Tao
  • Chao Xu

Deep convolutional neural networks (CNNs) are successfully used in a number of applications. However, their storage and computational requirements have largely prevented their widespread use on mobile devices. Here we present an effective CNN compression approach in the frequency domain, which focuses not only on smaller weights but on all the weights and their underlying connections. By treating convolutional filters as images, we decompose their representations in the frequency domain as common parts (i. e. , cluster centers) shared by other similar filters and their individual private parts (i. e. , individual residuals). A large number of low-energy frequency coefficients in both parts can be discarded to produce high compression without significantly compromising accuracy. We relax the computational burden of convolution operations in CNNs by linearly combining the convolution responses of discrete cosine transform (DCT) bases. The compression and speed-up ratios of the proposed algorithm are thoroughly analyzed and evaluated on benchmark image datasets to demonstrate its superiority over state-of-the-art methods.

PDF Details
v2026.09.13