Author name cluster

Lei Deng

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

21 papers

1 author row

AAAI Conference 2026 Conference Paper

Hierarchical Structure-Property Alignment for Data-Efficient Molecular Generation and Editing

Ziyu Fan
Zhijian Huang
Yahan Li
Xiaowen Hu
Siyuan Shen
Yunliang Wang
Zeyu Zhong
Shuhong Liu

Property-constrained molecular generation and editing are crucial in AI-driven drug discovery but remain hindered by two factors: (i) capturing the complex relationships between molecular structures and multiple properties remains challenging, and (ii) the narrow coverage and incomplete annotations of molecular properties weaken the effectiveness of property-based models. To tackle these limitations, we propose HSPAG, a data-efficient framework featuring hierarchical structure–property alignment. By treating SMILES and molecular properties as complementary modalities, the model learns their relationships at atom, substructure, and whole-molecule levels. Moreover, we select representative samples through scaffold clustering and hard samples via an auxiliary variational auto-encoder (VAE), substantially reducing the required pre-training data. In addition, we incorporate a property relevance-aware masking mechanism and diversified perturbation strategies to enhance generation quality under sparse annotations. Experiments demonstrate that HSPAG captures fine-grained structure–property relationships and supports controllable generation under multiple property constraints. Two real-world case studies further validate the editing capabilities of HSPAG.

PDF Details DOI

EAAI Journal 2026 Journal Article

Numerical simulation-driven machine learning and particle swarm optimization of burner fuel distribution for cleaner combustion in a thermal power plant

Aleksandar Milićević
Srđan Belošević
Ivan Tomanović
Nenad Crnomarković
Lei Deng
Defu Che

Coal-fired power plants remain important energy source in many countries, but releasing significant nitrogen oxides (NOx) emissions with serious environmental and health impacts. This study focuses on optimizing lignite combustion in a large utility boiler to minimize NOx emission by adjusting fuel distribution over the burner tiers. The proposed methodology integrates an in-house developed numerical code, correlation analysis, Extreme Gradient Boosting (XGBoost) model and Particle Swarm Optimization (PSO) algorithm. A database of computational fluid dynamics (CFD) simulations was generated to develop machine learning (ML) models for predicting NOx emission and furnace exit gas temperature (FEGT). Based on the ML models, PSO was applied to minimize NOx emissions while maintaining FEGT close to reference case. The framework is applied to a real scale utility boiler, demonstrating practical applicability and measurable environmental impact. Its multidisciplinary significance lies in the synergistic combination of CFD, artificial intelligence, and PSO for combustion control in large-scale energy system. Numerical simulation confirmed the accuracy of the PSO-based optimization, showing excellent agreement between predicted and simulated NOx emission and FEGT. The optimized case corresponds to a 30% reduction in NOx emission compared to the reference scenario. The results also indicated risk of localized overheating and slagging at the furnace walls and possible problems with ascending flame, highlighting the importance of adjusting the operating conditions and flame control to ensure stable boiler performance. The presented comprehensive method provides a novel and effective solution to improve the environmental performance of thermal power plants through intelligent optimization of combustion parameters.

Details DOI

EAAI Journal 2026 Journal Article

Physics-enhanced simulation-to-measurement translation for rolling bearing fault diagnosis under limited samples

Zhen Ming
Baoping Tang
Lei Deng
Xiaolong Zhang
Zihao Li

The application of deep learning in fault diagnosis is constrained by limited samples. While dynamic simulation data have been widely utilized to augment datasets, significant distribution discrepancies between simulation and measurement data remain, leading to suboptimal diagnostic performance. To bridge this gap, this paper proposes a Physics-Enhanced Generative Adversarial Network (PEGAN), which incorporates physics-informed learning to achieve high-fidelity simulation-to-measurement translation. The framework includes a bearing dynamics simulation model that generates abundant simulation data, providing comprehensive prior knowledge for PEGAN. To enhance feature alignment, an adaptive noise injection module is introduced to embed real-world noise characteristics into simulation data. In addition, a time-frequency bidomain-aware adjustment module is designed to perform joint time-frequency alignment of simulation data, thereby enhancing the network's feature adjustment capability. Furthermore, a physics-embedded loss function is introduced to optimize the network by minimizing discrepancies in key physical features between generated and measurement data. The augmented dataset generated by PEGAN is then used to train diagnostic classifiers. Comprehensive experimental validations on two test rig datasets and one real-world engineering case demonstrate that PEGAN significantly improves fault diagnosis accuracy under limited sample conditions, offering an innovative physics-guided paradigm for industrial fault diagnosis.

Details DOI

EAAI Journal 2026 Journal Article

Vibration mechanism driven discrete wavelet hybrid attention weighted transfer network for partial domain fault diagnosis of gearboxes under data scarcity

Peng Zhu
Baoping Tang
Lei Deng
Jing Wei
Zihao Li
Qikang Li

Data-driven intelligent fault diagnosis methods play a vital role in maintaining safe and reliable operation of gearboxes. Currently, some simulation data-driven transfer learning methods have been developed to achieve gearboxes fault diagnosis under the scarcity of high-quality labeled data. However, these methods still have problems such as insufficient domain-invariant feature extraction capabilities and negative transfer caused by large discrepancies between simulation and measured data, which poses challenges to their deployment in real-world industrial environments. To address these challenges, a vibration mechanism driven discrete wavelet hybrid attention weighted transfer network (DWAWTN) is proposed for partial domain fault diagnosis of gearboxes. Firstly, a vibration signal model that can reflect the gear failure vibration response mechanism is established, which can generate simulation data of different fault types. Secondly, a hybrid attention-guided multi-scale discrete wavelet convolutional network is proposed to extract domain-invariant features of simulation and measured data, which dynamically focuses on the multi-resolution fault features after wavelet decomposition from the channel and spatial dimensions. Then, a domain adaptation method combining the maximum likelihood weight estimation strategy and pseudo-label self-learning technology is constructed, which can not only suppress the negative transfer effect of outlier classes data in the source domain, but also reduce the uncertainty of the prediction of measured target data. Finally, comprehensive experimental verification and analysis are carried out on two gearbox datasets. Experimental results show that DWAWTN outperforms other compared methods in diagnostic performance.

Details DOI

JBHI Journal 2025 Journal Article

AGCLNDA: Enhancing the Prediction of ncRNA-Drug Resistance Association Using Adaptive Graph Contrastive Learning

Yanhao Fan
Che Zhang
Xiaowen Hu
Zhijian Huang
Lei Deng

Non-coding RNAs (ncRNAs), which do not encode proteins, have been implicated in chemotherapy resistance in cancer treatment. Given the high costs and time requirements of traditional biological experiments, there is an increasing need for computational models to predict ncRNA-drug resistance associations. In this study, we introduce AGCLNDA, an adaptive contrastive learning method designed to uncover these associations. AGCLNDA begins by constructing a bipartite graph from existing ncRNA-drug resistance data. It then utilizes a light graph convolutional network (LightGCN) to learn vector representations for both ncRNAs and drugs. The method assesses resistance association scores through the inner product of these vectors. To tackle data sparsity and noise, AGCLNDA incorporates learnable augmented view generators and denoised view generators, which provide contrastive views for enhanced data augmentation. Comparative experiments demonstrate that AGCLNDA outperforms five other advanced methods. Case studies further validate AGCLNDA as an effective tool for predicting ncRNA-drug resistance associations.

Details DOI

EAAI Journal 2025 Journal Article

Discriminative feature learning using class-aware and selective transfer adversarial network for partial cross-domain fault diagnosis of gearboxes

Peng Zhu
Baoping Tang
Lei Deng
Qikang Li
Zihao Li

Recently, partial domain adaptation (PDA) fault diagnosis methods have gained significant attention, as they assume that the target domain (TD) data class labels are only a subset of the source domain (SD) data class labels, making them more closely aligned with actual industrial scenarios. Most existing PDA methods adopt reweighting strategies to suppress the influence of outlier classes in the SD. However, these methods still face two major challenges: involving all SD samples in network training causes premature domain discrepancy, hindering the improvement of model transfer performance in the later stage of training; the relationship between target class samples is ignored, which makes the network classifier insufficient in extracting TD discriminable features. To solve these challenges, this study proposes a class-aware and selective transfer adversarial network (CSTAN) for partial cross-domain fault diagnosis of gearboxes. Firstly, the source data selector (SDS) in the CSTAN network is proposed to automatically filter out the source outlier data to avoid negative transfer; Secondly, the residual multi-channel attention mechanism feature extractor network is constructed to extract the features from the source data filtered by SDS and the target data, and the entropy-enhanced domain adversarial metric is adopted to promote positive transfer; Thirdly, to improve the model's ability to extract discriminable features of the TD, a target class-aware classification mechanism based on pseudo-label self-training technology is designed; Finally, the superiority and effectiveness of the CSTAN method are verified on the datasets of drivetrain diagnostics simulator test bench and wind turbine gearboxes.

Details DOI

JBHI Journal 2025 Journal Article

Enhancing Predictions of Drug Solubility Through Multidimensional Structural Characterization Exploitation

Ziyu Fan
Linying Chen
Xinyi Wu
Zhijian Huang
Lei Deng

Solubility is not only a significant physical property of molecules but also a vital factor in small-molecule drug development. Determining drug solubility demands stringent equipment, controlled environments, and substantial human and material resources. The accurate prediction of drug solubility using computational methods has long been a goal for researchers. In this study, we introduce MSCSol, a solubility prediction model that integrates multidimensional molecular structure information. We incorporate a graph neural network with geometric vector perceptrons (GVP-GNN) to encode 3D molecular structures, representing spatial arrangement and orientation of atoms, as well as atomic sequences and interactions. We also employ Selective Kernel Convolution combined with Global and Local attention mechanisms to capture molecular features context at different scales. Additionally, various descriptors are calculated to enrich the molecular representation. For the 2D and 3D structural data of molecules, we design different data augmentation strategies to enhance generalization ability and prevent the model from learning irrelevant information. Extensive experiments on benchmark and independent datasets demonstrate MSCSol's superior performance. Ablation studies further confirm the effectiveness of different modules. Interpretability analysis highlights the importance of various atomic groups and substructures for solubility and verifies that our model effectively captures functional molecular structures and higher-order knowledge.

Details DOI

EAAI Journal 2025 Journal Article

Physics-informed cross layer temporal frequency transformer network for remaining useful life prediction of rolling bearings

Runyan Zhao
Baoping Tang
Xiaolong Zhang
Lei Deng

In response to the challenge of accurately capturing long-term degradation dependencies in the remaining useful life (RUL) prediction task for rolling bearings under extreme operating conditions, which leads to low accuracy of prediction results, a physics-informed cross-layer temporal frequency Transformer method for rolling bearing RUL prediction, named WeiCLTF, has been proposed. Firstly, a cross-layer information transfer mechanism is established between adjacent layers within the Transformer, which introduces cross-layer regularization to facilitate alternate information sharing between adjacent Transformer layers, reducing the discrepancy between the key matrix of the multi-head attention module and the query matrix of the preceding layer. The current layer's feed-forward module weights are then utilized as the weights for the subsequent lower layer, ensuring that the higher layers of the Transformer can accurately refine the information from the lower layers, thus enhancing the network's ability to capture long-term dependencies. Secondly, The self-attention layers in the first two layers of the Transformer are replaced with Fourier modules. This alternation between the encoder and decoder enables the model to capture time-frequency information from the alternating Fourier transform and inverse Fourier transform, thereby enhancing the accuracy of predictions. Finally, use the Weibull failure model to describe the failure probability of the bearing over its lifecycle, and integrate it into the loss function of the neural network training to enhance the physical consistency of the training. The proposed model was applied and verified using both public datasets and datasets of wind turbine gearboxes from real wind farms to confirm its superiority.

Details DOI

JBHI Journal 2024 Journal Article

AntiViralDL: Computational Antiviral Drug Repurposing Using Graph Neural Network and Self-Supervised Learning

Pan Zhang
Xiaowen Hu
Guangdi Li
Lei Deng

Viral infections have emerged as significant public health concerns for decades. Antiviral drugs, specifically designed to combat these infections, have the potential to reduce the disease burden substantially. However, traditional drug development methods, based on biological experiments, are resource-intensive, time-consuming, and low efficiency. Therefore, computational approaches for identifying antiviral drugs can enhance drug development efficiency. In this study, we introduce AntiViralDL, a computational framework for predicting virus-drug associations using self-supervised learning. Initially, we construct a reliable virus-drug association dataset by integrating the existing Drugvirus2 database and FDA-approved virus-drug associations. Utilizing these two datasets, we create a virus-drug association bipartite graph and employ the Light Graph Convolutional Network (LightGCN) to learn embedding representations of viruses and drugs. To address the sparsity of virus-drug association pairs, AntiViralDL incorporates contrastive learning to improve prediction accuracy. We implement data augmentation by adding random noise to the embedding representation space of virus and drug nodes, as opposed to traditional edge and node dropout. Finally, we calculate an inner product to predict virus-drug association relationships. Experimental results reveal that AntiViralDL achieves AUC and AUPR values of 0. 8450 and 0. 8494, respectively, outperforming four benchmarked virus-drug association prediction models. The case study further highlights the efficacy of AntiViralDL in predicting anti-COVID-19 drug candidates.

Details DOI

JBHI Journal 2024 Journal Article

DeepFusionCDR: Employing Multi-Omics Integration and Molecule-Specific Transformers for Enhanced Prediction of Cancer Drug Responses

Xiaowen Hu
Pan Zhang
Jiaxuan Zhang
Lei Deng

Deep learning approaches have demonstrated remarkable potential in predicting cancer drug responses (CDRs), using cell line and drug features. However, existing methods predominantly rely on single-omics data of cell lines, potentially overlooking the complex biological mechanisms governing cell line responses. This paper introduces DeepFusionCDR, a novel approach employing unsupervised contrastive learning to amalgamate multi-omics features, including mutation, transcriptome, methylome, and copy number variation data, from cell lines. Furthermore, we incorporate molecular SMILES-specific transformers to derive drug features from their chemical structures. The unified multi-omics and drug signatures are combined, and a multi-layer perceptron (MLP) is applied to predict IC50 values for cell line-drug pairs. Moreover, this MLP can discern whether a cell line is resistant or sensitive to a particular drug. We assessed DeepFusionCDR's performance on the GDSC dataset and juxtaposed it against cutting-edge methods, demonstrating its superior performance in regression and classification tasks. We also conducted ablation studies and case analyses to exhibit the effectiveness and versatility of our proposed approach. Our results underscore the potential of DeepFusionCDR to enhance CDR predictions by harnessing the power of multi-omics fusion and molecular-specific transformers. The prediction of DeepFusionCDR on TCGA patient data and case study highlight the practical application scenarios of DeepFusionCDR in real-world environments.

Details DOI

EAAI Journal 2024 Journal Article

Global probability distribution structure-sparsity filter pruning for edge fault diagnosis in resource constrained wireless sensor networks

Chunhua Zhao
Baoping Tang
Lei Deng
Yi Huang
Hao Tan

In this paper, a global probability distribution structure-sparsity filter pruning is proposed to address the problem of difficult deployment of diagnostic models in resource constrained wireless sensor networks (WSNs) for edge fault diagnosis. Firstly, local and global weight distributions are analyzed. A global probability distribution weight sparsity method is proposed to obtain the global sparse boundary and the important features of the model. Secondly, according to the distribution of weight amplitude, the problem of uneven weight distribution is disclosed. A computational acceleration sensing method is proposed to reduce the high computational cost caused by small weight amplitude with low importance estimation. Moreover, a novel layer filter clustering pruning scheme is proposed. The intra-layer filters are classified and redundant filters are removed by using the clustering idea according to the predetermined pruning rate. Thus, the structure-sparsity filter pruning is realized. Compared with other advanced filter pruning methods, the proposed method analyzes the global and local model weight distribution and the influence of weight importance on the computational complexity of the model. The proposed method then performs filter pruning on the importance of overall filters for each network layer rather than individual filters. The accuracy of the proposed method on experimental dataset for pruned ResNet8 model under pruning rate at 0. 95 is 99. 12% with 3. 668K parameters and 3. 636MFLOPs. The experimental results elucidate that the proposed method is more suitable for deployment in resource-constrained WSNs for fault diagnosis of rotating machinery. This provides a potential solution for practical engineering applications.

Details DOI

EAAI Journal 2024 Journal Article

Simulation data-driven adaptive frequency filtering focal network for rolling bearing fault diagnosis

Zhen Ming
Baoping Tang
Lei Deng
Qikang Li

The scarcity of well-labeled samples severely limits the application of deep learning-based fault diagnosis methods. To address this issue, this paper proposes a novel domain-adaptive intelligent diagnostic method, termed a simulation data-driven adaptive frequency filtering focal network, which transfers knowledge from dynamic simulation data to unlabeled measured data. First, a dynamic simulation model is established to simulate the coupled behavior of rolling bearings and generate substantial labeled simulation data. Subsequently, an adaptive frequency filter is introduced to suppress random interference components in the frequency domain, complementing conventional time-domain feature extraction methods, thereby reducing the distribution discrepancy between simulation and measured data. Finally, a domain-aware weighted focusing strategy is proposed to adjust sample weights based on predicted domain labels, helping the model focus on difficult samples with significant distribution discrepancies and reduce misclassification. Experimental results demonstrate that the proposed method effectively integrates simulation and measured data, achieving high-accuracy fault diagnosis of rolling bearings in unlabeled measured data. The proposed approach offers a promising fault diagnosis solution in scenarios where obtaining labeled samples is challenging or impractical.

Details DOI

JBHI Journal 2024 Journal Article

Tab-Cox: An Interpretable Deep Survival Analysis Model for Patients With Nasopharyngeal Carcinoma Based on TabNet

Huamei Qi
Yuxuan Hu
Ruohao Fan
Lei Deng

The nutritional status of cancer patients is closely associated with the clinical progression of the disease. A survival analysis model combined with a neural network can predict future disease trends in patients, facilitating early prevention and assisting physicians in making diagnoses. However, the complexity of neural networks and their incompatibility with medical tabular data can reduce the interpretability of the model. To address this issue, thr paper propose a novel survival analysis model called Tab-Cox, which combines TabNet and Cox models. This model is specifically designed to predict the survival outcomes of patients with nasopharyngeal carcinoma. The model utilizes TabNet's sequential attention mechanism to extract more interpretable features, providing an interpretable method for identifying disease risk factors. Consequently, the model ensures accurate survival prediction while also making the results more comprehensible for both patients and doctors. The paper tested the efficacy of the model by conducting experiments on various diverse datasets in comparison with other commonly used survival models. The results showed that the proposed model delivered the highest or second-highest accuracy across all datasets. Furthermore, the paper conducted a comparative interpretability analysis against the classical Cox model. In addition and compare the interpretability of the Tab-Cox model with the classical Cox model and discuss the advantages and disadvantages of its interpretability. This demonstrates that Tab-Cox can assist doctors in identifying risk factors that are challenging to capture using artificial methods.

Details DOI

IJCAI Conference 2022 Conference Paper

Survey on Graph Neural Network Acceleration: An Algorithmic Perspective

Xin Liu
Mingyu Yan
Lei Deng
Guoqi Li
Xiaochun Ye
Dongrui Fan
Shirui Pan
Yuan Xie

Graph neural networks (GNNs) have been a hot spot of recent research and are widely utilized in diverse applications. However, with the use of huger data and deeper models, an urgent demand is unsurprisingly made to accelerate GNNs for more efficient execution. In this paper, we provide a comprehensive survey on acceleration methods for GNNs from an algorithmic perspective. We first present a new taxonomy to classify existing acceleration methods into five categories. Based on the classification, we systematically discuss these methods and highlight their correlations. Next, we provide comparisons from aspects of the efficiency and characteristics of these methods. Finally, we suggest some promising prospects for future research.

PDF Details DOI

NeurIPS Conference 2022 Conference Paper

Toward Robust Spiking Neural Network Against Adversarial Perturbation

Ling Liang
Kaidi Xu
Xing Hu
Lei Deng
Yuan Xie

As spiking neural networks (SNNs) are deployed increasingly in real-world efficiency critical applications, the security concerns in SNNs attract more attention. Currently, researchers have already demonstrated an SNN can be attacked with adversarial examples. How to build a robust SNN becomes an urgent issue. Recently, many studies apply certified training in artificial neural networks (ANNs), which can improve the robustness of an NN model promisely. However, existing certifications cannot transfer to SNNs directly because of the distinct neuron behavior and input formats for SNNs. In this work, we first design S-IBP and S-CROWN that tackle the non-linear functions in SNNs' neuron modeling. Then, we formalize the boundaries for both digital and spike inputs. Finally, we demonstrate the efficiency of our proposed robust training method in different datasets and model architectures. Based on our experiment, we can achieve a maximum $37. 7\%$ attack error reduction with $3. 7\%$ original accuracy loss. To the best of our knowledge, this is the first analysis on robust training of SNNs.

PDF Details

IJCAI Conference 2021 Conference Paper

Exploiting Spiking Dynamics with Spatial-temporal Feature Normalization in Graph Learning

Mingkun Xu
YuJie Wu
Lei Deng
Faqiang Liu
Guoqi Li
Jing Pei

Biological spiking neurons with intrinsic dynamics underlie the powerful representation and learning capabilities of the brain for processing multimodal information in complex environments. Despite recent tremendous progress in spiking neural networks (SNNs) for handling Euclidean-space tasks, it still remains challenging to exploit SNNs in processing non-Euclidean-space data represented by graph data, mainly due to the lack of effective modeling framework and useful training techniques. Here we present a general spike-based modeling framework that enables the direct training of SNNs for graph learning. Through spatial-temporal unfolding for spiking data flows of node features, we incorporate graph convolution filters into spiking dynamics and formalize a synergistic learning paradigm. Considering the unique features of spike representation and spiking dynamics, we propose a spatial-temporal feature normalization (STFN) technique suitable for SNN to accelerate convergence. We instantiate our methods into two spiking graph models, including graph convolution SNNs and graph attention SNNs, and validate their performance on three node-classification benchmarks, including Cora, Citeseer, and Pubmed. Our model can achieve comparable performance with the state-of-the-art graph neural network (GNN) models with much lower computation costs, demonstrating great benefits for the execution on neuromorphic hardware and prompting neuromorphic applications in graphical scenarios.

PDF Details DOI

EAAI Journal 2021 Journal Article

Fault source location of wind turbine based on heterogeneous nodes complex network

Kai Zhang
Baoping Tang
Lei Deng
Xiaoxia Yu
Jing Wei

Previous studies regarding wind turbine health management based on wind turbine supervisory control and data acquisition (SCADA) signals focused on detecting incipient failure. In this case, most current SCADA alarm systems adopt a single or multi-signal threshold trigger alarm. When a wind turbine is operating in a large wind farm, it is catastrophic for maintenance personnel lied in the reason that the current SCADA system may trigger many alarms in the short term. Therefore, the fault source must be identified and the candidate fault sources sorted. This study aims to obtain the fault location of wind turbines based on a complex network, which does not require a large number of historical monitoring signals. A directed graph of the wind turbine and fault location theoretical framework is established; an improvement is achieved based on the heterogeneity of nodes abstracted by components of a wind turbine and variables of SCADA. The proposed fault location method based on heterogeneous nodes complex networks (HNCN) is improved based on degree-based mean-field theory and is verified by spreading simulated data and four SCADA alarm cases at a wind farm located in Liaoning, China.

Details DOI

AAAI Conference 2021 Conference Paper

Going Deeper With Directly-Trained Larger Spiking Neural Networks

Hanle Zheng
YuJie Wu
Lei Deng
Yifan Hu
Guoqi Li

Spiking neural networks (SNNs) are promising in a bioplausible coding for spatio-temporal information and eventdriven signal processing, which is very suited for energyefficient implementation in neuromorphic hardware. However, the unique working mode of SNNs makes them more difficult to train than traditional networks. Currently, there are two main routes to explore the training of deep SNNs with high performance. The first is to convert a pre-trained ANN model to its SNN version, which usually requires a long coding window for convergence and cannot exploit the spatio-temporal features during training for solving temporal tasks. The other is to directly train SNNs in the spatio-temporal domain. But due to the binary spike activity of the firing function and the problem of gradient vanishing or explosion, current methods are restricted to shallow architectures and thereby difficult in harnessing large-scale datasets (e. g. ImageNet). To this end, we propose a threshold-dependent batch normalization (tdB- N) method based on the emerging spatio-temporal backpropagation, termed “STBP-tdBN”, enabling direct training of a very deep SNN and the efficient implementation of its inference on neuromorphic hardware. With the proposed method and elaborated shortcut connection, we significantly extend directly-trained SNNs from a shallow structure (<10 layer) to a very deep structure (50 layers). Furthermore, we theoretically analyze the effectiveness of our method based on “Block Dynamical Isometry” theory. Finally, we report superior accuracy results including 93. 15% on CIFAR-10, 67. 8% on DVS-CIFAR10, and 67. 05% on ImageNet with very few timesteps. To our best knowledge, it’s the first time to explore the directly-trained deep SNNs with high performance on ImageNet. We believe this work shall pave the way of fully exploiting the advantages of SNNs and attract more researchers to contribute in this field.

PDF Details

AAAI Conference 2019 Conference Paper

Direct Training for Spiking Neural Networks: Faster, Larger, Better

YuJie Wu
Lei Deng
Guoqi Li
Jun Zhu
Yuan Xie
Luping Shi

Spiking neural networks (SNNs) that enables energy efficient implementation on emerging neuromorphic hardware are gaining more attention. Yet now, SNNs have not shown competitive performance compared with artificial neural networks (ANNs), due to the lack of effective learning algorithms and efficient programming frameworks. We address this issue from two aspects: (1) We propose a neuron normalization technique to adjust the neural selectivity and develop a direct learning algorithm for deep SNNs. (2) Via narrowing the rate coding window and converting the leaky integrate-and-fire (LIF) model into an explicitly iterative version, we present a Pytorch-based implementation method towards the training of large-scale SNNs. In this way, we are able to train deep SNNs with tens of times speedup. As a result, we achieve significantly better accuracy than the reported works on neuromorphic datasets (N-MNIST and DVS- CIFAR10), and comparable accuracy as existing ANNs and pre-trained SNNs on non-spiking datasets (CIFAR10). To our best knowledge, this is the first work that demonstrates direct training of deep SNNs with high performance on CIFAR10, and the efficient implementation provides a new way to explore the potential of SNNs.

PDF Details

NeurIPS Conference 2018 Conference Paper

HitNet: Hybrid Ternary Recurrent Neural Network

Peiqi Wang
Xinfeng Xie
Lei Deng
Guoqi Li
Dongsheng Wang
Yuan Xie

Quantization is a promising technique to reduce the model size, memory footprint, and massive computation operations of recurrent neural networks (RNNs) for embedded devices with limited resources. Although extreme low-bit quantization has achieved impressive success on convolutional neural networks, it still suffers from huge accuracy degradation on RNNs with the same low-bit precision. In this paper, we first investigate the accuracy degradation on RNN models under different quantization schemes, and the distribution of tensor values in the full precision model. Our observation reveals that due to the difference between the distributions of weights and activations, different quantization methods are suitable for different parts of models. Based on our observation, we propose HitNet, a hybrid ternary recurrent neural network, which bridges the accuracy gap between the full precision model and the quantized model. In HitNet, we develop a hybrid quantization method to quantize weights and activations. Moreover, we introduce a sloping factor motivated by prior work on Boltzmann machine to activation functions, further closing the accuracy gap between the full precision model and the quantized model. Overall, our HitNet can quantize RNN models into ternary values, {-1, 0, 1}, outperforming the state-of-the-art quantization methods on RNN models significantly. We test it on typical RNN models, such as Long-Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), on which the results outperform previous work significantly. For example, we improve the perplexity per word (PPW) of a ternary LSTM on Penn Tree Bank (PTB) corpus from 126 (the state-of-the-art result to the best of our knowledge) to 110. 3 with a full precision model in 97. 2, and a ternary GRU from 142 to 113. 5 with a full precision model in 102. 7.

PDF Details

NeurIPS Conference 2018 Conference Paper

TETRIS: TilE-matching the TRemendous Irregular Sparsity

Yu Ji
Ling Liang
Lei Deng
Youyang Zhang
Youhui Zhang
Yuan Xie

Compressing neural networks by pruning weights with small magnitudes can significantly reduce the computation and storage cost. Although pruning makes the model smaller, it is difficult to get practical speedup in modern computing platforms such as CPU and GPU due to the irregularity. Structural pruning has attract a lot of research interest to make sparsity hardware-friendly. Increasing the sparsity granularity can lead to better hardware utilization, but it will compromise the sparsity for maintaining accuracy. In this work, we propose a novel method, TETRIS, to achieve both better hardware utilization and higher sparsity. Just like a tile-matching game, we cluster the irregularly distributed weights with small value into structured groups by reordering the input/output dimension and structurally prune them. Results show that it can achieve comparable sparsity with the irregular element-wise pruning and demonstrate negligible accuracy loss. The experiments also shows ideal speedup, which is proportional to the sparsity, on GPU platforms. Our proposed method provides a new solution toward algorithm and architecture co-optimization for accuracy-efficiency trade-off.

PDF Details