Author name cluster

Lei Xie

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

17 papers

1 author row

AAAI Conference 2026 Conference Paper

Hearing More with Less: Multi-Modal Retrieval-and-Selection Augmented Conversational LLM-Based ASR

Bingshen Mu
Hexin Liu
Hongfei Xue
Kun Wei
Lei Xie

Automatic Speech Recognition (ASR) aims to convert human speech content into corresponding text. In conversational scenarios, effectively utilizing context can enhance its accuracy. Large Language Models' (LLMs) exceptional long-context understanding and reasoning abilities enable LLM-based ASR (LLM-ASR) to leverage historical context for recognizing conversational speech, which has a high degree of contextual relevance. However, existing conversational LLM-ASR methods use a fixed number of preceding utterances or the entire conversation history as context, resulting in significant ASR confusion and computational costs due to massive irrelevant and redundant information. This paper proposes a multi-modal retrieval-and-selection method named MARS that augments conversational LLM-ASR by enabling it to retrieve and select the most relevant acoustic and textual historical context for the current utterance. Specifically, multi-modal retrieval obtains a set of candidate historical contexts, each exhibiting high acoustic or textual similarity to the current utterance. Multi-modal selection calculates the acoustic and textual similarities for each retrieved candidate historical context and, by employing our proposed near-ideal ranking method to consider both similarities, selects the best historical context. Evaluations on the Interspeech 2025 Multilingual Conversational Speech Language Model Challenge dataset show that the LLM-ASR, when trained on only 1.5K hours of data and equipped with the MARS, outperforms the state-of-the-art top-ranking system trained on 179K hours of data.

PDF Details DOI

AAAI Conference 2026 Conference Paper

KALL-E: Autoregressive Speech Synthesis with Next-Distribution Prediction

Kangxiang Xia
Xinfa Zhu
Jixun Yao
Wenjie Tian
Wenhao Li
Lei Xie

We introduce KALL-E, a novel autoregressive (AR) language model for text-to-speech (TTS) synthesis that operates by predicting the next distribution of continuous speech frames. Unlike existing methods, KALL-E directly models the continuous speech distribution conditioned on text, eliminating the need for any diffusion-based components. Specifically, we utilize a Flow-VAE to extract a continuous latent speech representation from waveforms, instead of relying on discrete speech tokens. A single AR Transformer is then trained to predict these continuous speech distributions from text, optimizing a Kullback–Leibler divergence loss as its objective. Experimental results demonstrate that KALL-E achieves superior speech synthesis quality and can even adapt to a target speaker from just a single sample. Importantly, KALL-E provides a more direct and effective approach for utilizing continuous speech representations in TTS.

PDF Details DOI

AAAI Conference 2026 Conference Paper

WenetSpeech-Yue: A Large-Scale Cantonese Speech Corpus with Multi-dimensional Annotation

Longhao Li
Zhao Guo
Hongjie Chen
Yuhang Dai
Ziyu Zhang
Hongfei Xue
Tianlun Zuo
Chengyou Wang

The development of speech understanding and generation has been significantly accelerated by the availability of large-scale, high-quality speech datasets. Among these, ASR and TTS are regarded as the most established and fundamental tasks. However, for Cantonese (Yue Chinese), spoken by approximately 84.9 million native speakers worldwide, limited annotated resources have hindered progress and resulted in suboptimal ASR and TTS performance. To address this challenge, we propose WenetSpeech-Pipe, an integrated pipeline for building large-scale speech corpus with multi-dimensional annotation tailored for speech understanding and generation. Based on this pipeline, we release WenetSpeech-Yue, the first large-scale Cantonese speech corpus with multi-dimensional annotation for ASR and TTS, covering 21,800 hours across 10 domains with annotations including ASR transcription, text confidence, speaker identity, age, gender, speech quality scores, among other annotations. We also release WSYue-eval, a comprehensive Cantonese benchmark with two components: WSYue-ASR-eval, a manually annotated set for evaluating ASR on short and long utterances, code-switching, and diverse acoustic conditions, and WSYue-TTS-eval, with base and coverage subsets for standard and generalization testing. Experimental results show that models trained on WenetSpeech-Yue achieve competitive results against state-of-the-art (SOTA) Cantonese ASR and TTS systems, including commercial and LLM-based models, highlighting the value of our dataset and pipeline.

PDF Details DOI

AIIM Journal 2025 Journal Article

ContraDTI: Improved drug–target interaction prediction via multi-view contrastive learning

Zhirui Liao
Lei Xie
Shanfeng Zhu

Drug–target interaction (DTI) identification is one of the crucial issues in the field of drug discovery. Machine learning approaches offer efficient ways to address this issue, reducing expensive and time-consuming laboratory experiments. However, the scarcity of annotated drug data with labels restricts supervised machine learning applications to DTI prediction. Drawing inspiration from recent advances in contrastive learning, we present ContraDTI—a novel framework that adopts multi-view contrastive learning to overcome data limitations in this paper. Our model considers the molecular graph of a drug as the main view and the SMILES string of a drug as the side view, employing two types of loss functions for the contrast of the main view and the cross-view alignment between the main and the side views. Extensive experiments on both single-target and multi-target DTI datasets demonstrate that ContraDTI enhances the classification performance of DTI prediction, particularly when labeled data is scarce. ContraDTI can be a powerful tool for DTI prediction in data-limited scenarios. The code of this paper is available at https: //github. com/zhiruiliao/ContraDTI.

Details DOI

AAAI Conference 2025 Conference Paper

Drop the Beat! Freestyler for Accompaniment Conditioned Rapping Voice Generation

Ziqian Ning
Shuai Wang
Yuepeng Jiang
Jixun Yao
Lei He
Shifeng Pan
Jie Ding
Lei Xie

Rap, a prominent genre of vocal performance, remains underexplored in vocal generation. General vocal synthesis depends on precise note and duration inputs, requiring users to have related musical knowledge, which limits flexibility. In contrast, rap typically features simpler melodies, with a core focus on a strong rhythmic sense that harmonizes with accompanying beats. In this paper, we propose Freestyler, the first system that generates rapping vocals directly from lyrics and accompaniment inputs. Freestyler utilizes language model-based token generation, followed by a conditional flow matching model to produce spectrograms and a neural vocoder to restore audio. It allows a 3-second prompt to enable zero-shot timbre control. Due to the scarcity of publicly available rap datasets, we also present RapBank, a rap song dataset collected from the internet, alongside a meticulously designed processing pipeline. Experimental results show that Freestyler produces high-quality rapping voice generation with enhanced naturalness and strong alignment with accompanying beats, both stylistically and rhythmically.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

MixSignGraph: A Sign Sequence is Worth Mixed Graphs of Nodes

Shiwei Gan
Yafeng Yin
Zhiwei Jiang
Lei Xie
Sanglu Lu
Hongkai Wen

Recent advances in sign language research have benefited from CNN-based backbones, which are primarily transferred from traditional computer vision tasks (\eg object detection, image recognition). However, these CNN-based backbones usually excel at extracting features like contours and texture, but may struggle with capturing sign-related features. To capture such sign-related features, SignGraph model extracts the cross-region sign features by building the Local Sign Graph (LSG) module and the Temporal Sign Graph (TSG) module. However, we emphasize that although capturing cross-region dependencies can improve sign language performance, it may degrade the representation quality of local regions. To mitigate this, we introduce MixSignGraph, which represents sign sequences as a group of mixed graphs for feature extraction. Specifically, besides the LSG module and TSG module that model the intra-frame and inter-frame cross-regions features, we design a simple yet effective Hierarchical Sign Graph (HSG) module, which enhances local region representations following the extraction of cross-region features, by aggregating the same-region features from different-granularity feature maps of a frame, \ie to boost discriminative local features. In addition, to further improve the performance of gloss-free sign language task, we propose a simple yet counter-intuitive Text-based CTC Pre-training (TCTC) method, which generates pseudo gloss labels from text sequences for model pre-training. Extensive experiments conducted on the current five sign language datasets demonstrate that MixSignGraph surpasses the most current models on multiple sign language tasks across several datasets, without relying on any additional cues. Code and models are available at: \href{https: //github. com/gswycf/SignLanguage}{\textcolor{blue}{https: //github. com/gswycf/SignLanguage}}.

PDF Details

AAAI Conference 2025 Conference Paper

StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching

Jixun Yao
Yang Yuguang
Yu Pan
Ziqian Ning
Jianhao Ye
Hongbin Zhou
Lei Xie

Zero-shot voice conversion (VC) aims to transfer the timbre from the source speaker to an arbitrary unseen speaker while preserving the original linguistic content. Despite recent advancements in zero-shot VC using language model-based or diffusion-based approaches, several challenges remain: 1) current approaches primarily focus on adapting timbre from unseen speakers and are unable to transfer style and timbre to different unseen speakers independently; 2) these approaches often suffer from slower inference speeds due to the autoregressive modeling methods or the need for numerous sampling steps; 3) the quality and similarity of the converted samples are still not fully satisfactory. To address these challenges, we propose a Style controllable zero-shot VC approach named StableVC, which aims to transfer timbre and style from source speech to different unseen target speakers. Specifically, we decompose speech into linguistic content, timbre, and style, and then employ a conditional flow matching module to reconstruct the high-quality mel-spectrogram based on these decomposed features. To effectively capture timbre and style in a zero-shot manner, we introduce a novel dual attention mechanism with an adaptive gate, rather than using conventional feature concatenation. With this non-autoregressive design, StableVC can efficiently capture the intricate timbre and style from different unseen speakers and generate high-quality speech significantly faster than real-time. Experiments demonstrate that our proposed StableVC outperforms state-of-the-art baseline systems in zero-shot VC and achieves flexible control over timbre and style from different unseen speakers. Moreover, StableVC offers approximately 25x and 1.65x faster sampling compared to autoregressive and diffusion-based baselines.

PDF Details DOI

AAAI Conference 2024 Conference Paper

A Diffusion-Based Framework for Multi-Class Anomaly Detection

Haoyang He
Jiangning Zhang
Hongxu Chen
Xuhai Chen
Zhishan Li
Xu Chen
Yabiao Wang
Chengjie Wang

Reconstruction-based approaches have achieved remarkable outcomes in anomaly detection. The exceptional image reconstruction capabilities of recently popular diffusion models have sparked research efforts to utilize them for enhanced reconstruction of anomalous images. Nonetheless, these methods might face challenges related to the preservation of image categories and pixel-wise structural integrity in the more practical multi-class setting. To solve the above problems, we propose a Difusion-based Anomaly Detection (DiAD) framework for multi-class anomaly detection, which consists of a pixel-space autoencoder, a latent-space Semantic-Guided (SG) network with a connection to the stable diffusion’s denoising network, and a feature-space pre-trained feature extractor. Firstly, The SG network is proposed for reconstructing anomalous regions while preserving the original image’s semantic information. Secondly, we introduce Spatial-aware Feature Fusion (SFF) block to maximize reconstruction accuracy when dealing with extensively reconstructed areas. Thirdly, the input and reconstructed images are processed by a pre-trained feature extractor to generate anomaly maps based on features extracted at different scales. Experiments on MVTec-AD and VisA datasets demonstrate the effectiveness of our approach which surpasses the state-of-the-art methods, e.g., achieving 96.8/52.6 and 97.2/99.0 (AUROC/AP) for localization and detection respectively on multi-class MVTec-AD dataset. Code will be available at https://lewandofskee.github.io/projects/diad.

PDF Details DOI

YNIMG Journal 2024 Journal Article

Bundle-specific tractogram distribution estimation using higher-order streamline differential equation

Yuanjing Feng
Lei Xie
Jingqiang Wang
Qiyuan Tian
Jianzhong He
Qingrun Zeng
Fei Gao

Streamline tractography locally traces peak directions extracted from fiber orientation distribution (FOD) functions, lacking global information about the trend of the whole fiber bundle. Therefore, it is prone to producing erroneous tracks while missing true positive connections. In this work, we propose a new bundle-specific tractography (BST) method based on a bundle-specific tractogram distribution (BTD) function, which directly reconstructs the fiber trajectory from the start region to the termination region by incorporating the global information in the fiber bundle mask. A unified framework for any higher-order streamline differential equation is presented to describe the fiber bundles with disjoint streamlines defined based on the diffusion vectorial field. At the global level, the tractography process is simplified as the estimation of BTD coefficients by minimizing the energy optimization model, and is used to characterize the relations between BTD and diffusion tensor vector under the prior guidance by introducing the tractogram bundle information to provide anatomic priors. Experiments are performed on simulated Hough, Sine, Circle data, ISMRM 2015 Tractography Challenge data, FiberCup data, and in vivo data from the Human Connectome Project (HCP) for qualitative and quantitative evaluation. Results demonstrate that our approach reconstructs complex fiber geometry more accurately. BTD reduces the error deviation and accumulation at the local level and shows better results in reconstructing long-range, twisting, and large fanning tracts.

Details DOI

NeurIPS Conference 2024 Conference Paper

D-LLM: A Token Adaptive Computing Resource Allocation Strategy for Large Language Models

Yikun Jiang
Huanyu Wang
Lei Xie
Hanbin Zhao
Chao Zhang
Hui Qian
John C. Lui

Large language models have shown an impressive societal impact owing to their excellent understanding and logical reasoning skills. However, such strong ability relies on a huge amount of computing resources, which makes it difficult to deploy LLMs on computing resource-constrained platforms. Currently, LLMs process each token equivalently, but we argue that not every word is equally important. Some words should not be allocated excessive computing resources, particularly for dispensable terms in simple questions. In this paper, we propose a novel dynamic inference paradigm for LLMs, namely D-LLMs, which adaptively allocate computing resources in token processing. We design a dynamic decision module for each transformer layer that decides whether a network unit should be executed or skipped. Moreover, we tackle the issue of adapting D-LLMs to real-world applications, specifically concerning the missing KV-cache when layers are skipped. To overcome this, we propose a simple yet effective eviction policy to exclude the skipped layers from subsequent attention calculations. The eviction policy not only enables D-LLMs to be compatible with prevalent applications but also reduces considerable storage resources. Experimentally, D-LLMs show superior performance, in terms of computational cost and KV storage utilization. It can reduce up to 45\% computational cost and KV storage on Q&A, summarization, and math solving tasks, 50\% on commonsense reasoning tasks.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection

Haoyang He
Yuhu Bai
Jiangning Zhang
Qingdong He
Hongxu Chen
Zhenye Gan
Chengjie Wang
Xiangtai Li

Recent advancements in anomaly detection have seen the efficacy of CNN- and transformer-based approaches. However, CNNs struggle with long-range dependencies, while transformers are burdened by quadratic computational complexity. Mamba-based models, with their superior long-range modeling and linear efficiency, have garnered substantial attention. This study pioneers the application of Mamba to multi-class unsupervised anomaly detection, presenting MambaAD, which consists of a pre-trained encoder and a Mamba decoder featuring (Locality-Enhanced State Space) LSS modules at multi-scales. The proposed LSS module, integrating parallel cascaded (Hybrid State Space) HSS blocks and multi-kernel convolutions operations, effectively captures both long-range and local information. The HSS block, utilizing (Hybrid Scanning) HS encoders, encodes feature maps into five scanning methods and eight directions, thereby strengthening global connections through the (State Space Model) SSM. The use of Hilbert scanning and eight directions significantly improves feature sequence modeling. Comprehensive experiments on six diverse anomaly detection datasets and seven metrics demonstrate state-of-the-art performance, substantiating the method's effectiveness. The code and models are available at https: //lewandofskee. github. io/projects/MambaAD.

PDF Details DOI

EAAI Journal 2024 Journal Article

Towards efficient filter pruning via adaptive automatic structure search

Xiaozhou Xu
Jun Chen
Zhishan Li
Hongye Su
Lei Xie

Filter pruning is a critical technique for compressing large convolutional neural networks, making it possible to deploy deep networks on resource-limited edge devices. However, previous pruning methods typically concentrate on removing filters with rule-of-thumb designs and empirically set the pruning rate per layer, which is prone to produce sub-optimal pruning. To address this issue, we develop a Filter Pruning method via Adaptive Automatic Structure Search (FP-AASS), which treats filter pruning as a structure optimization task. In FP-AASS, we employ the artificial bee colony algorithm to automatically search for the optimal pruned structure that meets the FLOPs and parameters constraints. The structure search process is divided into two stages through an additional adjustment phase to reduce the time consumption caused by the large search space. We also adopt adaptive batch normalization in the evaluation process to accelerate the search process and reinforce the relevance between the evaluation accuracy and the final converged accuracy achieved by the pruning candidates. Through extensive experiments, FP-AASS has been demonstrated to be an effective automatic filter pruning method. For instance, FP-AASS slims a standard ResNet-50 to a narrower one with 56. 8% FLOPs and 55. 0% parameters reduction with only a 1. 50% decrease in top-1 accuracy on ImageNet-1K. In contrast, under similar FLOPs pruning rates, other representative methods show a top-1 accuracy decrease of at least 1. 70% or more.

Details DOI

IJCAI Conference 2023 Conference Paper

Contrastive Learning for Sign Language Recognition and Translation

Shiwei Gan
Yafeng Yin
Zhiwei Jiang
Kang Xia
Lei Xie
Sanglu Lu

There are two problems that widely exist in current end-to-end sign language processing architecture. One is the CTC spike phenomenon which weakens the visual representational ability in Continuous Sign Language Recognition (CSLR). The other one is the exposure bias problem which leads to the accumulation of translation errors during inference in Sign Language Translation (SLT). In this paper, we tackle these issues by introducing contrast learning, aiming to enhance both visual-level feature representation and semantic-level error tolerance. Specifically, to alleviate CTC spike phenomenon and enhance visual-level representation, we design a visual contrastive loss by minimizing visual feature distance between different augmented samples of frames in one sign video, so that the model can further explore features by utilizing numerous unlabeled frames in an unsupervised way. To alleviate exposure bias problem and improve semantic-level error tolerance, we design a semantic contrastive loss by re-inputting the predicted sentence into semantic module and comparing features of ground-truth sequence and predicted sequence, for exposing model to its own mistakes. Besides, we propose two new metrics, i. e. , Blank Rate and Consecutive Wrong Word Rate to directly reflect our improvement on the two problems. Extensive experimental results on current sign language datasets demonstrate the effectiveness of our approach, which achieves state-of-the-art performance.

PDF Details DOI

AAAI Conference 2023 Conference Paper

UniSyn: An End-to-End Unified Model for Text-to-Speech and Singing Voice Synthesis

Yi Lei
Shan Yang
Xinsheng Wang
Qicong Xie
Jixun Yao
Lei Xie
Dan Su

Text-to-speech (TTS) and singing voice synthesis (SVS) aim at generating high-quality speaking and singing voice according to textual input and music scores, respectively. Unifying TTS and SVS into a single system is crucial to the applications requiring both of them. Existing methods usually suffer from some limitations, which rely on either both singing and speaking data from the same person or cascaded models of multiple tasks. To address these problems, a simplified elegant framework for TTS and SVS, named UniSyn, is proposed in this paper. It is an end-to-end unified model that can make a voice speak and sing with only singing or speaking data from this person. To be specific, a multi-conditional variational autoencoder (MC-VAE), which constructs two independent latent sub-spaces with the speaker- and style-related (i.e. speak or sing) conditions for flexible control, is proposed in UniSyn. Moreover, supervised guided-VAE and timbre perturbation with the Wasserstein distance constraint are leveraged to further disentangle the speaker timbre and style. Experiments conducted on two speakers and two singers demonstrate that UniSyn can generate natural speaking and singing voice without corresponding training data. The proposed approach outperforms the state-of-the-art end-to-end voice generation work, which proves the effectiveness and advantages of UniSyn.

PDF Details DOI

NeurIPS Conference 2022 Conference Paper

A Transformer-Based Object Detector with Coarse-Fine Crossing Representations

Zhishan Li
Ying Nie
Kai Han
Jianyuan Guo
Lei Xie
Yunhe Wang

Transformer-based object detectors have shown competitive performance recently. Compared with convolutional neural networks limited by the relatively small receptive fields, the advantage of transformer for visual tasks is the capacity to perceive long-range dependencies among all image patches, while the deficiency is that the local fine-grained information is not fully excavated. In this paper, we introduce the Coarse-grained and Fine-grained crossing representations to build an efficient Detection Transformer (CFDT). Specifically, we propose a local-global cross fusion module to establish the connection between local fine-grained features and global coarse-grained features. Besides, we propose a coarse-fine aware neck which enables detection tokens to interact with both coarse-grained and fine-grained features. Furthermore, an efficient feature integration module is presented for fusing multi-scale representations from different stages. Experimental results on the COCO dataset demonstrate the effectiveness of the proposed method. For instance, our CFDT achieves 48. 1 AP with 173G FLOPs, which possesses higher accuracy and less computation compared with the state-of-the-art transformer-based detector ViDT. Code will be available at https: //gitee. com/mindspore/models/tree/master/research/cv/CFDT.

PDF Details

IJCAI Conference 2020 Conference Paper

Improving Attention Mechanism in Graph Neural Networks via Cardinality Preservation

Shuo Zhang
Lei Xie

Graph Neural Networks (GNNs) are powerful for the representation learning of graph-structured data. Most of the GNNs use a message-passing scheme, where the embedding of a node is iteratively updated by aggregating the information from its neighbors. To achieve a better expressive capability of node influences, attention mechanism has grown to be popular to assign trainable weights to the nodes in aggregation. Though the attention-based GNNs have achieved remarkable results in various tasks, a clear understanding of their discriminative capacities is missing. In this work, we present a theoretical analysis of the representational properties of the GNN that adopts the attention mechanism as an aggregator. Our analysis determines all cases when those attention-based GNNs can always fail to distinguish certain distinct structures. Those cases appear due to the ignorance of cardinality information in attention-based aggregation. To improve the performance of attention-based GNNs, we propose cardinality preserved attention (CPA) models that can be applied to any kind of attention mechanisms. Our experiments on node and graph classification confirm our theoretical analysis and show the competitive performance of our CPA models. The code is available online: https: //github. com/zetayue/CPA.

PDF Details DOI

NeurIPS Conference 2020 Conference Paper

Sequence to Multi-Sequence Learning via Conditional Chain Mapping for Mixture Signals

Jing Shi
Xuankai Chang
Pengcheng Guo
Shinji Watanabe
Yusuke Fujita
Jiaming Xu
Bo Xu
Lei Xie

Neural sequence-to-sequence models are well established for applications which can be cast as mapping a single input sequence into a single output sequence. In this work, we focus on one-to-many sequence transduction problems, such as extracting multiple sequential sources from a mixture sequence. We extend the standard sequence-to-sequence model to a conditional multi-sequence model, which explicitly models the relevance between multiple output sequences with the probabilistic chain rule. Based on this extension, our model can conditionally infer output sequences one-by-one by making use of both input and previously-estimated contextual output sequences. This model additionally has a simple and efficient stop criterion for the end of the transduction, making it able to infer the variable number of output sequences. We take speech data as a primary test field to evaluate our methods since the observed speech data is often composed of multiple sources due to the nature of the superposition principle of sound waves. Experiments on several different tasks including speech separation and multi-speaker speech recognition show that our conditional multi-sequence models lead to consistent improvements over the conventional non-conditional models.

PDF Details