Author name cluster

Tao Yang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

32 papers

2 author rows

JBHI Journal 2026 Journal Article

Advanced Camera-Based Scoliosis Screening via Deep Learning Detection and Fusion of Trunk, Limb, and Skeleton Features

Ziyan Wang
Yi Zhou
Ninghui Xu
Yuqin Zhou
Heran Zhao
Zhiyong Chang
Zhigang Hu
Xiao Han

Scoliosis significantly impacts quality of life, highlighting the need for effective early scoliosis screening (SS) and intervention. However, current SS methods often involve physical contact, undressing, or radiation exposure. This study introduces an innovative, non-invasive SS approach utilizing a monocular RGB camera that eliminates the need for undressing, sensor attachment, and radiation exposure. We introduce a novel approach that employs Parameterized Human 3D Reconstruction (PH3DR) to reconstruct 3D human models, thereby effectively eliminating clothing obstructions, seamlessly integrated with an ISANet segmentation network, which has been enhanced by Multi-Scale Fusion Attention (MSFA) module we proposed for facilitating the segmentation of distinct human trunk and limb features (HTLF), capturing body surface asymmetries related to scoliosis. Additionally, we propose a Swin Transformer-enhanced CMU-Pose to extract human skeleton features (HSF), identifying skeletal asymmetries crucial for SS. Finally, we develop a fusion model that integrates the HTLF and HSF, combining surface morphology and skeletal features to improve the precision of SS. The experiments demonstrated that PH3DR and MSFA significantly improved the segmentation and extraction of HTLF, whereas ST-based CMU-Pose substantially enhanced the extraction of HSF. Our final model achieved a comparable F1 (0. 895 $\pm$ 0. 014) to the best-performing baseline model, with only 0. 79% of the parameters and 1. 64% of the FLOPs, achieving 36 FPS–significantly higher than the best-performing baseline model (10 FPS). Moreover, our model outperformed two spine surgeons, one less experienced and the other moderately experienced. With its patient-friendly, privacy-preserving, and easily deployable solution, this approach is particularly well-suited for early SS and routine monitoring.

Details DOI

EAAI Journal 2026 Journal Article

Appearance-semantic graphical model for human-object interaction detection

Qing Ye
Tao Yang
Xikun Wang
Yongmei Zhang

Human object interaction detection has important applications in fields such as surveillance system and autonomous driving. But the existing methods of human-object interaction detection include the following limitations: insufficient use of spatial information and insufficient referential information when detecting interaction categories. Aiming at these problems, we proposed an appearance-semantic graphical model for human-object interaction detection algorithm. To solve the insufficient use of spatial information problem, we proposed a Depthwise Separable convolutional Multi-Resolution feature Enhancement Network, which extracts features through multiple resolution feature branches in parallel and utilizes the attention mechanism to shift the focus of the network to the region where the interaction occurs. This network can more adequately extract the image features and enhance the feature expression. To solve the problem of insufficient referential information, we proposed an Appearance - Semantic interaction reasoning Graphical Model. The model consists of a two-layer graphical model, constructing the graphical model with the target appearance feature vectors and word vectors as the nodes of the appearance layer and the semantic layer. The corresponding nodes are connected by unidirectional edges, using graph convolution for information transfer, which enriches the referential information in the recognition process. The mean average precision of the algorithm proposed in this paper is 63. 31% (Scenario 2) and 31. 86% (Known Object) on the interaction behavior recognition benchmark datasets Verbs in Common Objects in Context and Humans Interacting with Common Objects, respectively. The effectiveness of our algorithm has been verified in comparison with some other human-object interaction detection methods. The method shows great potential in practical applications such as surveillance system and autonomous driving.

Details DOI

JBHI Journal 2026 Journal Article

Head-and-Neck Organs Segmentation in CT Based on Spatial Prior and Shape Description

Chengyang An
Tao Yang
Xiao Sun
Yu Qiao
Yubing Li
Jilan Jiang
Ling Zhu
LIsheng Wang

Accurate delineation of organs at risk (OARs) is critical for effective radiotherapy in head and neck cancer, and different deep learning methods have been proposed for this task. Although these methods can effectively segment large organs, they all face challenges in segmenting different small organs with high accuracy, due to large numbers, complex distributions, and diverse shapes of small organs in the head and neck region. To address this challenge, this paper proposes a novel segmentation framework that incorporates spatial distribution information of all organs and shape priors of small organs into deep networks to constrain and enhance small organ segmentation. First, a spatial guidance network (SG-Net) is proposed to generate spatial guidance maps (SGMs) of organs, emphasizing the boundaries of different organs and their spatial positional relationships, thereby providing useful spatial cues to constrain organ segmentation. Second, for small-volume organs, we specifically design a deep shape description module (DSDM) to extract organ-specific shape features from CT images and integrate them into the original deep features to enhance the features' sensitivity to shape constraints. Finally, a regularization term is employed to reduce excessive smoothing in the predicted probability maps of the deep network, preserving the shape details of small organs. With this framework, while the segmentation accuracy of large organs is maintained, small organ segmentation is significantly improved. Experimental results demonstrate its effectiveness for segmentation of small organs, with a significant improvement over state-of-the-art methods.

Details DOI

AAAI Conference 2026 Conference Paper

HiNCoT: Hierarchical Nonlinear Continuous Transform-based Tensor Representation for Multi-Dimensional Data Recovery

Tao Yang
Weihao Wu
Tingzhu Huang

Recently, continuous transform-based tensor representation has emerged as a promising tool for multi-dimensional data recovery. However, the existing continuous transforms are essentially single-layer linear mappings, which limits their ability to capture the complex relationships inherent in multi-dimensional data. To overcome this limitation, we propose a Hierarchical Nonlinear Continuous Transform-based Tensor Representation (HiNCoT) for multi-dimensional data recovery. By leveraging the hierarchical nonlinear continuous transform, HiNCoT constructs the recovered tensor from a latent tensor, which is generated by the deep representation module with a low-rank core tensor as input. Compared with the existing continuous transform-based methods, HiNCoT can more effectively capture the complex nonlinear relationships inherent in multi-dimensional data along the third dimension. To evaluate the effectiveness of the proposed HiNCoT, we suggest an HiNCoT-based multi-dimensional data recovery model. Extensive experiments on diverse degeneration scenarios demonstrate the superiority of our hierarchical nonlinear transform-based method over existing single-layer linear transform-based methods.

PDF Details DOI

JBHI Journal 2026 Journal Article

Morphology Prior Enhanced Teeth Segmentation for High-Resolution Oral Scans

Yuxian Jiang
Xiuying Wang
Tao Yang
Changkai Ji
Lanshan He
Yusheng Liu
Wei Wang
Min Liu

Deep learning methods have been proposed for tooth segmentation on high-resolution intra-oral scans (IOS) that plays a crucial role in clinical dental practice. However, they generally segment teeth in a low-resolution data with a fixed receptive field and generate final segmentation by up-sampling interpolation, and neglect teeth’s morphology priors: their similar dental arch structures and significantly different curvatures in different parts of each tooth. They thus lack adaptability to different parts of each tooth, and show less accurate segmentation of boundary points between teeth and gums due to the up-sampling computation. Further, cluttered poses of IOS limit their generalization and usability of teeth location and geometric information. To address these limitations, a morphology prior enhanced teeth segmentation framework is proposed in this paper. Firstly, a robust preprocessing is introduced to align poses of different IOS by computing their dental arch orientations, thereby improving segmentation generalization and usability of IOS geometric information. Secondly, a decomposition-merging strategy is designed to avoid the up-sampling limitation, which decomposes an IOS into multiple low-resolution data and merges their segmentation outcomes into a high-resolution result. Thirdly, an innovative module integrating semantic and geometric features is proposed to adaptively select deformable receptive fields. It geometrically samples within a variable probability space to construct receptive fields with varied graph relationships for different points, facilitating adaptive segmentation of different parts of each tooth. Experimental results on 6238 IOS from four centers demonstrate that our method significantly outperforms 11 state-of-the-art methods, achieving a 6. 93% enhancement for cross-center testing.

Details DOI

AAAI Conference 2026 Conference Paper

Proxy Zero-Shot Hashing with Multimodal Fusion via Stable Diffusion

Hui Zhang
Weikang Gao
Tao Yang
Yuan Cao

With the rapid growth of visual content in open-world environments, zero-shot hashing image retrieval (ZSHIR) has emerged to tackle the challenge of recognizing novel classes using attribute-level and semantic information. However, existing methods often rely on shallow fusion of multi-source cues (e.g., attributes, labels, and visual features) through external supervision or feature concatenation, failing to capture the underlying semantic structure in a generative way. Particularly, current bridging strategies between modalities suffer from information fragmentation and weak alignment, hindering the model's ability to fully understand complex attribute-visual relations. Moreover, subtle semantic gaps or “semantic drift” between seen and unseen classes further degrade inter-class separability and the scalability of hashing models. To address these issues, we propose a novel framework called Proxy Zero-Shot Hashing with Multimodal Fusion via Stable Diffusion (PZSH), which integrates generative modeling and contrastive learning. PZSH leverages a pre-trained Stable Diffusion (SD) model to synthesize multimodal content, and uses dual BLIP encoders to enhance semantic alignment across modalities. We further design a proxy hashing loss to enforce discriminative binary representations. Extensive experiments on benchmark datasets show that PZSH achieves state-of-the-art performance with stronger generalization to unseen classes.

PDF Details DOI

JBHI Journal 2025 Journal Article

A Dual Domain Collaborative Network for Polyp Segmentation

Yao Tong
Zuojian Zhou
Kongfa Hu
Tao Yang
Andr´e Kaup
Xin Li

Accurate polyp segmentation in colonoscopy images is essential for early colorectal cancer detection but remains a challenging problem due to the limitations in existing methods for optimizing boundary features and aligning cross-level representations. Specifically, the indistinct polyp boundaries and scale variations across different feature levels pose significant challenges for segmentation accuracy. To address these issues, we propose a dual domain collaborative network (DDCNet) that introduces two novel modules: a frequency context enhancement module (FCEM), which operates in the frequency domain to refine high- and low-frequency features, and a cross-level shift recalibrated fusion module (CSFM), which improves multi scale feature alignment in the spatial domain. The FCEM improves boundary precision by adaptively refining high frequency boundary features and enhancing low-frequency contextual information, while the CSFM mitigates cross level feature misalignment by dynamically recalibrating multi-scale features throughout the encoder-decoder architecture. Additionally, we design a hybrid loss function that integrates boundary, cross-entropy, and frequency consistency losses to further boost segmentation performance. Experimental results on three benchmark datasets (Kvasir SEG, CVC-ClinicDB, and CVC-ColonDB) demonstrate that DDCNet achieves state-of-the-art performance, with Dice coefficients of 0. 9343, 0. 9447, and 0. 8155, respectively. These results represent improvements of 1. 0%–1. 5% over the best existing methods. Ablation studies further validate the individual contributions of FCEM, CSFM, and the hybrid loss function. Additionally, we compared the proposed loss function with three commonly used functions.

Details DOI

NeurIPS Conference 2025 Conference Paper

AF-UMC: An Alignment-Free Fusion Framework for Unaligned Multi-View Clustering

Bohang Sun
Yuena Lin
Tao Yang
Zhen Zhu
Zhen Yang
Gengyu Lyu

The Unaligned Multi-view Clustering (UMC) aims to learn a discriminative cluster structure from unaligned multi-view data, where the features of samples are not completely aligned across multiple views. Most existing methods usually prioritize employing various alignment strategies to align sample representations across views and then conduct cross-view fusion on aligned representations for subsequent clustering. However, due to the heterogeneity of representations across different views, these alignment strategies often fail to achieve ideal view-alignment results, inevitably leading to unreliable alignment-based fusion. To address this issue, we propose an alignment-free consistency fusion framework named AF-UMC, which bypasses the traditional view-alignment operation and directly extracts consistent representations from each view to perform global cross-view consistency fusion. Specifically, we first construct a cross-view consistent basis space by a cross-view reconstruction loss and a designed Structural Clarity Regularization (SCR), where autoencoders extract consistent representations from each view through projecting view-specific data to the constructed basis space. Afterwards, these extracted representations are globally pulled together for further cross-view fusion according to a designed Instance Global Contrastive Fusion (IGCF). Compared with previous methods, AF-UMC directly extracts consistent representations from each view for global fusion instead of alignment for fusion, which significantly mitigates the degraded fusion performance caused by undesired view-alignment results while greatly reducing algorithm complexity and enhancing its efficiency. Extensive experiments on various datasets demonstrate that our AF-UMC exhibits superior performance against other state-of-the-art methods.

PDF Details

NeurIPS Conference 2025 Conference Paper

CaliGCL: Calibrated Graph Contrastive Learning via Partitioned Similarity and Consistency Discrimination

Yuena Lin
Hao Wei
Hai-Chun Cai
Bohang Sun
Tao Yang
Zhen Yang
Gengyu Lyu

Graph contrastive learning (GCL) aims to learn self-supervised representations by distinguishing positive and negative sample pairs generated from multiple augmented graph views. Despite showing promising performance, GCL still suffers from two critical biases: (1) Similarity estimation bias arises when feature elements that support positive pair alignment are suppressed by conflicting components within the representation, causing truly positive pairs to appear less similar. (2) Semantic shift bias occurs when random augmentations alter the underlying semantics of samples, leading to incorrect positive or negative assignments and injecting noise into training. To address these issues, we propose CaliGCL, a GCL model for calibrating the biases by integrating an exponential partitioned similarity measure and a semantics-consistency discriminator. The exponential partitioned similarity computes the similarities among fine-grained partitions obtained through splitting representation vectors and uses exponential scaling to emphasize aligned (positive) partitions while reducing the influence of misaligned (negative) ones. The discriminator dynamically identifies whether augmented sample pairs maintain semantic consistency, enabling correction of misleading contrastive supervision signals. These components jointly reduce biases in similarity estimation and sample pairing, guiding the encoder to learn more robust and semantically meaningful representations. Extensive experiments on multiple benchmarks show that CaliGCL effectively mitigates both types of biases and achieves state-of-the-art performance.

PDF Details

JBHI Journal 2025 Journal Article

DMSACNN: Deep Multiscale Attentional Convolutional Neural Network for EEG-Based Motor Decoding

Ke Liu
Xin Xing
Tao Yang
Zhuliang Yu
Bin Xiao
Guoyin Wang
Wei Wu

Objective: Accurate decoding of electroencephalogram (EEG) signals has become more significant for the brain-computer interface (BCI). Specifically, motor imagery and motor execution (MI/ME) tasks enable the control of external devices by decoding EEG signals during imagined or real movements. However, accurately decoding MI/ME signals remains a challenge due to the limited utilization of temporal information and ineffective feature selection methods. Methods: This paper introduces DMSACNN, an end-to-end deep multiscale attention convolutional neural network for MI/ME-EEG decoding. DMSACNN incorporates a deep multiscale temporal feature extraction module to capture temporal features at various levels. These features are then processed by a spatial convolutional module to extract spatial features. Finally, a local and global feature fusion attention module is utilized to combine local and global information and extract the most discriminative spatiotemporal features. Main results: DMSACNN achieves impressive accuracies of 78. 20%, 96. 34% and 70. 90% for hold-out analysis on the BCI-IV-2a, High Gamma and OpenBMI datasets, respectively, outperforming most of the state-of-the-art methods. Conclusion and significance: These results highlight the potential of DMSACNN in robust BCI applications. Our proposed method provides a valuable solution to improve the accuracy of the MI/ME-EEG decoding, which can pave the way for more efficient and reliable BCI systems.

Details DOI

ICRA Conference 2025 Conference Paper

Enabling In-Flight Metamorphosis in Multirotors with a Center-Driven Scissor Extendable Airframe for Adaptive Navigation

Tao Yang
Peng Li 0019
Gang Wang 0024
Yantao Shen 0001

To address complex mission tasks, multirotors benefit from in-flight reconfiguration that enhances their morphological adaptability. This paper presents the Center-Driven Scissor Extendable Airframe (CDSEA), a novel one-degree-of-freedom (DOF) morphing airframe designed to replace traditional fixed-size airframes. The CDSEA allows a quadrotor to achieve significant morphological changes during flight, with rotors deploying radially from a central point. This capability facilitates substantial variations in footprint radius and ensures smooth transitions. The paper details the mechanical design, as well as kinematic and dynamic analyses, and discusses the actuator selection strategy for the CDSEA. Experimental results with a prototype demonstrate that the CDSEA achieves a footprint-radius deformation ratio of 2. 5 and a morphing time of 0. 3 seconds, surpassing existing solutions. Additionally, the design improves obstacle avoidance and wind resistance. These results underscore the CDSEA's potential as an advanced solution for enhancing UAV adaptive navigation performance in complex environments.

Details

NeurIPS Conference 2025 Conference Paper

MPS-Prover: Advancing Stepwise Theorem Proving by Multi-Perspective Search and Data Curation

Zhenwen Liang
Linfeng Song
Yang Li
Tao Yang
Haitao Mi
Dong Yu

Automated Theorem Proving (ATP) in formal languages remains a formidable challenge in AI, demanding rigorous logical deduction and navigating vast search spaces. While large language models (LLMs) have shown promising performance, existing stepwise provers often suffer from biased search guidance, leading to inefficiencies and suboptimal proof strategies. This paper introduces the Multi-Perspective Search Prover (MPS-Prover), a novel stepwise ATP system designed to overcome these limitations. MPS-Prover incorporates two key innovations: a highly effective post-training data curation strategy that prunes approximately 40\% of redundant training data without sacrificing performance, and a multi-perspective tree search mechanism. This search integrates a learned critic model with strategically designed heuristic rules to diversify tactic selection, prevent getting trapped in unproductive states, and enhance search robustness. Extensive evaluations demonstrate that MPS-Prover achieves state-of-the-art performance on multiple challenging benchmarks, including miniF2F and ProofNet, outperforming prior 7B parameter models. Furthermore, our analyses reveal that MPS-Prover generates significantly shorter and more diverse proofs compared to existing stepwise and whole-proof methods, highlighting its efficiency and efficacy. Our work advances the capabilities of LLM-based formal reasoning and offers a robust framework and a comprehensive analysis for developing more powerful theorem provers.

PDF Details

IROS Conference 2025 Conference Paper

Multi-target Association and Localization with Distributed Drone Following: A Factor Graph Approach

Kaixiao Ye
Weiyu Shao
Yuhang Zheng
Bohui Fang
Tao Yang

Vision-based multi-drone multi-object tracking technology enables autonomous target situational awareness for unmanned aerial systems. Distributed observer drones dynamically estimate the spatio-temporal states of multiple targets through collaborative sensor fusion, enabling simultaneous localization and persistent following of the target of interest in cluttered airspaces. The challenge lies in distinguishing targets in different drones’ views and keeping the target of interest within the field of view. This paper proposes a factor graph method for joint multi-target association and localization with distributed drone following. Sensor measurements and control constraints are integrated into a probabilistic factor graph to solve the bundle adjustment and model predictive control, respectively. Both simulation and real-world experiments prove the effectiveness and robustness of our proposed approach. The source code will be available at: https://github.com/npu-ius-lab/MLMF.

Details

NeurIPS Conference 2025 Conference Paper

PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding

Kangcong Li
Peng Ye
Chongjun Tu
Lin Zhang
Chunfeng Song
Jiamin Wu
Tao Yang
Qihao Zheng

While Large Language Models (LLMs) demonstrate strong performance across domains, their long-context capabilities are limited by transient neural activations causing information decay and unstructured feed-forward network (FFN) weights leading to semantic fragmentation. Inspired by the brain’s working memory and cortical modularity, we propose PaceLLM, featuring two innovations: (1) a Persistent Activity (PA) Mechanism that mimics prefrontal cortex (PFC) neurons’ persistent firing by introducing an activation-level memory bank to dynamically retrieve, reuse, and update critical FFN states, addressing contextual decay; and (2) Cortical Expert (CE) Clustering that emulates task-adaptive neural specialization to reorganize FFN weights into semantic modules, establishing cross-token dependencies and mitigating fragmentation. Extensive evaluations show that PaceLLM achieves 6% improvement on LongBench’s Multi-document QA and 12. 5–17. 5% performance gains on $\infty$-Bench tasks, while extending measurable context length to 200K tokens in Needle-In-A-Haystack (NIAH) tests. This work pioneers brain-inspired LLM optimization and is complementary to other works. Besides, it can be generalized to any model and enhance their long-context performance and interpretability without structural overhauls.

PDF Details

JBHI Journal 2025 Journal Article

PhenoAlign: A Hybrid Data-Knowledge-Driven Approach for Precisely Aligning Phenotype Information in Medical Texts

Tao Yang
Luming Chen
Lizong Deng
Taijiao Jiang

Precisely aligning phenotypic information within medical texts is paramount in advancing intelligent medical applications, such as similar patient case retrieval. However, despite its criticality, an algorithm specifically designed for this task is lacking. We previously introduced a fine-grained semantic information model, the semantic structured unit of phenotypes (PhenoSSU), and an automatic extraction algorithm. This model accurately characterizes and extracts phenotypic information from medical texts. In this study, we explore different PhenoSSU alignment strategies. The results show that the data-knowledge-driven approach best aligns PhenoSSUs. Specifically, employing a BERT-based pre-trained language model (PLMs) to align phrase-type PhenoSSUs and a knowledge-based method for logic-type PhenoSSUs demonstrates efficacy. Moreover, by successfully integrating the PhenoSSU alignment and extraction algorithms, we have developed PhenoAlign, a novel medical text phenotype alignment tool. This tool facilitates precisely aligning phenotypic information by processing two medical texts, generating accurate alignment outcomes. PhenoAlign exhibited satisfactory medical text phenotypic alignment using the expert annotated gold standard test set, with an end-to-end F1 score of 0. 820. The F1 scores for phenoSSU extraction and alignment were 0. 885 and 0. 927, respectively. Our analysis extends to the potential application of ChatGPT in phenotype alignment tasks, and find significant challenges encountered by large language models in this domain. We developed a simple, effective tool for medical text phenotype information alignment. This tool will be valuable to intelligent medical applications, facilitating patient care and medical research advancements.

Details DOI

EAAI Journal 2024 Journal Article

A time series and deep fusion framework for rotating machinery fault diagnosis

Jiasheng Zhang
Di Hu
Tao Yang
Hongkuan Zhou
Xianling Li

For shafting rotation equipment, in the fault diagnosis based on vibration analysis, the sampling signal is a waveform of a short moment, which is quasi-static information at the corresponding moment. A single quasi-static piece of information does not necessarily contain obvious fault information, however, in a longer period much larger than the sampling period, the time series information composed of multiple quasi-static combinations may well-characterize the fault. Traditional deep learning algorithms often focus on fault features in quasi-static information, ignoring time series features, which results in low diagnostic accuracy. To solve the above problems, this paper firstly presents a fault diagnosis framework of time series and deep fusion network (TDFN). Then, a method to extract time series information and quasi-static information based on expert experience knowledge is proposed, and finally studies the method of using this framework to train the network and fuse the two types of features. Different from existing deep learning methods, TDFN includes time series blocks, depth blocks and fusion blocks. This paper uses laboratory data and industrial rotor data for verification, the results show that the overall accuracy of TDFN reached 93. 8% and it does not experience a drop in accuracy after fusion. The diagnosis accuracy is higher than traditional networks, providing guiding suggestions for the management and operation of on-site industrial equipment.

Details DOI

NeurIPS Conference 2024 Conference Paper

Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement

Tao Yang
Cuiling Lan
Yan Lu
Nanning Zheng

Disentangled representation learning strives to extract the intrinsic factors within the observed data. Factoring these representations in an unsupervised manner is notably challenging and usually requires tailored loss functions or specific structural designs. In this paper, we introduce a new perspective and framework, demonstrating that diffusion models with cross-attention itself can serve as a powerful inductive bias to facilitate the learning of disentangled representations. We propose to encode an image into a set of concept tokens and treat them as the condition of the latent diffusion model for image reconstruction, where cross attention over the concept tokens is used to bridge the encoder and the U-Net of the diffusion model. We analyze that the diffusion process inherently possesses the time-varying information bottlenecks. Such information bottlenecks and cross attention act as strong inductive biases for promoting disentanglement. Without any regularization term in the loss function, this framework achieves superior disentanglement performance on the benchmark datasets, surpassing all previous methods with intricate designs. We have conducted comprehensive ablation studies and visualization analyses, shedding a light on the functioning of this model. We anticipate that our findings will inspire more investigation on exploring diffusion model for disentangled representation learning towards more sophisticated data analysis and understanding.

PDF Details DOI

JBHI Journal 2024 Journal Article

MSVTNet: Multi-Scale Vision Transformer Neural Network for EEG-Based Motor Imagery Decoding

Ke Liu
Tao Yang
Zhuliang Yu
Weibo Yi
Hong Yu
Guoyin Wang
Wei Wu

Object: Transformer-based neural networks have been applied to the electroencephalography (EEG) decoding for motor imagery (MI). However, most networks focus on applying the self-attention mechanism to extract global temporal information, while the cross-frequency coupling features between different frequencies have been neglected. Additionally, effectively integrating different neural networks poses challenges for the advanced design of decoding algorithms. Methods: This study proposes a novel end-to-end Multi-Scale Vision Transformer Neural Network (MSVTNet) for MI-EEG classification. MSVTNet first extracts local spatio-temporal features at different filtered scales through convolutional neural networks (CNNs). Then, these features are concatenated along the feature dimension to form local multi-scale spatio-temporal feature tokens. Finally, Transformers are utilized to capture cross-scale interaction information and global temporal correlations, providing more distinguishable feature embeddings for classification. Moreover, auxiliary branch loss is leveraged for intermediate supervision to ensure the effective integration of CNNs and Transformers. Results: The performance of MSVTNet was assessed through subject-dependent (session-dependent and session-independent) and subject-independent experiments on three MI datasets, i. e. , the BCI competition IV 2a, 2b and OpenBMI datasets. The experimental results demonstrate that MSVTNet achieves state-of-the-art performance in all analyses. Conclusion: MSVTNet shows superiority and robustness in enhancing MI decoding performance.

Details DOI

EAAI Journal 2024 Journal Article

Unmanned Aerial Vehicles anomaly detection model based on sensor information fusion and hybrid multimodal neural network

Hongli Deng
Yu Lu
Tao Yang
Ziyu Liu
JiangChuan Chen

The use of Unmanned Aerial Vehicle (UAV) in various industries is increasing, which places higher requirements on the reliability of UAV. One of the ways to ensure the safety of UAV flights is by detecting anomalies in their flight. However, traditional UAV anomaly detection models have some shortcomings. First, they fail to integrate data from multiple sensors across time and frequency domains, hampering the anomaly detection model's ability to accurately assess the UAV's status. Second, they apply the same prediction error loss to all classes, which result in excessive false positives in some key classes. Finally, most of them used unimodal classification models to process data from multiple heterogeneous sensors, which makes it difficult for the models to extract targeted features. This paper proposes a UAV anomaly detection model based on sensor information fusion and hybrid multimodal neural network (IF-HMNN). Firstly, facilitated by the newly devised Multi-source Heterogeneous UAV Sensor Information Alignment algorithm (MHSIA), IF-HMNN can realize information fusion from multiple sensors. Secondly, a classes weight assignment mechanism is designed to increase the IF-HMNN's focus on key classes. Finally, the neural networks of two modalities are trained separately according to different time-frequency domain features, and their classification outcomes are amalgamated through a hybrid soft voting mechanism. Experimental results show that IF-HMNN achieves accuracy of 0. 99, 0. 9991, and 0. 9967 on three datasets respectively. The accuracy of IF-HMNN model on the test set is about 2 %–3 % higher than similar models. We will publish our code as well as the dataset here: https: //github. com/FishLuYu/IF-HMNN.

Details DOI

ICML Conference 2023 Conference Paper

A Flexible Diffusion Model

Weitao Du
He Zhang
Tao Yang
Yuanqi Du

Denoising diffusion (score-based) generative models have become a popular choice for modeling complex data. Recently, a deep connection between forward-backward stochastic differential equations (SDEs) and diffusion-based models has been established, leading to the development of new SDE variants such as sub-VP and critically-damped Langevin. Despite the empirical success of some hand-crafted forward SDEs, many potentially promising forward SDEs remain unexplored. In this work, we propose a general framework for parameterizing diffusion models, particularly the spatial part of forward SDEs, by leveraging the symplectic and Riemannian geometry of the data manifold. We introduce a systematic formalism with theoretical guarantees and connect it with previous diffusion models. Finally, we demonstrate the theoretical advantages of our method from a variational optimization perspective. We present numerical experiments on synthetic datasets, MNIST and CIFAR10 to validate the effectiveness of our framework.

Details

EAAI Journal 2023 Journal Article

A geometry-aware deep network for depth estimation in monocular endoscopy

Yongming Yang
Shuwei Shao
Tao Yang
Peng Wang
Zhuo Yang
Chengdong Wu
Hao Liu

Monocular depth estimation is critical for endoscopists to perform spatial perception and 3D navigation of surgical sites. However, most of the existing methods ignore the important geometric structural consistency, which inevitably leads to performance degradation and distortion of 3D reconstruction. To address this issue, we introduce a gradient loss to penalize edge fluctuations ambiguous around stepped edge structures and a normal loss to explicitly express the sensitivity to frequently small structures, and propose a geometric consistency loss to spreads the spatial information across the sample grids to constrain the global geometric anatomy structures. In addition, we develop a synthetic RGB-Depth dataset that captures the anatomical structures under reflections and illumination variations. The proposed method is extensively validated across different datasets and clinical images and achieves mean RMSE values of 0. 066 (stomach), 0. 029 (small intestine), and 0. 139 (colon) on the EndoSLAM dataset. The generalizability of the proposed method achieves mean RMSE values of 12. 604 (T1-L1), 9. 930 (T2-L2), and 13. 893 (T3-L3) on the ColonDepth dataset. The experimental results show that our method exceeds previous state-of-the-art competitors and generates more consistent depth maps and reasonable anatomical structures. The quality of intraoperative 3D structure perception from endoscopic videos of the proposed method meets the accuracy requirements of video-CT registration algorithms for endoscopic navigation. The dataset and the source code will be available at https: //github. com/YYM-SIA/LINGMI-MR.

Details DOI

NeurIPS Conference 2023 Conference Paper

DisDiff: Unsupervised Disentanglement of Diffusion Probabilistic Models

Tao Yang
Yuwang Wang
Yan Lu
Nanning Zheng

Targeting to understand the underlying explainable factors behind observations and modeling the conditional generation process on these factors, we connect disentangled representation learning to diffusion probabilistic models (DPMs) to take advantage of the remarkable modeling ability of DPMs. We propose a new task, disentanglement of (DPMs): given a pre-trained DPM, without any annotations of the factors, the task is to automatically discover the inherent factors behind the observations and disentangle the gradient fields of DPM into sub-gradient fields, each conditioned on the representation of each discovered factor. With disentangled DPMs, those inherent factors can be automatically discovered, explicitly represented and clearly injected into the diffusion process via the sub-gradient fields. To tackle this task, we devise an unsupervised approach, named DisDiff, and for the first time achieving disentangled representation learning in the framework of DPMs. Extensive experiments on synthetic and real-world datasets demonstrate the effectiveness of DisDiff.

PDF Details

AAAI Conference 2023 Conference Paper

Orders Are Unwanted: Dynamic Deep Graph Convolutional Network for Personality Detection

Tao Yang
Jinghao Deng
Xiaojun Quan
Qifan Wang

Predicting personality traits based on online posts has emerged as an important task in many fields such as social network analysis. One of the challenges of this task is assembling information from various posts into an overall profile for each user. While many previous solutions simply concatenate the posts into a long text and then encode the text by sequential or hierarchical models, they introduce unwarranted orders for the posts, which may mislead the models. In this paper, we propose a dynamic deep graph convolutional network (D-DGCN) to overcome the above limitation. Specifically, we design a learn-to-connect approach that adopts a dynamic multi-hop structure instead of a deterministic structure, and combine it with the DGCN module to automatically learn the connections between posts. The modules of post encoder, learn-to-connect, and DGCN are jointly trained in an end-to-end manner. Experimental results on the Kaggle and Pandora datasets show the superior performance of D-DGCN to state-of-the-art baselines. Our code is available at https://github.com/djz233/D-DGCN.

PDF Details DOI

ICLR Conference 2022 Conference Paper

Actor-Critic Policy Optimization in a Large-Scale Imperfect-Information Game

Haobo Fu
Weiming Liu 0004
Shuang Wu
Yijia Wang
Tao Yang
Kai Li 0022
Junliang Xing
Bin Li 0025

The deep policy gradient method has demonstrated promising results in many large-scale games, where the agent learns purely from its own experience. Yet, policy gradient methods with self-play suffer convergence problems to a Nash Equilibrium (NE) in multi-agent situations. Counterfactual regret minimization (CFR) has a convergence guarantee to a NE in 2-player zero-sum games, but it usually needs domain-specific abstractions to deal with large-scale games. Inheriting merits from both methods, in this paper we extend the actor-critic algorithm framework in deep reinforcement learning to tackle a large-scale 2-player zero-sum imperfect-information game, 1-on-1 Mahjong, whose information set size and game length are much larger than poker. The proposed algorithm, named Actor-Critic Hedge (ACH), modifies the policy optimization objective from originally maximizing the discounted returns to minimizing a type of weighted cumulative counterfactual regret. This modification is achieved by approximating the regret via a deep neural network and minimizing the regret via generating self-play policies using Hedge. ACH is theoretically justified as it is derived from a neural-based weighted CFR, for which we prove the convergence to a NE under certain conditions. Experimental results on the proposed 1-on-1 Mahjong benchmark and benchmarks from the literature demonstrate that ACH outperforms related state-of-the-art methods. Also, the agent obtained by ACH defeats a human champion in 1-on-1 Mahjong.

Details

NeurIPS Conference 2022 Conference Paper

AD-DROP: Attribution-Driven Dropout for Robust Language Model Fine-Tuning

Tao Yang
Jinghao Deng
Xiaojun Quan
Qifan Wang
Shaoliang Nie

Fine-tuning large pre-trained language models on downstream tasks is apt to suffer from overfitting when limited training data is available. While dropout proves to be an effective antidote by randomly dropping a proportion of units, existing research has not examined its effect on the self-attention mechanism. In this paper, we investigate this problem through self-attention attribution and find that dropping attention positions with low attribution scores can accelerate training and increase the risk of overfitting. Motivated by this observation, we propose Attribution-Driven Dropout (AD-DROP), which randomly discards some high-attribution positions to encourage the model to make predictions by relying more on low-attribution positions to reduce overfitting. We also develop a cross-tuning strategy to alternate fine-tuning and AD-DROP to avoid dropping high-attribution positions excessively. Extensive experiments on various benchmarks show that AD-DROP yields consistent improvements over baselines. Analysis further confirms that AD-DROP serves as a strategic regularizer to prevent overfitting during fine-tuning.

PDF Details

EAAI Journal 2022 Journal Article

Multi-camera joint spatial self-organization for intelligent interconnection surveillance

Congcong Li
Jing Li
Yuguang Xie
Jiayang Nie
Tao Yang
Zhaoyang Lu

The construction of smart city makes information interconnection play an increasingly important role in intelligent surveillance systems. Especially the interconnection among massive cameras is the key to realizing the evolution from current fragmented monitoring to interconnection surveillance. However, it remains a challenging problem in practical systems due to large sensor quantity, various camera types, and complex spatial layout. Aimed at this problem, this paper proposes a novel multi-camera joint spatial self-organization approach, which realizes interconnection surveillance by unifying cameras into one imaging space. Differing from existing back-end data association strategy, our method takes front-end data calibration as a breakthrough to relate surveillance data. Specifically, this paper first initials camera spatial parameter by sequence complementary feature integration. Through integrating complementarity and redundancy among sequence features, our method has robustness under scene dynamic changes and noise. Then, we propose a multi-camera joint optimization method based on common monitoring coverage correlation analysis to estimate a more accurate relative relationship. By leveraging the two strategies, the spatial relationship and visual data association across monitoring cameras are returned finally. Our system organizes all cameras into a unified imaging space by itself. Extensive experimental evaluations on an actual campus environment demonstrate our method achieves remarkable performance.

Details DOI

JBHI Journal 2022 Journal Article

PIAT: An Evolutionarily Intelligent System for Deep Phenotyping of Chinese Electronic Health Records

Lizong Deng
Xu Zhang
Tao Yang
Mi Liu
Luming Chen
Taijiao Jiang

Electronic health record (EHR) resources are valuable but remain underexplored because most clinical information, especially phenotype information, is buried in the free text of EHRs. An intelligent annotation tool plays an important role in unlocking the full potential of EHRs by transforming free-text phenotype information into a computer-readable form. Deep phenotyping has shown its advantage in representing phenotype information in EHRs with high fidelity; however, most existing annotation tools are not suitable for the deep phenotyping task. Here, we developed an intelligent annotation tool named PIAT with a major focus on the deep phenotyping of Chinese EHRs. PIAT can improve the annotation efficiency for EHR-based deep phenotyping with a simple but effective interactive interface, automatic preannotation support, and a learning mechanism. Specifically, experts can proofread automatic annotation results from the annotation algorithm in the web-based interactive interface, and EHRs reviewed by experts can be used for evolving the underlying annotation algorithm. In this way, the annotation process of deep phenotyping EHRs will become easier. In conclusion, we create a powerful intelligent system for the deep phenotyping of Chinese EHRs. It is hoped that our work will inspire further studies in constructing intelligent systems for deep phenotyping English and non-English EHRs.

Details DOI

NeurIPS Conference 2022 Conference Paper

Visual Concepts Tokenization

Tao Yang
Yuwang Wang
Yan Lu
Nanning Zheng

Obtaining the human-like perception ability of abstracting visual concepts from concrete pixels has always been a fundamental and important target in machine learning research fields such as disentangled representation learning and scene decomposition. Towards this goal, we propose an unsupervised transformer-based Visual Concepts Tokenization framework, dubbed VCT, to perceive an image into a set of disentangled visual concept tokens, with each concept token responding to one type of independent visual concept. Particularly, to obtain these concept tokens, we only use cross-attention to extract visual information from the image tokens layer by layer without self-attention between concept tokens, preventing information leakage across concept tokens. We further propose a Concept Disentangling Loss to facilitate that different concept tokens represent independent visual concepts. The cross-attention and disentangling loss play the role of induction and mutual exclusion for the concept tokens, respectively. Extensive experiments on several popular datasets verify the effectiveness of VCT on the tasks of disentangled representation learning and scene decomposition. VCT achieves the state of the art results by a large margin.

PDF Details

EAAI Journal 2021 Journal Article

A novel Domain Adaptive Deep Recurrent Network for multivariate time series prediction

Tao Yang
Xia Yu
Ning Ma
Yuhang Zhao
Hongru Li

Multivariate time series prediction has attracted growing interest in many research fields. Recently, deep learning has been applied to multivariate time series prediction and has achieved encouraging results. However, in real-world scenarios, the insufficient data of multivariate time series at the beginning of the observation causes the deep learning model unable to exert its expected performance. Furthermore, there is the distribution discrepancy between different multivariate time series caused by many factors, making it unfeasible to reuse existing data or models directly. Therefore, a novel Domain Adaptive Deep Recurrent Network (DADRN) is proposed for multivariate time series prediction with insufficient data, which transferring the knowledge of the target-related time series (source domain) to the target time series (target domain) by minimizing distribution mismatch in the feature sharing space. The DADRN automatically learns the temporal dependence of predictive time series and the dynamic dependencies between multiple time variables through the deep recurrent neural network. Besides, a special transfer learning method, domain adaptation, is embedded in the constructed deep recurrent network to reduce the distribution discrepancy between different domains. The proposed domain independence strategy and domain weighted loss further enhance the DADRN’s transfer learning capability by improving the distribution estimation of the target domain and balancing the network’s learning on two domains. The reasonable combination of deep recurrent network and domain adaptation endows DADRN with favorable transfer learning capability, and its effectiveness is demonstrated by the experimental results on two real-world datasets.

Details DOI

EAAI Journal 2021 Journal Article

Fast just-in-time-learning recursive multi-output LSSVR for quality prediction and control of multivariable dynamic systems

Ping Zhou
Weiqi Chen
Chengming Yi
Zhaohui Jiang
Tao Yang
Tianyou Chai

Aiming at quality prediction and control of blast furnace (BF) ironmaking process characterized by complicated nonlinear time-varying dynamics, this paper proposes a just-in-time-learning (JITL) recursive multi-output least squares support vector regression (JITL-R-M-LSSVR) algorithm with fast nonlinear local learning capability for multivariable dynamic systems. The proposed fast JITL-R-M-LSSVR effectively combines the online local learning of JITL with the multi-output LSSVR (M-LSSVR) based on multi-task transfer learning, and focuses on how to ensure the rapid verification of the local model during online learning of M-LSSVR, and how to perform model pruning while recursively updating the model parameters to improve the calculation efficiency. To this end, the proposed algorithm uses a derived multi-output incremental learning algorithm to recursively update model parameters online in a gentle way, which has better modeling stability and smoothness than the traditional way that discards old models. At the same time, when the model is pruned, a novel multi-output reverse decremental learning algorithm is proposed to adaptively delete the modeling data, so as to effectively control the sample size and reduces the calculation cost. In particular, the model verification of the proposed algorithm only needs to construct the M-LSSVR modeling matrix and the matrix inverse operation once, and the matrix after deleting each modeling sample can be easily obtained by reverse decremental learning of the original modeling matrix, which can achieve fast and efficient model verification. Finally, the effectiveness and practicability of the proposed method are verified by applying it to prediction modeling and predictive control of the molten iron quality in BF ironmaking process.

Details DOI

AAAI Conference 2020 Conference Paper

On the Generation of Medical Question-Answer Pairs

Sheng Shen
Yaliang Li
Nan Du
Xian Wu
Yusheng Xie
Shen Ge
Tao Yang
Kai Wang

Question answering (QA) has achieved promising progress recently. However, answering a question in real-world scenarios like the medical domain is still challenging, due to the requirement of external knowledge and the insufﬁcient quantity of high-quality training data. In the light of these challenges, we study the task of generating medical QA pairs in this paper. With the insight that each medical question can be considered as a sample from the latent distribution of questions given answers, we propose an automated medical QA pair generation framework, consisting of an unsupervised key phrase detector that explores unstructured material for validity, and a generator that involves a multi-pass decoder to integrate structural knowledge for diversity. A series of experiments have been conducted on a real-world dataset collected from the National Medical Licensing Examination of China. Both automatic evaluation and human annotation demonstrate the effectiveness of the proposed method. Further investigation shows that, by incorporating the generated QA pairs for training, signiﬁcant improvement in terms of accuracy can be achieved for the examination QA system. 1

PDF Details

ICRA Conference 2020 Conference Paper

SNIAE-SSE Deformation Mechanism Enabled Scalable Multicopter: Design, Modeling and Flight Performance Validation

Tao Yang
Yujing Zhang
Peng Li 0019
Yantao Shen 0001
Yun-Hui Liu 0001
Haoyao Chen

This paper focuses on designing, modeling and validating a novel scalable multicopter whose deformation mechanism, called SNIAE-SSE, relies on a combination of simple non-intersecting angulated elements (SNIAEs) and straight scissor-like elements (SSEs). The proposed SNIAE-SSE mechanism has the advantages of single degree-of-freedom, fast actuation capability and large deformation ratio. In this work, enabled by the SNIAE-SSE mechanism, a quadcopter prototype with symmetrical and synchronous deformation is firstly developed, which facilitates a novel and controllably scalable multicopter system for us to analyze its modeling, as well as to validate its flight performance and dynamics during the deformation in several flight missions including hover, throwing, and morphing flying through a narrow window. Experimental results demonstrate that the developed scalable multicopter can maintain its stable flight behavior even both the folding and unfolding body deformations are fast performed, which indicates an excellent capability of the scalable multicopter to rapidly adapt to complex and dynamically changed environments.

Details