Xin Yang

EAAI Journal 2026 Journal Article

A multi-source domain-invariant acoustic feature extraction network for rotating machinery fault diagnosis under unknown cross-working conditions

• Xinyuan Zhang
• Xiangang Cao
• Hongwei Fan
• Xin Yang
• Yong Duan
• Fuyuan Zhao
• Xiangyu Li

In mechanical systems under cross-working conditions, acoustic feature extraction for rotating machinery faces numerous challenges, including difficulty in acquiring high-quality training data from multi-source domains and large divergence between inter-domain samples. Existing methods often suffer from insufficient constraints in the embedding space and limited model generalization. To address these issues, this paper proposes a Multi-source Domain-Invariant Acoustic Feature Extraction Network (DIAFENet) for unknown cross-working condition tasks. DIAFENet introduces a multi-granularity adversarial learning framework that jointly optimizes classification loss, domain adversarial loss, and a novel feature association-based class boundary constraint loss, aiming to learn discriminative and operation-condition-invariant acoustic representations. The core innovation lies in a three-level domain-invariant feature association strategy: (1) Global-level alignment minimizes overall domain divergence; (2) Subdomain-level alignment refines local feature distribution consistency; (3) Maximization of inter-class association distance within subdomains explicitly enlarges decision margins between classes. The framework integrates a spectrogram-based feature extractor with a self-attention pooling mechanism, and employs a Gradient Reversal Layer (GRL) to adversarially eliminate domain-specific information and promote domain-invariant representation learning. The effectiveness of DIAFENet is rigorously evaluated on 15 cross-operating-condition tasks across two public datasets and a Self-built dataset. The results showed that the average classification accuracy of DIAFENet was 98. 93 %, 97. 31 %, and 96. 55 %. The ablation experiment further verified that the proposed feature association constraint strategy improved accuracy by 2. 75 %, demonstrating its key role in enhancing the compactness and separability of the embedding space. This study provides a reliable acoustic feature extraction scheme for intelligent diagnosis of mechanical equipment under multi-source domain cross-working condition tasks.

AAAI Conference 2026 Conference Paper

AlignTrack: Top-Down Spatiotemporal Resolution Alignment for RGB-Event Visual Tracking

• Chuanyu Sun
• Jiqing Zhang
• Yang Wang
• Yuanchen Wang
• Yutong Jiang
• Baocai Yin
• Xin Yang

Most existing RGB-Event trackers rely on strictly aligned datasets, overlooking the asynchronous spatio-temporal resolutions common in real-world scenarios. This methodological limitation impedes effective RGB-Event feature alignment and ultimately degrades tracking performance. To overcome this limitation, we propose AlignTrack, a novel tracking framework built upon a Top-Down Alignment (TDA) strategy inspired by the human visual system. Our TDA framework follows an encode-decode-align paradigm: it first encodes multimodal features to generate target-related priors, which are then progressively decoded to guide a subsequent feature alignment pass. Within this framework, we introduce two key innovations: (1) a Cross-Prior Attention (CPA) module that effectively generates and integrates cross-modal priors, and (2) a Cross-Modal Semantic Alignment (CSA) loss that maximizes mutual information to enforce semantic consistency between modalities. Extensive experiments show that AlignTrack achieves state-of-the-art performance on four challenging RGB-Event tracking benchmarks, demonstrating its robustness in both aligned and unaligned scenarios. Ablation studies further validate the significant contribution of each proposed component.

AAAI Conference 2026 Conference Paper

BAT: Learning Event-based Optical Flow with Bidirectional Adaptive Temporal Correlation

• Gangwei Xu
• Haotong Lin
• Zhaoxing Zhang
• Hongcheng Luo
• Haiyang Sun
• Xin Yang

Event cameras deliver visual information characterized by a high dynamic range and high temporal resolution, offering significant advantages in estimating optical flow for complex lighting conditions and fast-moving objects. Current advanced optical flow methods for event cameras largely adopt established image-based frameworks. However, the spatial sparsity of event data limits their performance. In this paper, we present BAT, an innovative framework that estimates event-based optical flow using bidirectional adaptive temporal correlation. BAT includes three novel designs: 1) a bidirectional temporal correlation that transforms bidirectional temporally dense motion cues into spatially dense ones, enabling accurate and spatially dense optical flow estimation; 2) an adaptive temporal sampling strategy for maintaining temporal consistency in correlation; 3) spatially adaptive temporal motion aggregation to efficiently and adaptively aggregate consistent target motion features into adjacent motion features while suppressing inconsistent ones. Our BAT achieves state-of-the-art performance on the DSEC-Flow benchmark, outperforming existing methods by a large margin while also exhibiting sharp edges and high-quality details. Our BAT can accurately predict future optical flow using only past events, significantly outperforming E-RAFT’s warm-start approach.

AAAI Conference 2026 Conference Paper

Dynamic Weight Adaptation in Spiking Neural Networks Inspired by Biological Homeostasis

• Yunduo Zhou
• Bo Dong
• Chang Li
• Yuanchen Wang
• Xuefeng Yin
• Yang Wang
• Xin Yang

Homeostatic mechanisms play a crucial role in maintaining optimal functionality within the neural circuits of the brain. By regulating physiological and biochemical processes, these mechanisms ensure the stability of an organism’s internal environment, enabling it to better adapt to external changes. Among these mechanisms, the Bienenstock, Cooper, and Munro (BCM) theory has been extensively studied as a key principle for maintaining the balance of synaptic strengths in biological systems. Despite the extensive development of spiking neural networks (SNNs) as a model for bionic neural networks, no prior work in the machine learning community has integrated biologically plausible BCM formulations into SNNs to provide homeostasis. In this study, we propose a Dynamic Weight Adaptation Mechanism (DWAM) for SNNs, inspired by the BCM theory. DWAM can be integrated into the host SNN, dynamically adjusting network weights in real time to regulate neuronal activity, providing homeostasis to the host SNN without any fine-tuning. We validated our method through dynamic obstacle avoidance and continuous control tasks under both normal and specifically designed degraded conditions. Experimental results demonstrate that DWAM not only enhances the performance of SNNs without existing homeostatic mechanisms under various degraded conditions but also further improves the performance of SNNs that already incorporate homeostatic mechanisms.

AAAI Conference 2026 Conference Paper

Enhanced Federated Deep Multi-View Clustering Under Uncertainty Scenario

• Bingjun Wei
• Xuemei Cao
• Jiafen Liu
• Haoyang Liang
• Xin Yang

Traditional Federated Multi-View Clustering assumes uniform views across clients, yet practical deployments reveal heterogeneous view completeness with prevalent incomplete, redundant, or corrupted data. While recent approaches model view heterogeneity, they neglect semantic conflicts from dynamic view combinations, failing to address dual uncertainties: view uncertainty (semantic inconsistency from arbitrary view pairings) and aggregation uncertainty (divergent client updates with imbalanced contributions). To address these, we propose a novel Enhanced Federated Deep Multi-View Clustering framework: first align local semantics, hierarchical contrastive fusion within clients resolves view uncertainty by eliminating semantic conflicts; a view adaptive drift module mitigates aggregation uncertainty through global-local prototype contrast that dynamically corrects parameter deviations; and a balanced aggregation mechanism coordinates client updates. Experimental results demonstrate that EFDMVC achieves superior robustness against heterogeneous uncertain views across multiple benchmark datasets, consistently outperforming all state-of-the-art baselines in comprehensive evaluations.

AAAI Conference 2026 Conference Paper

FedRNC: Addressing Spatio-Temporal Label Misalignment in Federated Noisy Class-Incremental Learning

• Xingwei Huang
• Zhaobin Sun
• Junjie Shi
• Xin Yang
• Zengqiang Yan

Federated class-incremental learning (FCIL) aims to incrementally learn new classes across decentralized clients under non-IID data distributions. However, the pervasive challenge of label noise in FCIL has been completely overlooked. In this work, we introduce federated noisy class-incremental learning (FNCIL) and, for the first time, identify a novel form of label noise—spatio-temporal label misalignment—where samples from unseen classes are entirely mislabeled as known classes, with their correctly labeled counterparts appearing in latter tasks or other clients. This phenomenon undermines the effectiveness of existing centralized denoising strategies and creates a clear requirement for noise-robust methods in real-world FNCIL scenarios. To tackle this issue, we propose FedRNC, a dual-phase framework that leverages feature-space associations to establish spatio-temporal correspondences between clean global prototypes and noisy cached samples for progressive label correction. Experiments on standard benchmarks demonstrate FedRNC's superiority against existing baselines, along with its plug-and-play capability to upgrade FCIL systems for FNCIL.

TAAS Journal 2026 Journal Article

FlePo: GPU Multi-task Scheduling Optimization Framework for Dynamic Scenes

• Huanghuang Liang
• Xin Yang
• Rui Ge
• Yaqi Xia
• Chuang Hu
• Xiaobo Zhou
• Dazhao Cheng

Deep Neural Networks (DNNs) are widely used in intelligent applications, driving increasing computational demands on GPUs. However, modern GPU multitasking scheduling algorithms fail to effectively balance real-time task performance and resource utilization, especially under dynamic workloads with highly variable DNN computational demands. The complex and workload-dependent execution times of DNN kernels often lead to inefficient resource allocation, degraded system throughput, and missed real-time constraints. To address these challenges, we propose FlePo (Flexible Parallel Orchestrator), a GPU multitasking scheduling framework designed to optimize resource utilization and maintain real-time task performance within acceptable limits for soft real-time systems. FlePo integrates two key techniques: Adaptive Padding Dispatch (APD), which dynamically schedules best-effort tasks while leveraging the predictable execution characteristics of DNN kernels to maintain real-time predictability; and Dynamic Parallel Fusion (DPF), which employs kernel fusion to create computational isolation, reducing interference in parallel job execution. By combining offline profiling with online adaptation, FlePo efficiently responds to workload variations. We evaluate FlePo on two heterogeneous GPU platforms, NVIDIA Tesla V100 and AMD MI50, achieving up to a 50% increase in throughput while keeping real-time overhead below 2%. Our work enhances GPU multitasking in dynamic environments, with potential applications in autonomous driving, smart homes, and intelligent healthcare.

AAAI Conference 2026 Conference Paper

Generalized Geometry Encoding Volume for Real-time Stereo Matching

• Jiaxin Liu
• Gangwei Xu
• Xianqi Wang
• Chengliang Zhang
• Xin Yang

Real-time stereo matching methods primarily focus on enhancing in-domain performance but often overlook the critical importance of generalization in real-world applications. In contrast, recent stereo foundation models leverage monocular foundation models (MFMs) to improve generalization, but typically suffer from substantial inference latency. To address this trade-off, we propose Generalized Geometry Encoding Volume (GGEV), a novel real-time stereo matching network that achieves strong generalization. We first extract depth-aware features that encode domain-invariant structural priors as guidance for cost aggregation. Subsequently, we introduce a Depth-aware Dynamic Cost Aggregation (DDCA) module that adaptively incorporates these priors into each disparity hypothesis, effectively enhancing fragile matching relationships in unseen scenes. Both steps are lightweight and complementary, leading to the construction of a generalized geometry encoding volume with strong generalization capability. Experimental results demonstrate that our GGEV surpasses all existing real-time methods in zero-shot generalization capability, and achieves state-of-the-art performance on the KITTI 2012, KITTI 2015, and ETH3D benchmarks.

EAAI Journal 2026 Journal Article

Lightweight method of foreign matter detection in coal conveying based on improved you only look once version 8 and embedded equipment

• Guanfeng Du
• Hongzheng Zhang
• Yupeng Luo
• Zhibo Bao
• Zhiwei Li
• Mingxin Zhou
• Zhelin Liu
• Shengxian Cao

During the process of conveying pulverized coal, the mixed foreign matter will not only affect the combustion efficiency of pulverized coal, but also cause safety accidents in coal conveying equipment. Therefore, it is very important to monitor the foreign matter in the process of conveying coal. Due to the limited scope of the actual conveying site, embedded equipment is needed for inspection. Aiming at the computing and memory challenges of embedded equipment, an improved lightweight YOLOv8 (you only look once version 8) algorithm is proposed. In the backbone of the algorithm, cross stage partial with 2 convolutions and lightweight PoolFormer (C2f_LPF) module is used to extract lightweight features, and foreign matter information is extracted by using multi-scale concerns in cross stage partial with deformable convolution (CSPDC) module. Then the part of the feature aggregation (PFA) module of the neck is used for lightweight feature fusion. The proposed C2f_LPF+CSPDC+PFA combination realizes a more balanced optimization of lightweight performance and detection accuracy and provides a solution to the contradiction between the limitation of computing resources of embedded equipment and the demand for real-time detection of accuracy in coal conveying. A self-made datasets contain scrap iron, stones, wooden stick and branch are trained and compared with faster region-based convolutional neural network (Faster R-CNN) and YOLO (you only look once) series algorithm on computer and embedded equipment. The datasets consist of 612 images with 4413 examples of foreign matter, which are collected on a lab-scale self-made coal conveying platform. The mean average precision (mAP), Giga floating-point operations per second (GFLOPS), parameters and frame per second (FPS) on embedded equipment are 0. 963, 6. 1, 2. 44 million and 37. 04, respectively. Compared with the original YOLOv8, the computation and parameters are reduced by 24. 7 % and 18. 9 % respectively, and the FPS is improved by 29. 6 %. By contrast, it also has better results than other algorithms, which is well compatible with the configuration requirements of embedded equipment and achieves a good balance between precision and speed. This shows a promising performance on a lab-scale platform and may be extended to real industrial lines after further validation.

JBHI Journal 2026 Journal Article

OnUVS: An Online Motion Transfer Framework with Content-Texture Decoupling for High-Fidelity Ultrasound Video Synthesis

• Han Zhou
• Rusi Chen
• Xin Yang
• Ao Chang
• Junxuan Yu
• Yuhao Huang
• Ruobing Huang
• Xinrui Zhou

Ultrasound (US) imaging plays a crucial role in diagnosing heart and pelvic diseases, where sonographers tend to evaluate dynamic motion and structure. However, the scarcity of US videos for rare cases limits training opportunities for novice sonographers and deep learning models, hindering detection rates and clinical di agnostic applications. US video synthesis is a promising solution to this issue. Nevertheless, accurately imitating the intricate motion of the anatomy while preserving image f idelity presents a significant challenge. In this work, we propose OnUVS, a novel online feature-decoupling frame work for high-fidelity US video synthesis. First, to simulate realistic motion, we incorporate keypoints into anatomical learning through a weakly supervised training approach, which enhances motion representation and minimizes the need for fully annotated data. Second, we implement a dual decoder generator that effectively balances content and textural features of generated frames, significantly enhancing the image fidelity of US videos. Third, a multi-scale discriminator further refines the sharpness and fine details, ensuring high-fidelity video synthesis. Fourth, an online learning strategy is designed to smooth coherence between frames by constraining the keypoint trajectories during inference. Validation on echocardiographic and pelvic floor US datasets demonstrates that OnUVS outperforms existing methods, achieving a 22. 08% improvement in motion consistency (FVD) and 25. 04% in image fidelity (FID). To facilitate reproducibility, we publicly release the code of OnUVSat: https://github.com/LucyChen159/OnUVS.

JBHI Journal 2026 Journal Article

Parallel Multi-Attention and Gated Fusion for Visual Question Localized Answering in Surgical Scenes

• Zeyu Wang
• Ming Wang
• Peixi Peng
• Wanshu Fan
• Zhongbin Han
• Xin Yang
• Dongsheng Zhou

Surgical Visual Question Localized Answering (Surgical-VQLA) is an emerging task that supports surgical education by generating accurate answers and localizing relevant anatomical regions based on visual content and textual queries. This task requires precise spatial reasoning and tight semantic alignment across modalities, which remain challenging for current models due to limited spatial sensitivity and insufficient semantic integration. Mitigating these limitations, we propose EndoVisLoc, a dedicated framework that enhances visual-textual interaction through structured attention and gated fusion. Specifically, we design a Parallel Multi Attention Module (PMAM) to capture different visual features, improving the perception of anatomical structures. We further develop a Dynamic Gated Fusion Module (DGFM) to adaptively inject semantic priors into visual features via gated control, facilitating robust cross-modal fusion. Finally, we introduce a Hierarchical Classifier Head (HCH) to refine the fused representations and jointly optimize answer prediction and spatial localization. Extensive experiments on the EndoVis-18-VQLA and EndoVis-17-VQLA datasets demonstrate the superior performance of EndoVisLoc, surpassing the state-of-the-art OTAS model by +5. 72% ACC, +5. 73% F-score, and +1. 82% mIoU on EndoVis-18-VQLA and by +1. 59% ACC, +2. 48% F-score, and +0. 33% mIoU on EndoVis-17-VQLA. These results confirm the consistent advantage of EndoVisLoc in both answer accuracy and precise anatomical localization.

JBHI Journal 2026 Journal Article

RT-SAM: Visual-Prompt Fusion and Uncertainty Enhancement for Nasopharyngeal Carcinoma Radiotherapy Target Delineation

• Hee Guan Khor
• Xin Yang
• Yihua Sun
• Sijuan Huang
• Yingni Wang
• Jie Wang
• Shaobin Wang
• Lu Bai

Precise delineation of the clinical target volume (CTV) and nodal CTV (CTV $_{{\mathit{nd}}}$ ) is crucial for effective radiotherapy planning in nasopharyngeal carcinoma (NPC). Manual contouring is labor-intensive and subject to substantial inter-observer variability, particularly in regions with complex anatomy and indistinct boundaries. This study presents RT-SAM, a novel framework that adapts the Medical Segment Anything Model 2 (MedSAM-2) for automated CTV (i. e. , primary CTV and CTV $_{nd}$ ) contouring in NPC computed tomography (CT) images. The framework synergistically integrates a generalist foundation model (MedSAM-2) with a domain-specific specialist network (2D U-Net) through three principal contributions: (1) automated generation of multi-modal prompts—comprising mask, bounding box, and point representations—derived from specialist network predictions to guide the generalist model; (2) a Visual-Prompt Fusion Attention (ViPFA) mechanism that optimizes feature-prompt interactions through bidirectional cross-modal attention; and (3) an Uncertainty-Enhanced Prediction Adjustment (UEPA) mechanism that enhances model robustness via confidence-based refinement and selective domain adaptation. Comprehensive evaluation on a multi-center cohort of 256 clinical NPC cases from Sun Yat-sen University Cancer Center and 212 public NPC cases from the SegRap2025 lymph node CTV dataset using 5- fold cross-validation demonstrates that RT-SAM achieves a mean DICE coefficient of 0. 796 $\pm$ 0. 033 (mean $\pm$ standard deviation), significantly outperforming current state-of-the-art methods. Clinical validation by eight radiation oncologists demonstrates that RT-SAM contours are clinically indistinguishable from expert delineations in blinded Turing assessments, achieve superior quality ratings in 75% of comparisons with mean scores of 2. 73 for RT-SAM versus 2. 66 for manual expert contours, and attain clinically acceptable ratings in over 97% of cases. These results demonstrate that RT-SAM is a clinically feasible solution for automated CTV contouring, with strong potential to standardize treatment planning and mitigate inter-observer variability in NPC radiotherapy.

YNIMG Journal 2026 Journal Article

The regulations on topological organization of prefrontal-sensorimotor cortical network elicited by tonifying and reducing manipulations of acupuncture: A graph theory analysis study based on fNIRS

• Jingya Cao
• Yuzhu Qu
• Li Chen
• Tianyu Liu
• Jing Guo
• Zilei Tian
• Chongkai Luo
• Yulai Gong

This study aims to investigate the similarities and differences in the topological organization of prefrontal-sensorimotor cortical network (PFC-SMC) induced by traditional tonifying and reducing manipulations of acupuncture (TRMs), including the tonifying, reducing, and even tonifying-reducing manipulations. Thirty-five healthy participants underwent all three types of TRMs while functional near-infrared spectroscopy data were recorded. To characterize the network properties, graph-theoretical analysis was applied to calculate topological metrics at both global and nodal levels. Based on these metrics, k-means++ clustering analysis was subsequently performed to assess the representational and discriminative capacity of the derived features across different acupuncture manipulation. The results revealed that all three acupuncture manipulations perserved the typical small-world properties of the PFC-SMC network (p < 0. 05, false discovery rate [FDR] corrected). However, further comparison showed that both the even tonifying-reducing manipulation and the reducing manipulation elicited significantly higher clustering coefficient and network efficiency than the tonifying manipulation (pFDR < 0. 05). At the nodal level, the tonifying manipulation showed lower nodal betweenness centrality and nodal degree centrality than the other two acupuncture manipulations (pFDR < 0. 05). Notably, these topological properties enabled reliable differentiation among the three TRMs, as confirmed by the clustering analysis. These findings highlight that different acupuncture manipulations exert distinct modulatory effects on the brain functional network, providing visualized evidence for facilitating the understanding and clinical application of TRMs.

AAAI Conference 2026 Conference Paper

View-on-Graph: Zero-Shot 3D Visual Grounding via Vision-Language Reasoning on Scene Graphs

• Yuanyuan Liu
• Haiyang Mei
• Dongyang Zhan
• Jiayue Zhao
• Dongsheng Zhou
• Bo Dong
• Xin Yang

3D visual grounding (3DVG) identifies objects in 3D scenes from language descriptions. Existing zero-shot approaches leverage 2D vision–language models (VLMs) by converting 3D spatial information (SI) into forms amenable to VLM processing, typically as composite inputs such as specified-view renderings or video sequences with overlaid object markers. However, this VLM ⊕ SI paradigm yields entangled visual representations that compel the VLM to process entire cluttered cues, making it hard to exploit spatial–semantic relationships effectively. In this work, we propose a new VLM ⊗ SI paradigm that externalizes the 3D SI into a form enabling the VLM to incrementally retrieve only what it needs during reasoning. We instantiate this paradigm with a novel View-on-Graph (VoG) method, which organizes the scene into a multi-modal, multi-layer scene graph and allows the VLM to operate as an active agent that selectively accesses necessary cues as it traverses the scene. This design offers two intrinsic advantages: (i) by structuring 3D context into a spatially and semantically coherent scene graph rather than confounding the VLM with densely entangled visual inputs, it lowers the VLM's reasoning difficulty; and (ii) by actively exploring and reasoning over the scene graph, it naturally produces transparent, step-by-step traces for interpretable 3DVG. Extensive experiments show that VoG achieves state-of-the-art zero-shot performance, establishing structured scene exploration as a promising strategy for advancing zero-shot 3DVG.

NeurIPS Conference 2025 Conference Paper

Diversity-oriented Deep Multi-modal Clustering

• Wang Yanzheng
• Xin Yang
• Yujun Wang
• Shizhe Hu
• Mingliang Xu

Deep multi-modal clustering (DMC) aims to explore the correlated information from different modalities to improve the clustering performance. Most existing DMCs attempt to investigate the consistency or/and complementarity information by fusing all modalities, but this will lead to the following challenges: 1) Information conflicts between modalities emerge. 2) Information-rich modalities may be weakened. To address the above challenges, we propose a diversity-oriented deep multi-modal clustering (DDMC) method, where the core is dominant modality enhancement instead of multi-modal fusion. Specifically, we select the modality with the highest average silhouette coefficient as the dominant modality, then learn the diversity information between the dominant madality and the remaining ones with diversity learning, and finally enhance the dominant modality for clustering. Extensive experiments show the superiority of the proposed method over several compared DMC methods. To our knowledge, this is the first work to perform multi-modal clustering by enhancing the dominant modality instead of fusion.

AAAI Conference 2025 Conference Paper

DriveGazen: Event-Based Driving Status Recognition Using Conventional Camera

• Xiaoyin Yang
• Xin Yang

We introduce a wearable driving status recognition device and our open-source dataset, along with a new real-time method robust to changes in lighting conditions for identifying driving status from eye observations of drivers. The core of our method is generating event frames from conventional intensity frames, and the other is a newly designed Attention Driving State Network (ADSN). Compared to event cameras, conventional cameras offer complete information and lower hardware costs, enabling captured frames to encode rich spatial information. However, these textures lack temporal information, posing challenges in effectively identifying driving status. DriveGazen addresses this issue from three perspectives. First, we utilize video frames to generate realistic synthetic dynamic vision sensor (DVS) events.Second, we adopt a spiking neural network to decode pertinent temporal information. Lastly, ADSN extracts crucial spatial cues from corresponding intensity frames and conveys spatial attention to convolutional spiking layers during both training and inference through a novel guide attention module to guide the feature learning and feature enhancement of the event frame. We specifically collected the Driving Status (DriveGaze) dataset to demonstrate the effectiveness of our approach. Additionally, we validate the superiority of the DriveGazen on the Single-eye Event-based Emotion (SEE) dataset. To the best of our knowledge, our method is the first to utilize guide attention spiking neural networks and eye-based event frames generated from conventional cameras for driving status recognition.Please refer to our project page and supplementary materials for more details.

AAAI Conference 2025 Conference Paper

Dynamic Uncertainty Estimation for Offline Reinforcement Learning

• Jiesheng Wang
• Lin Li
• Wei Wei
• Yujia Zhang
• Xin Yang

Offline reinforcement learning confronts the distributional shift challenge, a consequence of learning policy from static datasets. Current methods primarily handle this issue by aligning the learned policy with the behavior policy or conservatively estimating Q-values for out-of-distribution (OOD) actions. However, these approaches can lead to overly pessimistic estimation of Q-values of the OOD actions in unfamiliar situations, resulting in a suboptimal policy. To address this, we propose a new method, Dynamic Uncertainty estimation for Offline Reinforcement Learning. This method introduces a base density-truncated OOD data sampling approach to reduce the impact of extrapolation errors on uncertainty estimation. It enables conservative estimation of Q-values for OOD actions while avoiding negative impacts on in-distribution data. We also develop a dynamic uncertainty estimation mechanism to prevent excessive pessimism and enhance the generalization of the Q-function. This mechanism dynamically adjusts the degree of pessimism in the Q-function by minimizing the error between target and estimated values. Our method outperforms existing algorithms, as demonstrated by experimental results based on the D4RL benchmark, and proves its superiority in addressing the distributional shift challenge.

IROS Conference 2025 Conference Paper

Enhancing Single Image to 3D Generation using Gaussian Splatting and Hybrid Diffusion Priors

• Hritam Basak
• Hadi Tabatabaee
• Shreekant Gayaka
• Ming-Feng Li
• Xin Yang
• Cheng-Hao Kuo
• Arnie Sen
• Min Sun 0001

3D object generation from a single unposed RGB image is essential for robotic perception, as reconstructing complete geometry and texture is essential for precise manipulation, grasping, and scene understanding, which is key for autonomous navigation and dexterous interaction. Recent advancements in image-to-3D employ Gaussian Splatting with pre-trained 2D or 3D diffusion models, but a disparity exists: 2D models generate high-fidelity textures yet lack geometric consistency, while 3D models ensure structural coherence but produce overly smooth textures. To address this, we introduce a two-stage frequency-based distillation loss integrated with Gaussian Splatting, leveraging geometric priors from a 3D diffusion model’s low-frequency spectrum for structural consistency and a 2D diffusion model’s high-frequency details for sharper textures. Our approach achieves state-of-the-art 3D reconstruction quality, significantly improving robotic perception pipelines. Additionally, we demonstrate the easy adaptability of our method for highly accurate object pose estimation and tracking, which is critical for precise robotic grasping, manipulation, and scene understanding. Additional results can be found in the supplementary file.

AAAI Conference 2025 Conference Paper

ERF: A Benchmark Dataset for Robust Semantic Segmentation Under Extreme Rainfall Conditions

• Xin Yang
• Xin Zhang
• Xinchao Wang

As climate change reshapes global weather patterns, the increasing frequency and intensity of extreme rainfall events have amplified the safety imperatives for autonomous driving systems. During such events, rainfall can escalate from heavy to violent, as defined by the World Meteorological Organization, severely impairing images with diverse and significant degradations. Many existing semantic segmentation models perform well under light to heavy rain, but there is a notable absence of datasets addressing violent rain conditions for these models to validate and learn from. In this paper, we introduce the Extreme RainFall (ERF) dataset for semantic segmentation in both image and video tasks under violent rain conditions. Our dataset comprises 14,757 unlabeled frames and 100 labeled frames, all captured during four different violent rainfall periods. We use our dataset to evaluate the robustness of various methods against violent rainfall, focusing on four approaches: 1) image-based foundation models, 2) image-based domain generalization methods, 3) image-based domain adaptation methods, and 4) video-based methods. The results reveal that none of the existing models tested is capable of withstanding the extreme challenges posed by violent rainfall conditions. By analyzing the results, we offer insights and suggestions for developing more robust models under extreme rainfall events.

NeurIPS Conference 2025 Conference Paper

Faster Algorithms for Structured John Ellipsoid Computation

• Yang Cao
• Xiaoyu Li
• Zhao Song
• Xin Yang
• Tianyi Zhou

The famous theorem of Fritz John states that any convex body has a unique maximal volume inscribed ellipsoid, known as the John Ellipsoid. Computing the John Ellipsoid is a fundamental problem in convex optimization. In this paper, we focus on approximating the John Ellipsoid inscribed in a convex and centrally symmetric polytope defined by $P: = \{ x \in \mathbb{R}^d: -\mathbf{1}_n \leq A x \leq \mathbf{1}_n \}, $ where $ A \in \mathbb{R}^{n \times d}$ is a rank-$d$ matrix and $ \mathbf{1}_n \in \mathbb{R}^n $ is the all-ones vector. We develop two efficient algorithms for approximating the John Ellipsoid. The first is a sketching-based algorithm that runs in nearly input-sparsity time $ \widetilde{O}(\mathrm{nnz}(A) + d^\omega) $, where $ \mathrm{nnz}(A) $ denotes the number of nonzero entries in the matrix $A$ and $ \omega \approx 2. 37$ is the current matrix multiplication exponent. The second is a treewidth-based algorithm that runs in time $ \widetilde{O}(n \tau^2)$, where $\tau$ is the treewidth of the dual graph of the matrix $A$. Our algorithms significantly improve upon the state-of-the-art running time of $ \widetilde{O}(n d^2) $ achieved by [Cohen, Cousins, Lee, and Yang, COLT 2019].

AAAI Conference 2025 Conference Paper

FlowMamba: Learning Point Cloud Scene Flow with Global Motion Propagation

• Min Lin
• Gangwei Xu
• Yun Wang
• Xianqi Wang
• Xin Yang

Scene flow methods based on deep learning have achieved impressive performance. However, current top-performing methods still struggle with ill-posed regions, such as extensive flat regions or occlusions, due to insufficient local evidence. In this paper, we propose a novel global-aware scene flow estimation network with global motion propagation, named FlowMamba. The core idea of FlowMamba is a novel Iterative Unit based on the State Space Model (ISU), which first propagates global motion patterns and then adaptively integrates the global motion information with previously hidden states. As the irregular nature of point clouds limits the performance of ISU in global motion propagation, we propose a feature-induced ordering strategy (FIO). The FIO leverages semantic-related and motion-related features to order points into a sequence characterized by spatial continuity. Extensive experiments demonstrate the effectiveness of FlowMamba, with 21.9% and 20.5% EPE3D reduction from the best published results on FlyingThings3D and KITTI datasets. Specifically, our FlowMamba is the first method to achieve millimeter-level prediction accuracy in FlyingThings3D and KITTI. Furthermore, the proposed ISU can be seamlessly embedded into existing iterative networks as a plug-and-play module, improving their estimation accuracy significantly.

NeurIPS Conference 2025 Conference Paper

Fully Autonomous Neuromorphic Navigation and Dynamic Obstacle Avoidance

• Xiaochen Shang
• Pengwei Luo
• Xinning Wang
• Jiayue Zhao
• Huilin Ge
• Bo Dong
• Xin Yang

Unmanned aerial vehicles could accurately accomplish complex navigation and obstacle avoidance tasks under external control. However, enabling unmanned aerial vehicles (UAVs) to rely solely on onboard computation and sensing for real-time navigation and dynamic obstacle avoidance remains a significant challenge due to stringent latency and energy constraints. Inspired by the efficiency of biological systems, we propose a fully neuromorphic framework achieving end-to-end obstacle avoidance during navigation with an overall latency of just 2. 3 milliseconds. Specifically, our bio-inspired approach enables accurate moving object detection and avoidance without requiring target recognition or trajectory computation. Additionally, we introduce the first monocular event-based pose correction dataset with over 50, 000 paired and labeled event streams. We validate our system on an autonomous quadrotor using only onboard resources, demonstrating reliable navigation and avoidance of diverse obstacles moving at speeds up to 10 m/s.

AAAI Conference 2025 Conference Paper

Leveraging Consistent Spatio-Temporal Correspondence for Robust Visual Odometry

• Zhaoxing Zhang
• Junda Cheng
• Gangwei Xu
• Xiaoxiang Wang
• Can Zhang
• Xin Yang

Recent approaches to VO have significantly improved performance by using deep networks to predict optical flow between video frames. However, existing methods still suffer from noisy and inconsistent flow matching, making it difficult to handle challenging scenarios and long-sequence estimation.To overcome these challenges, we introduce Spatio-Temporal Visual Odometry (STVO), a novel deep network architecture that effectively leverages inherent spatio-temporal cues to enhance the accuracy and consistency of multi-frame flow matching. With more accurate and consistent flow matching, STVO can achieve better pose estimation through the bundle adjustment (BA).Specifically, STVO introduces two innovative components: 1) the Temporal Propagation Module that utilizes multi-frame information to extract and propagate temporal cues across adjacent frames, maintaining temporal consistency; 2) the Spatial Activation Module that utilizes geometric priors from the depth maps to enhance spatial consistency while filtering out excessive noise and incorrect matches.Our STVO achieves state-of-the-art performance on TUM-RGBD, EuRoc MAV, ETH3D and KITTI Odometry benchmarks. Notably, it improves accuracy by 77.8% on ETH3D benchmark and 38.9% on KITTI Odometry benchmark over the previous best methods.

IROS Conference 2025 Conference Paper

LiDAR-Inertial Odometry in Dynamic Driving Scenarios using Label Consistency Detection

• Zikang Yuan
• Xiaoxiang Wang
• Jingying Wu
• Junda Cheng
• Xin Yang

In this paper, a LiDAR-inertial odometry (LIO) method that eliminates the influence of moving objects in dynamic driving scenarios is proposed. This method constructs binarized labels for 3D points of current sweep, and utilizes the label difference between each point and its surrounding points in global map to identify moving objects. The surrounding points in global map are localized by voxel-location-based nearest neighbor search, without involving any massive computations. In addition, the proposed method is embeded into a LIO system (i. e. , Dynamic-LIO), and achieves state-of-the-art performance on public datasets with extremlely low computational overhead (i. e. , 1~9ms/sweep). We have released the source code of this work for the development of the community.

IJCAI Conference 2025 Conference Paper

Multi-granularity Knowledge Transfer for Continual Reinforcement Learning

• Chaofan Pan
• Lingfei Ren
• Yihui Feng
• Linbo Xiong
• Wei Wei
• Yonghao Li
• Xin Yang

Continual reinforcement learning (CRL) empowers RL agents with the ability to learn a sequence of tasks, accumulating knowledge learned in the past and using the knowledge for problemsolving or future task learning. However, existing methods often focus on transferring fine-grained knowledge across similar tasks, which neglects the multi-granularity structure of human cognitive control, resulting in insufficient knowledge transfer across diverse tasks. To enhance coarse-grained knowledge transfer, we propose a novel framework called MT-Core (as shorthand for Multi-granularity knowledge Transfer for Continual reinforcement learning). MT-Core has a key characteristic of multi-granularity policy learning: 1) a coarsegrained policy formulation for utilizing the powerful reasoning ability of the large language model (LLM) to set goals, and 2) a fine-grained policy learning through RL which is oriented by the goals. We also construct a new policy library (knowledge base) to store policies that can be retrieved for multi-granularity knowledge transfer. Experimental results demonstrate the superiority of the proposed MT-Core in handling diverse CRL tasks versus popular baselines.

AAAI Conference 2025 Conference Paper

Multi-Granularity Open Intent Classification via Adaptive Granular-Ball Decision Boundary

• Yanhua Li
• Xiaocao Ouyang
• Chaofan Pan
• Jie Zhang
• Sen Zhao
• Shuyin Xia
• Xin Yang
• Guoyin Wang

Open intent classification is critical for the development of dialogue systems, aiming to accurately classify known intents into their corresponding classes while identifying unknown intents. Prior boundary-based methods assumed known intents fit within compact spherical regions, focusing on coarse-grained representation and precise spherical decision boundaries. However, these assumptions are often violated in practical scenarios, making it difficult to distinguish known intent classes from unknowns using a single spherical boundary. To tackle these issues, we propose a Multi-granularity Open intent classification method via adaptive Granular-Ball decision boundary (MOGB). Our MOGB method consists of two modules: representation learning and decision boundary acquiring. To effectively represent the intent distribution, we design a hierarchical representation learning method. This involves iteratively alternating between adaptive granular-ball clustering and nearest sub-centroid classification to capture fine-grained semantic structures within known intent classes. Furthermore, multi-granularity decision boundaries are constructed for open intent classification by employing granular-balls with varying centroids and radii. Extensive experiments conducted on three public datasets demonstrate the effectiveness of our proposed method.

NeurIPS Conference 2025 Conference Paper

Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformers

• Gangwei Xu
• Haotong Lin
• Hongcheng Luo
• Xianqi Wang
• JINGFENG YAO
• Lianghui Zhu
• Yuechuan Pu
• Cheng Chi_

This paper presents Pixel-Perfect Depth, a monocular depth estimation model based on pixel-space diffusion generation that produces high-quality, flying-pixel-free point clouds from estimated depth maps. Current generative depth estimation models fine-tune Stable Diffusion and achieve impressive performance. However, they require a VAE to compress depth maps into the latent space, which inevitably introduces flying pixels at edges and details. Our model addresses this challenge by directly performing diffusion generation in the pixel space, avoiding VAE-induced artifacts. To overcome the high complexity associated with pixel-space generation, we introduce two novel designs: 1) Semantics-Prompted Diffusion Transformers ( SP-DiT ), which incorporate semantic representations from vision foundation models into DiT to prompt the diffusion process, thereby preserving global semantic consistency while enhancing fine-grained visual details; and 2) Cascade DiT Design that progressively increases the number of tokens to further enhance efficiency and accuracy. Our model achieves the best performance among all published generative models across five benchmarks, and significantly outperforms all other models in edge-aware point cloud evaluation. Project page: https: //pixel-perfect-depth. github. io/.

AAAI Conference 2025 Conference Paper

Semantic Segmentation on Raindrop Degraded Images Using Two-Stage Dual Teacher-Student Learning

• Xin Yang
• Wending Yan
• Yuan Yuan
• Michael Bi Mi
• Robby T. Tan

Existing semantic segmentation methods face challenges when processing input images degraded by raindrops on the lens or windshield. Unlike other adverse conditions such as fog and nighttime, which degrade visual quality, raindrops not only impair visual appearances but also introduce misleading occlusion, leading to significant performance drops in current models. The novelty of our approach lies in our two-stage, dual teacher-student framework. We tackle the complex problem of raindrop degradation by dividing it into two distinct challenges: degraded visual appearance and raindrop occlusion. These challenges are then addressed individually in two stages, utilizing two pairs of teacher-student networks. This division enables the networks to develop specialized expertise in handling each aspect of raindrop degradation, enabling their collaboration to achieve superior performance. In the first stage, one teacher-student pair focuses on learning to extract information from visual degraded areas. Building on this, the second teacher-student pair focuses specially on the raindrop occlusion. As such, unlike the existing methods, our approach employs a collaborative approach to decompose and address raindrop-induced degradations. In the second stage, we introduce a mask-based recovery technique to identify and rectify areas that likely contain misleading information, thus further refining the predictions. Additionally, this stage encourages both pairs to expand knowledge by swapping their specialized expertise. Our method achieves a performance of 60.3 mIoU on Rainy WCity and 72.8 mIoU on ACDC Rainy, representing an improvement of +4.4 mIoU and +2.3 mIoU over the existing state-of-the-art methods, respectively.

AAAI Conference 2025 Conference Paper

Separating the Wheat from the Chaff: Spatio-Temporal Transformer with View-interweaved Attention for Photon-Efficient Depth Sensing

• Letian Yu
• Jiaxi Yang
• Bo Dong
• Qirui Bao
• Yuanbo Wang
• Felix Heide
• Xiaopeng Wei
• Xin Yang

Time-resolved imaging is an emerging sensing modality that has been shown to enable advanced applications, including remote sensing, fluorescence lifetime imaging, and even non-line-of-sight sensing. Single-photon avalanche diodes (SPADs) outperform relevant time-resolved imaging technologies thanks to their excellent photon sensitivity and superior temporal resolution on the order of tens of picoseconds. The capability of exceeding the sensing limits of conventional cameras for SPADs also draws attention to the photon-efficient imaging area. However, photon-efficient imaging under degraded conditions with low photon counts and low signal-to-background ratio (SBR) still remains an inevitable challenge. In this paper, we propose a spatio-temporal transformer network for photon-efficient imaging under low-flux scenarios. In particular, we introduce a view-interweaved attention mechanism (VIAM) to extract both spatial-view and temporal-view self-attention in each transformer block. We also design an adaptive-weighting scheme to dynamically adjust the weights between different views of self-attention in VIAM for different signal-to-background levels. We extensively validate and demonstrate the effectiveness of our approach on the simulated Middlebury dataset and a specially self-collected dataset with real-world-captured SPAD measurements and well-annotated ground truth depth maps.

AIIM Journal 2024 Journal Article

A clinically actionable and explainable real-time risk assessment framework for stroke-associated pneumonia

• Lutao Dai
• Xin Yang
• Hao Li
• Xingquan Zhao
• Lin Lin
• Yong Jiang
• Yongjun Wang
• Zixiao Li

The current medical practice is more responsive rather than proactive, despite the widely recognized value of early disease detection, including improving the quality of care and reducing medical costs. One of the cornerstones of early disease detection is clinically actionable predictions, where predictions are expected to be accurate, stable, real-time and interpretable. As an example, we used stroke-associated pneumonia (SAP), setting up a transformer-encoder-based model that analyzes highly heterogeneous electronic health records in real-time. The model was proven accurate and stable on an independent test set. In addition, it issued at least one warning for 98. 6 % of SAP patients, and on average, its alerts were ahead of physician diagnoses by 2. 71 days. We applied Integrated Gradient to glean the model's reasoning process. Supplementing the risk scores, the model highlighted critical historical events on patients' trajectories, which were shown to have high clinical relevance.

IJCAI Conference 2024 Conference Paper

Apprenticeship-Inspired Elegance: Synergistic Knowledge Distillation Empowers Spiking Neural Networks for Efficient Single-Eye Emotion Recognition

• Yang Wang
• Haiyang Mei
• Qirui Bao
• Ziqi Wei
• Mike Zheng Shou
• Haizhou Li
• Bo Dong
• Xin Yang

We introduce a novel multimodality synergistic knowledge distillation scheme tailored for efficient single-eye motion recognition tasks. This method allows a lightweight, unimodal student spiking neural network (SNN) to extract rich knowledge from an event-frame multimodal teacher network. The core strength of this approach is its ability to utilize the ample, coarser temporal cues found in conventional frames for effective emotion recognition. Consequently, our method adeptly interprets both temporal and spatial information from the conventional frame domain, eliminating the need for specialized sensing devices, e. g. , event-based camera. The effectiveness of our approach is thoroughly demonstrated using both existing and our compiled single-eye emotion recognition datasets, achieving unparalleled performance in accuracy and efficiency over existing state-of-the-art methods.

TIST Journal 2024 Journal Article

Discovering Expert-Level Air Combat Knowledge via Deep Excitatory-Inhibitory Factorized Reinforcement Learning

• Hai Yin Piao
• Shengqi Yang
• Hechang Chen
• Junnan Li
• Jin Yu
• Xuanqi Peng
• Xin Yang
• Zhen Yang

Artificial Intelligence (AI) has achieved a wide range of successes in autonomous air combat decision-making recently. Previous research demonstrated that AI-enabled air combat approaches could even acquire beyond human-level capabilities. However, there remains a lack of evidence regarding two major difficulties. First, the existing methods with fixed decision intervals are mostly devoted to solving what to act but merely pay attention to when to act, which occasionally misses optimal decision opportunities. Second, the method of an expert-crafted finite maneuver library leads to a lack of tactics diversity, which is vulnerable to an opponent equipped with new tactics. In view of this, we propose a novel Deep Reinforcement Learning (DRL) and prior knowledge hybrid autonomous air combat tactics discovering algorithm, namely deep E xcitatory-i N hibitory f ACT or I zed maneu VE r ( ENACTIVE ) learning. The algorithm consists of two key modules, i.e., ENHANCE and FACTIVE. Specifically, ENHANCE learns to adjust the air combat decision-making intervals and appropriately seize key opportunities. FACTIVE factorizes maneuvers and then jointly optimizes them with significant tactics diversity increments. Extensive experimental results reveal that the proposed method outperforms state-of-the-art algorithms with a 62% winning rate and further obtains a margin of a 2.85-fold increase in terms of global tactic space coverage. It also demonstrates that a variety of discovered air combat tactics are comparable to human experts’ knowledge.

NeurIPS Conference 2024 Conference Paper

End-to-End Video Semantic Segmentation in Adverse Weather using Fusion Blocks and Temporal-Spatial Teacher-Student Learning

• Xin Yang
• Yan Wending
• Michael B. Mi
• Yuan Yuan
• Robby T. Tan

Adverse weather conditions can significantly degrade the video frames, causing existing video semantic segmentation methods to produce erroneous predictions. In this work, we target adverse weather conditions and introduce an end-to-end domain adaptation strategy that leverages a fusion block, temporal-spatial teacher-student learning, and a temporal weather degradation augmentation approach. The fusion block integrates temporal information from adjacent frames at the feature level, trained end-to-end, eliminating the need for pretrained optical flow, distinguishing our method from existing approaches. Our teacher-student approach involves two teachers: one focuses on exploring temporal information from adjacent frames, and the other harnesses spatial information from the current frame. Finally, we apply temporal weather degradation augmentation to consecutive frames to more accurately represent adverse weather degradations. Our method achieves a performance of 25. 4 and 33. 0 mIoU on the adaptation from VIPER and Synthia to MVSS, respectively, representing an improvement of 4. 3 and 5. 8 mIoU over the existing state-of-the-art method.

AAAI Conference 2024 Conference Paper

Exploiting Polarized Material Cues for Robust Car Detection

• Wen Dong
• Haiyang Mei
• Ziqi Wei
• Ao Jin
• Sen Qiu
• Qiang Zhang
• Xin Yang

Car detection is an important task that serves as a crucial prerequisite for many automated driving functions. The large variations in lighting/weather conditions and vehicle densities of the scenes pose significant challenges to existing car detection algorithms to meet the highly accurate perception demand for safety, due to the unstable/limited color information, which impedes the extraction of meaningful/discriminative features of cars. In this work, we present a novel learning-based car detection method that leverages trichromatic linear polarization as an additional cue to disambiguate such challenging cases. A key observation is that polarization, characteristic of the light wave, can robustly describe intrinsic physical properties of the scene objects in various imaging conditions and is strongly linked to the nature of materials for cars (e.g., metal and glass) and their surrounding environment (e.g., soil and trees), thereby providing reliable and discriminative features for robust car detection in challenging scenes. To exploit polarization cues, we first construct a pixel-aligned RGB-Polarization car detection dataset, which we subsequently employ to train a novel multimodal fusion network. Our car detection network dynamically integrates RGB and polarization features in a request-and-complement manner and can explore the intrinsic material properties of cars across all learning samples. We extensively validate our method and demonstrate that it outperforms state-of-the-art detection methods. Experimental results show that polarization is a powerful cue for car detection. Our code is available at https://github.com/wind1117/AAAI24-PCDNet.

JBHI Journal 2024 Journal Article

Eye Gaze Guided Cross-Modal Alignment Network for Radiology Report Generation

• Peixi Peng
• Wanshu Fan
• Yue Shen
• Wenfei Liu
• Xin Yang
• Qiang Zhang
• Xiaopeng Wei
• Dongsheng Zhou

The potential benefits of automatic radiology report generation, such as reducing misdiagnosis rates and enhancing clinical diagnosis efficiency, are significant. However, existing data-driven methods lack essential medical prior knowledge, which hampers their performance. Moreover, establishing global correspondences between radiology images and related reports, while achieving local alignments between images correlated with prior knowledge and text, remains a challenging task. To address these shortcomings, we introduce a novel Eye Gaze Guided Cross-modal Alignment Network (EGGCA-Net) for generating accurate medical reports. Our approach incorporates prior knowledge from radiologists' Eye Gaze Region (EGR) to refine the fidelity and comprehensibility of report generation. Specifically, we design a Dual Fine-Grained Branch (DFGB) and a Multi-Task Branch (MTB) to collaboratively ensure the alignment of visual and textual semantics across multiple levels. To establish fine-grained alignment between EGR-related images and sentences, we introduce the Sentence Fine-grained Prototype Module (SFPM) within DFGB to capture cross-modal information at different levels. Additionally, to learn the alignment of EGR-related image topics, we introduce the Multi-task Feature Fusion Module (MFFM) within MTB to refine the encoder output information. Finally, a specifically designed label matching mechanism is designed to generate reports that are consistent with the anticipated disease states. The experimental outcomes indicate that the introduced methodology surpasses previous advanced techniques, yielding enhanced performance on two extensively used benchmark datasets: Open-i and MIMIC-CXR.

EAAI Journal 2024 Journal Article

Goal-CurveNet: A pedestrian trajectory prediction network using heterogeneous graph attention goal prediction and curve fitting

• Xiangchen Wang
• Xin Yang
• Dake Zhou

Pedestrian trajectory prediction in dynamic and strongly interactive scenes has become one of the most challenging problems in fields such as automated driving. In this paper, we propose Goal-CurveNet, a multimodal trajectory prediction network combining heterogeneous graph attention goal prediction and curve fitting. The model addresses the problems of pedestrian interaction modeling, multimodal trajectory prediction, and performance of predicted trajectories in pedestrian trajectory prediction. Goal-CurveNet can better model the historical trajectories and interaction behaviors in the scene systematically based on heterogeneous graph attention. It predicts the complete trajectories by curve fitting, which effectively improves the quality of predicted trajectories. The model architecture of “goal first and then trajectory” and targeted training paradigm also enhance the final performance. Through detailed training and testing on ETH & UCY datasets, we validate the effectiveness of each contribution of Goal-CurveNet. Compared to many state-of-the-art models, Goal-CurveNet achieves performance improvements in key metrics and effective prediction of pedestrian trajectories.

AAAI Conference 2024 Conference Paper

Learning to Prompt Knowledge Transfer for Open-World Continual Learning

• Yujie Li
• Xin Yang
• Hao Wang
• Xiangkun Wang
• Tianrui Li

This paper studies the problem of continual learning in an open-world scenario, referred to as Open-world Continual Learning (OwCL). OwCL is increasingly rising while it is highly challenging in two-fold: i) learning a sequence of tasks without forgetting knowns in the past, and ii) identifying unknowns (novel objects/classes) in the future. Existing OwCL methods suffer from the adaptability of task-aware boundaries between knowns and unknowns, and do not consider the mechanism of knowledge transfer. In this work, we propose Pro-KT, a novel prompt-enhanced knowledge transfer model for OwCL. Pro-KT includes two key components: (1) a prompt bank to encode and transfer both task-generic and task-specific knowledge, and (2) a task-aware open-set boundary to identify unknowns in the new tasks. Experimental results using two real-world datasets demonstrate that the proposed Pro-KT outperforms the state-of-the-art counterparts in both the detection of unknowns and the classification of knowns markedly. Code released at https://github.com/YujieLi42/Pro-KT.

JBHI Journal 2024 Journal Article

MFTrans: Modality-Masked Fusion Transformer for Incomplete Multi-Modality Brain Tumor Segmentation

• Junjie Shi
• Li Yu
• Qimin Cheng
• Xin Yang
• Kwang-Ting Cheng
• Zengqiang Yan

Brain tumor segmentation is a fundamental task and existing approaches usually rely on multi-modality magnetic resonance imaging (MRI) images for accurate segmentation. However, the common problem of missing/incomplete modalities in clinical practice would severely degrade their segmentation performance, and existing fusion strategies for incomplete multi-modality brain tumor segmentation are far from ideal. In this work, we propose a novel framework named M $^{2}$ FTrans to explore and fuse cross-modality features through modality-masked fusion transformers under various incomplete multi-modality settings. Considering vanilla self-attention is sensitive to missing tokens/inputs, both learnable fusion tokens and masked self-attention are introduced to stably build long-range dependency across modalities while being more flexible to learn from incomplete modalities. In addition, to avoid being biased toward certain dominant modalities, modality-specific features are further re-weighted through spatial weight attention and channel-wise fusion transformers for feature redundancy reduction and modality re-balancing. In this way, the fusion strategy in M $^{2}$ FTrans is more robust to missing modalities. Experimental results on the widely-used BraTS2018, BraTS2020, and BraTS2021 datasets demonstrate the effectiveness of M $^{2}$ FTrans, outperforming the state-of-the-art approaches with large margins under various incomplete modalities for brain tumor segmentation.

AAAI Conference 2024 Conference Paper

Semantic Segmentation in Multiple Adverse Weather Conditions with Domain Knowledge Retention

• Xin Yang
• Wending Yan
• Yuan Yuan
• Michael Bi Mi
• Robby T. Tan

Semantic segmentation's performance is often compromised when applied to unlabeled adverse weather conditions. Unsupervised domain adaptation is a potential approach to enhancing the model's adaptability and robustness to adverse weather. However, existing methods encounter difficulties when sequentially adapting the model to multiple unlabeled adverse weather conditions. They struggle to acquire new knowledge while also retaining previously learned knowledge. To address these problems, we propose a semantic segmentation method for multiple adverse weather conditions that incorporates adaptive knowledge acquisition, pseudo-label blending, and weather composition replay. Our adaptive knowledge acquisition enables the model to avoid learning from extreme images that could potentially cause the model to forget. In our approach of blending pseudo-labels, we not only utilize the current model but also integrate the previously learned model into the ongoing learning process. This collaboration between the current teacher and the previous model enhances the robustness of the pseudo-labels for the current target. Our weather composition replay mechanism allows the model to continuously refine its previously learned weather information while simultaneously learning from the new target domain. Our method consistently outperforms the state-of-the-art methods, and obtains the best performance with averaged mIoU (%) of 65.7 and the lowest forgetting (%) of 3.6 against 60.1 and 11.3, on the ACDC datsets for a four-target continual multi-target domain adaptation.

JBHI Journal 2023 Journal Article

Accurate Cobb Angle Estimation on Scoliosis X-Ray Images via Deeply-Coupled Two-Stage Network With Differentiable Cropping and Random Perturbation

• Yuanhuai Liang
• Jinxin Lv
• Dun Li
• Xin Yang
• Zhiwei Wang
• Qiang Li

Automated Cobb angle estimation on X-ray images is crucial to scoliosis diagnosis. The existing efforts are typically two extremes, which either laboriously detect the raw vertebral landmarks or directly regress Cobb angles from the entire image. In this paper, we propose a novel two-stage end-to-end method as a balanced solution, to avoid vulnerability to false landmarks, and to preserve flexibility in clinical usages. Concretely, we cascade two stages sequentially for detecting vertebrae and then regressing their bending directions instead of raw landmarks. In the detection stage, we combine two networks called LocNet and SegNet to robustly localize vertebrae, and meanwhile to suppress the false positives by additionally segmenting the whole spine. In the subsequent stage, we introduce a regression network named RegNet to accurately regress bending directions of localized vertebrae. Furthermore, the vertebra-aligned local regions on LocNet's intermediate features are cropped via RoIAlign-pooling, and RegNet inherits the cropped regions to learn only feature residuals. By doing so, the regression difficulty can be dramatically alleviated, and the two stages are deeply coupled and mutually guided in an end-to-end training. Moreover, a random perturbation on the inherited features further enhances RegNet's robustness. We benchmark our method on both public and private datasets, and the errors are 2. 92 $\pm$ 2. 34 $^{\circ }$ and 6. 87 $\pm$ 6. 26% in terms of CMAE and SMAPE on the widely-employed AASCE dataset, outperforming other state-of-the-arts by at least 16. 81% and 6. 15%, respectively. Also, a clinical user study verifies our promising flexibility for allowing convenient rectifications to further decrease errors by a large marge.

JBHI Journal 2023 Journal Article

Affinity Feature Strengthening for Accurate, Complete and Robust Vessel Segmentation

• Tianyi Shi
• Xiaohuan Ding
• Wei Zhou
• Feng Pan
• Zengqiang Yan
• Xiang Bai
• Xin Yang

Vessel segmentation is crucial in many medical image applications, such as detecting coronary stenoses, retinal vessel diseases and brain aneurysms. However, achieving high pixel-wise accuracy, complete topology structure and robustness to various contrast variations are critical and challenging, and most existing methods focus only on achieving one or two of these aspects. In this paper, we present a novel approach, the affinity feature strengthening network (AFN), which jointly models geometry and refines pixel-wise segmentation features using a contrast-insensitive, multiscale affinity approach. Specifically, we compute a multiscale affinity field for each pixel, capturing its semantic relationships with neighboring pixels in the predicted mask image. This field represents the local geometry of vessel segments of different sizes, allowing us to learn spatial- and scale-aware adaptive weights to strengthen vessel features. We evaluate our AFN on four different types of vascular datasets: X-ray angiography coronary vessel dataset (XCAD), portal vein dataset (PV), digital subtraction angiography cerebrovascular vessel dataset (DSA) and retinal vessel dataset (DRIVE). Extensive experimental results demonstrate that our AFN outperforms the state-of-the-art methods in terms of both higher accuracy and topological metrics, while also being more robust to various contrast changes.

AAAI Conference 2023 Short Paper

Cross-Regional Fraud Detection via Continual Learning (Student Abstract)

• Yujie Li
• Yuxuan Yang
• Qiang Gao
• Xin Yang

Detecting fraud is an urgent task to avoid transaction risks. Especially when expanding a business to new cities or new countries, developing a totally new model will bring the cost issue and result in forgetting previous knowledge. This study proposes a novel solution based on heterogeneous trade graphs, namely HTG-CFD, to prevent knowledge forgetting of cross-regional fraud detection. Specifically, a novel heterogeneous trade graph is meticulously constructed from original transactions to explore the complex semantics among different types of entities and relationships. Motivated by continual learning, we present a practical and task-oriented forgetting prevention method to alleviate knowledge forgetting in the context of cross-regional detection. Extensive experiments demonstrate that HTG-CFD promotes performance in both cross-regional and single-regional scenarios.

AAAI Conference 2023 Conference Paper

DPAUC: Differentially Private AUC Computation in Federated Learning

• Jiankai Sun
• Xin Yang
• Yuanshun Yao
• Junyuan Xie
• Di Wu
• Chong Wang

Federated learning (FL) has gained significant attention recently as a privacy-enhancing tool to jointly train a machine learning model by multiple participants. The prior work on FL has mostly studied how to protect label privacy during model training. However, model evaluation in FL might also lead to the potential leakage of private label information. In this work, we propose an evaluation algorithm that can accurately compute the widely used AUC (area under the curve) metric when using the label differential privacy (DP) in FL. Through extensive experiments, we show our algorithms can compute accurate AUCs compared to the ground truth. The code is available at https://github.com/bytedance/fedlearner/tree/master/example/privacy/DPAUC

JBHI Journal 2023 Journal Article

Dual-Input Transformer: An End-to-End Model for Preoperative Assessment of Pathological Complete Response to Neoadjuvant Chemotherapy in Breast Cancer Ultrasonography

• Tong Tong
• Dongyang Li
• Jionghui Gu
• Guo Chen
• Guotao Bai
• Xin Yang
• Kun Wang
• Tianan Jiang

Neoadjuvant chemotherapy (NAC) is the primary method to reduce the burden of tumor and metastasis; in the treatment of breast cancer, it may provide additional opportunities for breast-conserving surgery. Preoperative assessment of pathological complete response (PCR) to NAC is important for developing individualized treatment approaches and predicting patient prognosis. Compared to magnetic resonance imaging (MRI) and mammography, ultrasonography (US) has the advantages of simplicity, flexibility, and real-time imaging. Moreover, it does not require radiation and can provide multi-time acquisition of the tumor during NAC treatment. Recently, deep learning radiomics models based on multi-time-point US images for the prediction of NAC effectiveness have been proposed. To further improve the prediction performance, we carefully designed four supporting modules for our proposed dual-input transformer (DiT): isolated tokens-to-token patch embedding module, shared position embedding, time embedding, and weighted average pooling feature representation modules. The design of each module considers the characteristics of the US images at multiple time points. We validated our model on our retrospective US dataset composed of 484 cases from two centers whose consistency is not sufficiently high. Patients were allocated to training (n = 297), validation (n = 99), and external test (n = 88) sets. The results show that our model can achieve better performance than the Siamese CNN and the standard tokens-to-token vision transformer without using multi-time-point images. The ablation study also proved the effectiveness of each module designed for DiT.

IJCAI Conference 2023 Conference Paper

Dual-view Correlation Hybrid Attention Network for Robust Holistic Mammogram Classification

• Zhiwei Wang
• Junlin Xian
• Kangyi Liu
• Xin Li
• Qiang Li
• Xin Yang

Mammogram image is important for breast cancer screening, and typically obtained in a dual-view form, i. e. , cranio-caudal (CC) and mediolateral oblique (MLO), to provide complementary information for clinical decisions. However, previous methods mostly learn features from the two views independently, which violates the clinical knowledge and ignores the importance of dual-view correlation in the feature learning. In this paper, we propose a dual-view correlation hybrid attention network (DCHA-Net) for robust holistic mammogram classification. Specifically, DCHA-Net is carefully designed to extract and reinvent deep feature maps for the two views, and meanwhile to maximize the underlying correlations between them. A hybrid attention module, consisting of local relation and non-local attention blocks, is proposed to alleviate the spatial misalignment of the paired views in the correlation maximization. A dual-view correlation loss is introduced to maximize the feature similarity between corresponding strip-like regions with equal distance to the chest wall, motivated by the fact that their features represent the same breast tissues, and thus should be highly-correlated with each other. Experimental results on the two public datasets, i. e. , INbreast and CBIS-DDSM, demonstrate that the DCHA-Net can well preserve and maximize feature correlations across views, and thus outperforms previous state-of-the-art methods for classifying a whole mammogram as malignant or not.

JBHI Journal 2023 Journal Article

Geometry-Consistent Adversarial Registration Model for Unsupervised Multi-Modal Medical Image Registration

• Yanxia Liu
• Wenqi Wang
• Yuhong Li
• Haoyu Lai
• Sijuan Huang
• Xin Yang

Deformable multi-modal medical image registration aligns the anatomical structures of different modalities to the same coordinate system through a spatial transformation. Due to the difficulties of collecting ground-truth registration labels, existing methods often adopt the unsupervised multi-modal image registration setting. However, it is hard to design satisfactory metrics to measure the similarity of multi-modal images, which heavily limits the multi-modal registration performance. Moreover, due to the contrast difference of the same organ in multi-modal images, it is difficult to extract and fuse the representations of different modal images. To address the above issues, we propose a novel unsupervised multi-modal adversarial registration framework that takes advantage of image-to-image translation to translate the medical image from one modality to another. In this way, we are able to use the well-defined uni-modal metrics to better train the models. Inside our framework, we propose two improvements to promote accurate registration. First, to avoid the translation network learning spatial deformation, we propose a geometry-consistent training scheme to encourage the translation network to learn the modality mapping solely. Second, we propose a novel semi-shared multi-scale registration network that extracts features of multi-modal images effectively and predicts multi-scale registration fields in an coarse-to-fine manner to accurately register the large deformation area. Extensive experiments on brain and pelvic datasets demonstrate the superiority of the proposed method over existing methods, revealing our framework has great potential in clinical application.

AIIM Journal 2023 Journal Article

Identifying stroke-related quantified evidence from electronic health records in real-world studies

• Lin Yang
• Xiaoshuo Huang
• Jiayang Wang
• Xin Yang
• Lingling Ding
• Zixiao Li
• Jiao Li

Background Stroke is one of the leading causes of death and disability worldwide. The National Institutes of Health Stroke Scale (NIHSS) scores in electronic health records (EHRs), which quantitatively describe patients' neurological deficits in evidence-based treatment, are crucial in stroke-related clinical investigations. However, the free-text format and lack of standardization inhibit their effective use. Automatically extracting the scale scores from the clinical free text so that its potential value in real-world studies is realized has become an important goal. Objective This study aims to develop an automated method to extract scale scores from the free text of EHRs. Methods We propose a two-step pipeline method to identify NIHSS items and numerical scores and validate its feasibility using a freely accessible critical care database: MIMIC-III (Medical Information Mart for Intensive Care III). First, we utilize MIMIC-III to create an annotated corpus. Then, we investigate possible machine learning methods for two subtasks, NIHSS item and score recognition and item-score relation extraction. In the evaluation, we conduct both task-specific and end-to-end evaluations and compare our method with the rule-based method using precision, recall and F1 scores as evaluation metrics. Results We use all available discharge summaries of stroke cases in MIMIC-III. The annotated NIHSS corpus contains 312 cases, 2929 scale items, 2774 scores and 2733 relations. The results show that the best F1-score of our method was 0. 9006, which was attained by combining BERT-BiLSTM-CRF and Random Forest, and it outperformed the rule-based method (F1-score = 0. 8098). In the end-to-end task, our method could successfully recognize the item “1b level of consciousness questions”, the score “1” and their relation “(‘1b level of consciousness questions’, ‘1’, ‘has value’)” from the sentence “1b level of consciousness questions: said name = 1”, while the rule-based method could not. Conclusions The two-step pipeline method we propose is an effective approach to identify NIHSS items, scores and their relations. With its help, clinical investigators can easily retrieve and access structured scale data, thereby supporting stroke-related real-world studies.

NeurIPS Conference 2022 Conference Paper

Biologically Inspired Dynamic Thresholds for Spiking Neural Networks

• Jianchuan Ding
• Bo Dong
• Felix Heide
• Yufei Ding
• Yunduo Zhou
• Baocai Yin
• Xin Yang

The dynamic membrane potential threshold, as one of the essential properties of a biological neuron, is a spontaneous regulation mechanism that maintains neuronal homeostasis, i. e. , the constant overall spiking firing rate of a neuron. As such, the neuron firing rate is regulated by a dynamic spiking threshold, which has been extensively studied in biology. Existing work in the machine learning community does not employ bioinspired spiking threshold schemes. This work aims at bridging this gap by introducing a novel bioinspired dynamic energy-temporal threshold (BDETT) scheme for spiking neural networks (SNNs). The proposed BDETT scheme mirrors two bioplausible observations: a dynamic threshold has 1) a positive correlation with the average membrane potential and 2) a negative correlation with the preceding rate of depolarization. We validate the effectiveness of the proposed BDETT on robot obstacle avoidance and continuous control tasks under both normal conditions and various degraded conditions, including noisy observations, weights, and dynamic environments. We find that the BDETT outperforms existing static and heuristic threshold approaches by significant margins in all tested conditions, and we confirm that the proposed bioinspired dynamic threshold scheme offers homeostasis to SNNs in complex real-world tasks.

JBHI Journal 2022 Journal Article

Cell Localization and Counting Using Direction Field Map

• Yajie Chen
• Dingkang Liang
• Xiang Bai
• Yongchao Xu
• Xin Yang

Automatic cell counting in pathology images is challenging due to blurred boundaries, low-contrast, and overlapping between cells. In this paper, we train a convolutional neural network (CNN) to predict a two-dimensional direction field map and then use it to localize cell individuals for counting. Specifically, we define a direction field on each pixel in the cell regions (obtained by dilating the original annotation in terms of cell centers) as a two-dimensional unit vector pointing from the pixel to its corresponding cell center. Direction field for adjacent pixels in different cells have opposite directions departing from each other, while those in the same cell region have directions pointing to the same center. Such unique property is used to partition overlapped cells for localization and counting. To deal with those blurred boundaries or low contrast cells, we set the direction field of the background pixels to be zeros in the ground-truth generation. Thus, adjacent pixels belonging to cells and background will have an obvious difference in the predicted direction field. To further deal with cells of varying density and overlapping issues, we adopt geometry adaptive (varying) radius for cells of different densities in the generation of ground-truth direction field map, which guides the CNN model to separate cells of different densities and overlapping cells. Extensive experimental results on three widely used datasets ( i. e. , VGG Cell, CRCHistoPhenotype2016, and MBM datasets) demonstrate the effectiveness of the proposed approach.

AAAI Conference 2022 Conference Paper

CPRAL: Collaborative Panoptic-Regional Active Learning for Semantic Segmentation

• Yu Qiao
• Jincheng Zhu
• Chengjiang Long
• Zeyao Zhang
• Yuxin Wang
• Zhenjun Du
• Xin Yang

Acquiring the most representative examples via active learning (AL) can benefit many data-dependent computer vision tasks by minimizing efforts of image-level or pixel-wise annotations. In this paper, we propose a novel Collaborative Panoptic-Regional Active Learning framework (CPRAL) to address the semantic segmentation task. For a small batch of images initially sampled with pixel-wise annotations, we employ panoptic information to initially select unlabeled samples. Considering the class imbalance in the segmentation dataset, we import a Regional Gaussian Attention module (RGA) to achieve semantics-biased selection. The subset is highlighted by vote entropy and then attended by Gaussian kernels to maximize the biased regions. We also propose a Contextual Labels Extension (CLE) to boost regional annotations with contextual attention guidance. With the collaboration of semantics-agnostic panoptic matching and regionbiased selection and extension, our CPRAL can strike a balance between labeling efforts and performance and compromise the semantics distribution. We perform extensive experiments on Cityscapes and BDD10K datasets and show that CPRAL outperforms the cutting-edge methods with impressive results and less labeling proportion.

JBHI Journal 2022 Journal Article

Customized Federated Learning for Multi-Source Decentralized Medical Image Classification

• Jeffry Wicaksana
• Zengqiang Yan
• Xin Yang
• Yang Liu
• Lixin Fan
• Kwang-Ting Cheng

The performance of deep networks for medical image analysis is often constrained by limited medical data, which is privacy-sensitive. Federated learning (FL) alleviates the constraint by allowing different institutions to collaboratively train a federated model without sharing data. However, the federated model is often suboptimal with respect to the characteristics of each client's local data. Instead of training a single global model, we propose Customized FL (CusFL), for which each client iteratively trains a client-specific/private model based on a federated global model aggregated from all private models trained in the immediate previous iteration. Two overarching strategies employed by CusFL lead to its superior performance: 1) the federated model is mainly for feature alignment and thus only consists of feature extraction layers; 2) the federated feature extractor is used to guide the training of each private model. In that way, CusFL allows each client to selectively learn useful knowledge from the federated model to improve its personalized model. We evaluated CusFL on multi-source medical image datasets for the identification of clinically significant prostate cancer and the classification of skin lesions.

JBHI Journal 2022 Journal Article

Joint Landmark and Structure Learning for Automatic Evaluation of Developmental Dysplasia of the Hip

• Xindi Hu
• Limin Wang
• Xin Yang
• Xu Zhou
• Wufeng Xue
• Yan Cao
• Shengfeng Liu
• Yuhao Huang

The ultrasound (US) screening of the infant hip is vital for the early diagnosis of developmental dysplasia of the hip (DDH). The US diagnosis of DDH refers to measuring alpha and beta angles that quantify hip joint development. These two angles are calculated from key anatomical landmarks and structures of the hip. However, this measurement process is not trivial for sonographers and usually requires a thorough understanding of complex anatomical structures. In this study, we propose a multi-task framework to learn the relationships among landmarks and structures jointly and automatically evaluate DDH. Our multi-task networks are equipped with three novel modules. Firstly, we adopt Mask R-CNN as the basic framework to detect and segment key anatomical structures and add one landmark detection branch to form a new multi-task framework. Secondly, we propose a novel shape similarity loss to refine the incomplete anatomical structure prediction robustly and accurately. Thirdly, we further incorporate the landmark-structure consistent prior to ensure the consistency of the bony rim estimated from the segmented structure and the detected landmark. In our experiments, 1231 US images of the infant hip from 632 patients are collected, of which 247 images from 126 patients are tested. The average errors in alpha and beta angles are 2. 221 ${}^{\circ }$ and 2. 899 ${}^{\circ }$. About 93% and 85% estimates of alpha and beta angles have errors less than 5 degrees, respectively. Experimental results demonstrate that the proposed method can accurately and robustly realize the automatic evaluation of DDH, showing great potential for clinical application.

JBHI Journal 2022 Journal Article

Unsupervised Cross-Modality Domain Adaptation Network for X-Ray to CT Registration

• Shiqiang Zheng
• Xin Yang
• Yifan Wang
• Mingyue Ding
• Wenguang Hou

2D/3D registration that achieves high accuracy and real-time computation is one of the enabling technologies for radiotherapy and image-guided surgeries. Recently, the Convolutional Neural Network (CNN) has been explored to significantly improve the accuracy and efficiency of 2D/3D registration. A pair of intraoperative 2-D x-ray images and synthetic data from pre-operative volume are often required to model the nonconvex mappings between registration parameters and image residual. However, a large clinical dataset collection with accurate poses for x-ray images can be very challenging or even impractical, while exclusive training on synthetic data can frequently cause performance degradation when tested on x-rays. Thus, we propose to train a model on source domain (i. e. , synthetic data) to build appearance-pose relationship first and then use an unsupervised cross-modality domain adaptation network (UCMDAN) to adapt the model to target domain (i. e. , X-rays) through adversarial learning. We propose to narrow the significant domain gap by alignment in both pixel and feature space. In particular, the image appearance transformation and domain-invariance feature learning by multiple aspects are conducted synergistically. Extensive experiments on CT and CBCT dataset show that the proposed UCMDAN outperforms the existing state-of-the-art domain adaptation approaches.

ICLR Conference 2021 Conference Paper

CPT: Efficient Deep Neural Network Training via Cyclic Precision

• Yonggan Fu
• Han Guo
• Meng Li 0004
• Xin Yang
• Yining Ding
• Vikas Chandra
• Yingyan Celine Lin

Low-precision deep neural network (DNN) training has gained tremendous attention as reducing precision is one of the most effective knobs for boosting DNNs' training time/energy efficiency. In this paper, we attempt to explore low-precision training from a new perspective as inspired by recent findings in understanding DNN training: we conjecture that DNNs' precision might have a similar effect as the learning rate during DNN training, and advocate dynamic precision along the training trajectory for further boosting the time/energy efficiency of DNN training. Specifically, we propose Cyclic Precision Training (CPT) to cyclically vary the precision between two boundary values which can be identified using a simple precision range test within the first few training epochs. Extensive simulations and ablation studies on five datasets and eleven models demonstrate that CPT's effectiveness is consistent across various models/tasks (including classification and language modeling). Furthermore, through experiments and visualization we show that CPT helps to (1) converge to a wider minima with a lower generalization error and (2) reduce training variance which we believe opens up a new design knob for simultaneously improving the optimization and efficiency of DNN training.

JBHI Journal 2021 Journal Article

Learn Fine-Grained Adaptive Loss for Multiple Anatomical Landmark Detection in Medical Images

• Guang-Quan Zhou
• Juzheng Miao
• Xin Yang
• Rui Li
• En-Ze Huo
• Wenlong Shi
• Yuhao Huang
• Jikuan Qian

Automatic and accurate detection of anatomical landmarks is an essential operation in medical image analysis with a multitude of applications. Recent deep learning methods have improved results by directly encoding the appearance of the captured anatomy with the likelihood maps (i. e. , heatmaps). However, most current solutions overlook another essence of heatmap regression, the objective metric for regressing target heatmaps and rely on hand-crafted heuristics to set the target precision, thus being usually cumbersome and task-specific. In this paper, we propose a novel learning-to-learn framework for landmark detection to optimize the neural network and the target precision simultaneously. The pivot of this work is to leverage the reinforcement learning (RL) framework to search objective metrics for regressing multiple heatmaps dynamically during the training process, thus avoiding setting problem-specific target precision. We also introduce an early-stop strategy for active termination of the RL agent's interaction that adapts the optimal precision for separate targets considering exploration-exploitation tradeoffs. This approach shows better stability in training and improved localization accuracy in inference. Extensive experimental results on two different applications of landmark localization: 1) our in-house prenatal ultrasound (US) dataset and 2) the publicly available dataset of cephalometric X-Ray landmark detection, demonstrate the effectiveness of our proposed method. Our proposed framework is general and shows the potential to improve the efficiency of anatomical landmark detection.

EAAI Journal 2021 Journal Article

Meta-action reliability-based mechanical product optimization design under uncertainty environment

• Yifan Chen
• Yan Ran
• Zhichao Wang
• Xinlong Li
• Xin Yang
• Genbao Zhang

Reliability allocation is an important procedure in the early stage of product design. However, there is not a good design scheme for the reliability allocation of mechanical products. Firstly, the reliability allocation of mechanical products is still at the system level or component level, which makes it impossible to obtain more detailed reliability input parameters. Secondly, to simplify the allocation process, the interactions among allocation objects are usually ignored, which makes the result of reliability allocation not optimistic. Thirdly, the uncertainty of reliability-based mechanical product optimization has not been dealt with effectively. Therefore, this paper proposes a multi-criteria decision-making (MCDM) for mechanical product optimization design based on meta-action reliability, which integrates decision-making trial and evaluation laboratory (DEMATEL), uncertain linguistic ordered weighted averaging (ULOWA) operator, and preference ranking organization method for enrichment evaluations (PROMETHEE) II method. Also, the interval number is used to deal with the uncertainty flexibly in the decision-making process. Finally, the application and feasibility of the proposed method are illustrated with the automatic pallet changer (APC) of a CNC machine tool as an example. This method provides guidance for the reliability optimization design of mechanical products.

JBHI Journal 2021 Journal Article

Multi-Focus Network to Decode Imaging Phenotype for Overall Survival Prediction of Gastric Cancer Patients

• Liwen Zhang
• Di Dong
• Lianzhen Zhong
• Cong Li
• Chaoen Hu
• Xin Yang
• Zaiyi Liu
• Rongpin Wang

Gastric cancer (GC) is the third leading cause of cancer-associated deaths globally. Accurate risk prediction of the overall survival (OS) for GC patients shows significant prognostic value, which helps identify and classify patients into different risk groups to benefit from personalized treatment. Many methods based on machine learning algorithms have been widely explored to predict the risk of OS. However, the accuracy of risk prediction has been limited and remains a challenge with existing methods. Few studies have proposed a framework and pay attention to the low-level and high-level features separately for the risk prediction of OS based on computed tomography images of GC patients. To achieve high accuracy, we propose a multi-focus fusion convolutional neural network. The network focuses on low-level and high-level features, where a subnet to focus on lower-level features and the other enhanced subnet with lateral connection to focus on higher-level semantic features. Three independent datasets of 640 GC patients are used to assess our method. Our proposed network is evaluated by metrics of the concordance index and hazard ratio. Our network outperforms state-of-the-art methods with the highest concordance index and hazard ratio in independent validation and test sets. Our results prove that our architecture can unify the separate low-level and high-level features into a single framework, and can be a powerful method for accurate risk prediction of OS.

JBHI Journal 2021 Journal Article

Variation-Aware Federated Learning With Multi-Source Decentralized Medical Image Data

• Zengqiang Yan
• Jeffry Wicaksana
• Zhiwei Wang
• Xin Yang
• Kwang-Ting Cheng

Privacy concerns make it infeasible to construct a large medical image dataset by fusing small ones from different sources/institutions. Therefore, federated learning (FL) becomes a promising technique to learn from multi-source decentralized data with privacy preservation. However, the cross-client variation problem in medical image data would be the bottleneck in practice. In this paper, we propose a variation-aware federated learning (VAFL) framework, where the variations among clients are minimized by transforming the images of all clients onto a common image space. We first select one client with the lowest data complexity to define the target image space and synthesize a collection of images through a privacy-preserving generative adversarial network, called PPWGAN-GP. Then, a subset of those synthesized images, which effectively capture the characteristics of the raw images and are sufficiently distinct from any raw image, is automatically selected for sharing with other clients. For each client, a modified CycleGAN is applied to translate its raw images to the target image space defined by the shared synthesized images. In this way, the cross-client variation problem is addressed with privacy preservation. We apply the framework for automated classification of clinically significant prostate cancer and evaluate it using multi-source decentralized apparent diffusion coefficient (ADC) image data. Experimental results demonstrate that the proposed VAFL framework stably outperforms the current horizontal FL framework. As VAFL is independent of deep learning architectures for classification, we believe that the proposed framework is widely applicable to other medical image classification tasks.

EAAI Journal 2020 Journal Article

An extended MULTIMOORA method based on OWGA operator and Choquet integral for risk prioritization identification of failure modes

• Yifan Chen
• Yan Ran
• Zhichao Wang
• Xinlong Li
• Xin Yang
• Genbao Zhang

Failure mode and effects analysis (FMEA) is one of the important methods for risk analysis, and has been used in various fields to improve the reliability of systems. However, in the fuzzy environment, the aggregation, weight calculation, risk evaluation and prioritization of evaluation information limit its wide application. Therefore, this paper proposes an extended multi-objective optimization by ratio analysis plus full multiplicative form (MULTIMOORA) method based on the ordered weighted geometric averaging (OWGA) operator and Choquet integral for FMEA. Firstly, trapezoidal fuzzy numbers (TrFNs) are used for describing the fuzzy ratings of failure modes. Secondly, considering the uncertainty preferences of decision-makers, the OWGA operator is adopted for aggregating the evaluation information given by multiple decision-makers. Thirdly, the TrFNs ranking method based on the relative preference relation is combined with Choquet integral to model the interactions among risk factors, thus capturing their importance weights. On this basis, an extended MULTIMOORA method is proposed to determine the risk priority of failure modes. Finally, two case studies are provided to illustrate the effectiveness and practicability of the proposed method.

JBHI Journal 2020 Journal Article

Bi-Modality Medical Image Synthesis Using Semi-Supervised Sequential Generative Adversarial Networks

• Xin Yang
• Yi Lin
• Zhiwei Wang
• Xin Li
• Kwang-Ting Cheng

In this paper, we propose a bi-modality medical image synthesis approach based on sequential generative adversarial network (GAN) and semi-supervised learning. Our approach consists of two generative modules that synthesize images of the two modalities in a sequential order. A method for measuring the synthesis complexity is proposed to automatically determine the synthesis order in our sequential GAN. Images of the modality with a lower complexity are synthesized first, and the counterparts with a higher complexity are generated later. Our sequential GAN is trained end-to-end in a semi-supervised manner. In supervised training, the joint distribution of bi-modality images are learned from real paired images of the two modalities by explicitly minimizing the reconstruction losses between the real and synthetic images. To avoid overfitting limited training images, in unsupervised training, the marginal distribution of each modality is learned based on unpaired images by minimizing the Wasserstein distance between the distributions of real and fake images. We comprehensively evaluate the proposed model using two synthesis tasks based on three types of evaluate metrics and user studies. Visual and quantitative results demonstrate the superiority of our method to the state-of-the-art methods, and reasonable visual quality and clinical significance. Code is made publicly available at https://github.com/hust-linyi/Multimodal-Medical-Image-Synthesis.

JBHI Journal 2020 Journal Article

CR-Unet: A Composite Network for Ovary and Follicle Segmentation in Ultrasound Images

• Haoming Li
• Jinghui Fang
• Shengfeng Liu
• Xiaowen Liang
• Xin Yang
• Zixin Mai
• Manh The Van
• Tianfu Wang

Transvaginal ultrasound (TVUS) is widely used in infertility treatment. The size and shape of the ovary and follicles must be measured manually for assessing their physiological status by sonographers. However, this process is extremely time-consuming and operator-dependent. In this study, we propose a novel composite network, namely CR-Unet, to simultaneously segment the ovary and follicles in TVUS. The CR-Unet incorporates the spatial recurrent neural network (RNN) into a plain U-Net. It can effectively learn multi-scale and long-range spatial contexts to combat the challenges of this task, such as the poor image quality, low contrast, boundary ambiguity, and complex anatomy shapes. We further adopt deep supervision strategy to make model training more effective and efficient. In addition, self-supervision is employed to iteratively refine the segmentation results. Experiments on 3204 TVUS images from 219 patients demonstrate the proposed method achieved the best segmentation performance compared to other state-of-the-art methods for both the ovary and follicles, with a Dice Similarity Coefficient (DSC) of 0. 912 and 0. 858, respectively.

JBHI Journal 2019 Journal Article

A Three-Stage Deep Learning Model for Accurate Retinal Vessel Segmentation

• Zengqiang Yan
• Xin Yang
• Kwang-Ting Cheng

Automatic retinal vessel segmentation is a fundamental step in the diagnosis of eye-related diseases, in which both thick vessels and thin vessels are important features for symptom detection. All existing deep learning models attempt to segment both types of vessels simultaneously by using a unified pixel-wise loss that treats all vessel pixels with equal importance. Due to the highly imbalanced ratio between thick vessels and thin vessels (namely the majority of vessel pixels belong to thick vessels), the pixelwise loss would be dominantly guided by thick vessels and relatively little influence comes from thin vessels, often leading to low segmentation accuracy for thin vessels. To address the imbalance problem, in this paper, we explore to segment thick vessels and thin vessels separately by proposing a three-stage deep learning model. The vessel segmentation task is divided into three stages, namely thick vessel segmentation, thin vessel segmentation, and vessel fusion. As better discriminative features could be learned for separate segmentation of thick vessels and thin vessels, this process minimizes the negative influence caused by their highly imbalanced ratio. The final vessel fusion stage refines the results by further identifying nonvessel pixels and improving the overall vessel thickness consistency. The experiments on public datasets DRIVE, STARE, and CHASE_DB1 clearly demonstrate that the proposed three-stage deep learning model outperforms the current state-of-the-art vessel segmentation methods.

NeurIPS Conference 2019 Conference Paper

Total Least Squares Regression in Input Sparsity Time

• Huaian Diao
• Zhao Song
• David Woodruff
• Xin Yang

In the total least squares problem, one is given an $m \times n$ matrix $A$, and an $m \times d$ matrix $B$, and one seeks to ``correct'' both $A$ and $B$, obtaining matrices $\hat{A}$ and $\hat{B}$, so that there exists an $X$ satisfying the equation $\hat{A}X = \hat{B}$. Typically the problem is overconstrained, meaning that $m \gg \max(n, d)$. The cost of the solution $\hat{A}, \hat{B}$ is given by $\|A-\hat{A}\|_F^2 + \|B - \hat{B}\|_F^2$. We give an algorithm for finding a solution $X$ to the linear system $\hat{A}X=\hat{B}$ for which the cost $\|A-\hat{A}\|_F^2 + \|B-\hat{B}\|_F^2$ is at most a multiplicative $(1+\epsilon)$ factor times the optimal cost, up to an additive error $\eta$ that may be an arbitrarily small function of $n$. Importantly, our running time is $\tilde{O}(\nnz(A) + \nnz(B)) + \poly(n/\epsilon) \cdot d$, where for a matrix $C$, $\nnz(C)$ denotes its number of non-zero entries. Importantly, our running time does not directly depend on the large parameter $m$. As total least squares regression is known to be solvable via low rank approximation, a natural approach is to invoke fast algorithms for approximate low rank approximation, obtaining matrices $\hat{A}$ and $\hat{B}$ from this low rank approximation, and then solving for $X$ so that $\hat{A}X = \hat{B}$. However, existing algorithms do not apply since in total least squares the rank of the low rank approximation needs to be $n$, and so the running time of known methods would be at least $mn^2$. In contrast, we are able to achieve a much faster running time for finding $X$ by never explicitly forming the equation $\hat{A} X = \hat{B}$, but instead solving for an $X$ which is a solution to an implicit such equation. Finally, we generalize our algorithm to the total least squares problem with regularization.

NeurIPS Conference 2018 Conference Paper

Active Matting

• Xin Yang
• Ke Xu
• Shaozhe Chen
• Shengfeng He
• Baocai Yin Yin
• Rynson Lau

Image matting is an ill-posed problem. It requires a user input trimap or some strokes to obtain an alpha matte of the foreground object. A fine user input is essential to obtain a good result, which is either time consuming or suitable for experienced users who know where to place the strokes. In this paper, we explore the intrinsic relationship between the user input and the matting algorithm to address the problem of where and when the user should provide the input. Our aim is to discover the most informative sequence of regions for user input in order to produce a good alpha matte with minimum labeling efforts. To this end, we propose an active matting method with recurrent reinforcement learning. The proposed framework involves human in the loop by sequentially detecting informative regions for trivial human judgement. Comparing to traditional matting algorithms, the proposed framework requires much less efforts, and can produce satisfactory results with just 10 regions. Through extensive experiments, we show that the proposed model reduces user efforts significantly and achieves comparable performance to dense trimaps in a user-friendly manner. We further show that the learned informative knowledge can be generalized across different matting algorithms.

IJCAI Conference 2018 Conference Paper

Active Object Reconstruction Using a Guided View Planner

• Xin Yang
• Yuanbo Wang
• Yaru Wang
• Baocai Yin
• Qiang Zhang
• Xiaopeng Wei
• Hongbo Fu

Inspired by the recent advance of image-based object reconstruction using deep learning, we present an active reconstruction model using a guided view planner. We aim to reconstruct a 3D model using images observed from a planned sequence of informative and discriminative views. But where are such informative and discriminative views around an object? To address this we propose a unified model for view planning and object reconstruction, which is utilized to learn a guided information acquisition model and to aggregate information from a sequence of images for reconstruction. Experiments show that our model (1) increases our reconstruction accuracy with an increasing number of views (2) and generally predicts a more informative sequence of views for object reconstruction compared to other alternative methods.

AAAI Conference 2017 Conference Paper

Fine-Grained Recurrent Neural Networks for Automatic Prostate Segmentation in Ultrasound Images

• Xin Yang
• Lequan Yu
• Lingyun Wu
• Yi Wang
• Dong Ni
• Jing Qin
• Pheng-Ann Heng

Boundary incompleteness raises great challenges to automatic prostate segmentation in ultrasound images. Shape prior can provide strong guidance in estimating the missing boundary, but traditional shape models often suffer from hand-crafted descriptors and local information loss in the fitting procedure. In this paper, we attempt to address those issues with a novel framework. The proposed framework can seamlessly integrate feature extraction and shape prior exploring, and estimate the complete boundary with a sequential manner. Our framework is composed of three key modules. Firstly, we serialize the static 2D prostate ultrasound images into dynamic sequences and then predict prostate shapes by sequentially exploring shape priors. Intuitively, we propose to learn the shape prior with the biologically plausible Recurrent Neural Networks (RNNs). This module is corroborated to be effective in dealing with the boundary incompleteness. Secondly, to alleviate the bias caused by different serialization manners, we propose a multi-view fusion strategy to merge shape predictions obtained from different perspectives. Thirdly, we further implant the RNN core into a multiscale Auto-Context scheme to successively refine the details of the shape prediction map. With extensive validation on challenging prostate ultrasound images, our framework bridges severe boundary incompleteness and achieves the best performance in prostate boundary delineation when compared with several advanced methods. Additionally, our approach is general and can be extended to other medical image segmentation tasks, where boundary incompleteness is one of the main challenges.

AAAI Conference 2017 Conference Paper

Volumetric ConvNets with Mixed Residual Connections for Automated Prostate Segmentation from 3D MR Images

• Lequan Yu
• Xin Yang
• Hao Chen
• Jing Qin
• Pheng Ann Heng

Automated prostate segmentation from 3D MR images is very challenging due to large variations of prostate shape and indistinct prostate boundaries. We propose a novel volumetric convolutional neural network (ConvNet) with mixed residual connections to cope with this challenging problem. Compared with previous methods, our volumetric ConvNet has two compelling advantages. First, it is implemented in a 3D manner and can fully exploit the 3D spatial contextual information of input data to perform efficient, precise and volumeto-volume prediction. Second and more important, the novel combination of residual connections (i. e. , long and short) can greatly improve the training efficiency and discriminative capability of our network by enhancing the information propagation within the ConvNet both locally and globally. While the forward propagation of location information can improve the segmentation accuracy, the smooth backward propagation of gradient flow can accelerate the convergence speed and enhance the discrimination capability. Extensive experiments on the open MICCAI PROMISE12 challenge dataset corroborated the effectiveness of the proposed volumetric ConvNet with mixed residual connections. Our method ranked the first in the challenge, outperforming other competitors by a large margin with respect to most of evaluation metrics. The proposed volumetric ConvNet is general enough and can be easily extended to other medical image analysis tasks, especially ones with limited training data.

JBHI Journal 2015 Journal Article

Standard Plane Localization in Fetal Ultrasound via Domain Transferred Deep Neural Networks

• Hao Chen
• Dong Ni
• Jing Qin
• Shengli Li
• Xin Yang
• Tianfu Wang
• Pheng Ann Heng

Automatic localization of the standard plane containing complicated anatomical structures in ultrasound (US) videos remains a challenging problem. In this paper, we present a learning-based approach to locate the fetal abdominal standard plane (FASP) in US videos by constructing a domain transferred deep convolutional neural network (CNN). Compared with previous works based on low-level features, our approach is able to represent the complicated appearance of the FASP and hence achieve better classification performance. More importantly, in order to reduce the overfitting problem caused by the small amount of training samples, we propose a transfer learning strategy, which transfers the knowledge in the low layers of a base CNN trained from a large database of natural images to our task-specific CNN. Extensive experiments demonstrate that our approach outperforms the state-of-the-art method for the FASP localization as well as the CNN only trained on the limited US training samples. The proposed approach can be easily extended to other similar medical image computing problems, which often suffer from the insufficient training samples when exploiting the deep CNN to represent high-level features.

JBHI Journal 2013 Journal Article

ACM-Based Automatic Liver Segmentation From 3-D CT Images by Combining Multiple Atlases and Improved Mean-Shift Techniques

• Hongwei Ji
• Jiangping He
• Xin Yang
• Rudi Deklerck
• Jan Cornelis

In this paper, we present an autocontext model (ACM)-based automatic liver segmentation algorithm, which combines ACM, multiatlases, and mean-shift techniques to segment liver from 3-D CT images. Our algorithm is a learning-based method and can be divided into two stages. At the first stage, i. e. , the training stage, ACM is performed to learn a sequence of classifiers in each atlas space (based on each atlas and other aligned atlases). With the use of multiple atlases, multiple sequences of ACM-based classifiers are obtained. At the second stage, i. e. , the segmentation stage, the test image will be segmented in each atlas space by applying each sequence of ACM-based classifiers. The final segmentation result will be obtained by fusing segmentation results from all atlas spaces via a multi-classifier fusion technique. Specially, in order to speed up segmentation, given a test image, we first use an improved mean-shift algorithm to perform oversegmentation and then implement the region-based image labeling instead of the original inefficient pixel-based image labeling. The proposed method is evaluated on the datasets of MICCAI 2007 liver segmentation challenge. The experimental results show that the average volume overlap error and the average surface distance achieved by our method are 8. 3% and 1. 5 m, respectively, which are comparable to the results reported in the existing state-of-the-art work on liver segmentation.

ICML Conference 2006 Conference Paper

Semi-supervised nonlinear dimensionality reduction

• Xin Yang
• Haoying Fu
• Hongyuan Zha
• Jesse L. Barlow

ICRA Conference 2003 Conference Paper

A common reference object concept to cooperative transportation

• Xin Yang
• Keigo Watanabe
• Kiyotaka Izumi
• Kazuo Kiguchi

A concept called "common reference object" is proposed for cooperative transportation and decentralized system consisting of multiple noholonomic mobile robots is constructed to demonstrate the present concept. In this system, one agent acts as the leader, which is able to plan and manipulate the omnidirectional motion of the object. Other agents referred to, as followers are equipped with compliance arms and cooperatively transport the object by keeping a constant position relative to the object. During transportation operation, the leader robot can not only plan the motion of the object but also broadcast the local velocity of the object to other agents. Then, each follower receives such information and generates its own velocity in the local coordinate using a mapping process. In this paper, neural network (NN) and genetic algorithm (GA) are tested in identifying the mapping process. Simulation results show a good performance of the present system.