Dong Xu

JBHI Journal 2026 Journal Article

Applications of Large Language Models and Prompt Optimization for Knowledge Extraction From Biological Pathway Figures

• Muhammad Azam
• Shuai Zeng
• Hasanain Aldihis
• Mihail Popescu
• Dong Xu

Recent developments in Large Language Models (LLMs) have demonstrated remarkable capabilities for image comprehension. This study aims to automate and enhance the extraction of gene interactions from biological pathway images by integrating LLMs and a Genetic Algorithm (GA). A dataset of 200 tumor signaling pathway figures from the recent biological literature was employed to assess the performance of four AI chatbots: GPT-4oV, Claude-3. 5V, Gemini-1. 5V, and Llama-3. 2V, with GA used to optimize prompts for each model. Model performance was evaluated on both directional and non-directional gene relationship extraction. GA-optimized prompts significantly improved extraction accuracies across all LLMs, with GPT-4oV achieving an F1-score of 0. 645 (±0. 055) and Llama-3. 2V achieving an F1-score of 0. 616 (±0. 068). For non-directional interactions, GPT-4oV outperformed other models, reaching a precision of 0. 805, a recall of 0. 695, and an F1-score of 0. 757, followed by Llama-3. 2V and Claude-3. 5V with F1-scores of 0. 702 and 0. 697, respectively, while Gemini-1. 5V lagged with 0. 612. In directional interaction predictions, all models performed lower, with GPT-4oV leading at 0. 687 F1-score, followed by Llama-3. 2V at 0. 656, Claude-3. 5V at 0. 641, and Gemini-1. 5V at 0. 573. While these results demonstrate substantial improvements over traditional OCR-based approaches, further advances in model accuracy and explainability are needed for widespread adoption in critical biomedical applications. Nevertheless, these findings provide a valuable benchmark for the research community and a foundation for future development of specialized, fine-tuned models and scalable multimodal AI frameworks in biomedical data analysis.

JBHI Journal 2026 Journal Article

BreastUS-Net: An Attention-Guided Dual-Branch Network With Feature Fusion for Fine-Grained Breast Tumor Classification in Ultrasound Imaging

• Sohaib Asif
• Di Ou
• Fazal Hadi
• Yuqi Yan
• Enyu Wang
• Yang Zhang
• Dong Xu

Despite advances in deep learning (DL) and computer vision, breast cancer (BC) detection via ultrasound remains challenging. Existing methods often focus on single tasks using complex pipelines and publicly available datasets, limiting clinical applicability. To address this, we propose BreastUS-Net—a novel architecture for hierarchical BC classification using diverse datasets. Our approach uses a dual-branch MobileNet architecture with fine-tuned and frozen layers to capture both task-specific and general features, eliminating manual feature extraction. These features are then fused to create a comprehensive representation, which is subsequently aggregated and refined. The aggregation step merges the outputs from both branches, while the refinement module reduces complexity, highlights relevant patterns, and mitigates overfitting to improve generalization. Additionally, we integrate a multihead self-attention (MHSA) block to highlight diagnostically significant regions in ultrasound images, enhancing both accuracy and robustness. Finally, the orthogonal softmax layer (OSL) boosts discriminative power by enforcing orthogonality among weight vectors, reducing parameter co-adaptation and enabling more effective optimization. We used six diverse datasets from multiple centers, including: a large Zhejiang Cancer Hospital set (2, 171 images), public BUSI dataset (780 images), external test sets from Yunnan Cancer Hospital (351 images) and Sir Run Run Shaw Hospitals (365 images), fibroadenoma (FA) vs. phyllodes tumor (PT) classification, and a PT grading dataset. We use explainable AI (XAI) techniques—Grad-CAM, SHAP, and saliency maps—to enhance trust in breast ultrasound predictions. Our model achieves state-of-the-art performance, with accuracies of 94. 48% on a clinical dataset and 94. 23% on the BUSI dataset, highlighting its potential to improve BC diagnosis and personalized treatment.

AAAI Conference 2026 Conference Paper

Learning Diffusion Policy from Primitive Skills for Robot Manipulation

• Zhihao Gu
• Ming Yang
• Difan Zou
• Dong Xu

Diffusion policies have recently shown great promise for generating actions in robotic manipulation. However, existing approaches often rely on global instructions to produce short-term control signals, which can result in misalignment in action generation. We conjecture that the primitive skills, referred to as fine-grained, short-horizon manipulations, such as "move up" and "open the gripper", provide a more intuitive and effective interface for robot learning. To bridge this gap, we propose SDP, a skill-conditioned diffusion policy that integrates interpretable skill learning with conditional action planning. SDP abstracts eight reusable primitive skills across tasks and employs a vision-language model to extract discrete representations from visual observations and language instructions. Based on the representations, a lightweight router network is designed to assign a desired primitive skill for each state, which helps construct a single-skill policy to generate skill-aligned actions. By decomposing complex tasks into a sequence of primitive skills and selecting a single-skill policy, the proposed SDP ensures skill-consistent behavior across diverse tasks. Extensive experiments on two challenging simulation benchmarks and real-world robot deployments demonstrate that SDP consistently outperforms state-of-the-art methods, providing a new paradigm for skill-based robot learning with diffusion policies.

EAAI Journal 2026 Journal Article

Semi-supervised graph neural networks for fault diagnosis in marine machinery

• Chuyue Lou
• M. Amine Atoui
• Xiaomin Zhang
• Dong Xu
• Haijun Zhong

Recently, fault diagnosis methods for marine machinery systems based on deep learning models have attracted considerable attention. However, following the closed set assumption, most existing research suffer from performance failure when faced with unknown fault types (i. e. , out-of-distribution or open-set observations not present during training), which presents a serious challenge to its industrial deployment. The performance of current diagnosis methods for open set scenarios still has significant limitations. By relying on features optimized only for known classes and fixed detection thresholds, they tend to fail when unknown classes exhibit feature similarity to known ones. To address this challenge, this paper proposes a semi-supervised open-set fault diagnosis (SOFD) framework that enhances and extends the applicability of deep learning models in open-set fault diagnosis scenarios. The framework includes a reliability subset construction process, which uses a multi-layer fusion feature representation extracted by a supervised feature learning model to select an unlabeled test subset. The labeled training set and pseudo-labeled test subset are then fed into a semi-supervised diagnosis model to learn discriminative features for each class, enabling accurate classification of known faults and effective detection of unknown samples. Experimental results on a public maritime benchmark dataset demonstrate the effectiveness and superiority of the proposed SOFD framework.

NeurIPS Conference 2025 Conference Paper

CAD-Coder: Text-to-CAD Generation with Chain-of-Thought and Geometric Reward

• Yandong Guan
• Xilin Wang
• XiMing Xing
• Jing Zhang
• Dong Xu
• Qian Yu

In this work, we introduce CAD-Coder, a novel framework that reformulates text-to-CAD as the generation of CadQuery scripts—a Python-based, parametric CAD language. This representation enables direct geometric validation, a richer modeling vocabulary, and seamless integration with existing LLMs. To further enhance code validity and geometric fidelity, we propose a two-stage learning pipeline: (1) supervised fine-tuning on paired text–CadQuery data, and (2) reinforcement learning with Group Reward Policy Optimization (GRPO), guided by a CAD-specific reward comprising both a geometric reward (Chamfer Distance) and a format reward. We also introduce a chain-of-thought (CoT) planning process to improve model reasoning, and construct a large-scale, high-quality dataset of 110K text–CadQuery–3D model triplets and 1. 5K CoT samples via an automated pipeline. Extensive experiments demonstrate that CAD-Coder enables LLMs to generate diverse, valid, and complex CAD models directly from natural language, advancing the state of the art of text-to-CAD generation and geometric reasoning.

JAIR Journal 2025 Journal Article

ConSCompF: Consistency-focused Similarity Comparison Framework for Generative Large Language Models

• Alexey Karev
• Dong Xu

Large Language Models (LLM) are one of the most important discoveries in machine learning in recent years. LLM-based artificial intelligence (AI) assistants, such as ChatGPT, have consistently attracted attention from researchers, investors, and the general public, driving the rapid growth of this industry. With dozens of new LLMs released every month, it becomes quite challenging to differentiate between them, thereby creating a demand for new LLM comparison methods. In this research, the Consistency-focused Similarity Comparison Framework (ConSCompF) for generative large language models is proposed. It compares texts generated by two LLMs and produces a similarity score, indicating the overall degree of similarity between their responses. The main advantage of this framework is that it can operate on a small number of unlabeled data, such as chatbot instruction prompts, and does not require LLM developers to disclose any information about their product. To evaluate the efficacy of ConSCompF, two experiments aimed at identifying similarities between multiple LLMs are conducted. Additionally, these experiments examine the correlation between the similarity scores generated by ConSCompF and the differences in outputs produced by other benchmarking techniques, such as ROUGE-L. Finally, a series of few-shot LLM comparison experiments is conducted to evaluate the performance of ConSCompF in a few-shot LLM comparison scenario. The proposed framework can be used for calculating similarity matrices of multiple LLMs, which can be effectively visualized using principal component analysis (PCA). The outputs of ConSCompF may provide useful insights into data that might have been used during LLM training and help detect potential investment fraud attempts.

ICLR Conference 2025 Conference Paper

Improving Long-Text Alignment for Text-to-Image Diffusion Models

• Luping Liu
• Chao Du
• Tianyu Pang
• Zehan Wang 0001
• Chongxuan Li
• Dong Xu

The rapid advancement of text-to-image (T2I) diffusion models has enabled them to generate unprecedented results from given texts. However, as text inputs become longer, existing encoding methods like CLIP face limitations, and aligning the generated images with long texts becomes challenging. To tackle these issues, we propose LongAlign, which includes a segment-level encoding method for processing long texts and a decomposed preference optimization method for effective alignment training. For segment-level encoding, long texts are divided into multiple segments and processed separately. This method overcomes the maximum input length limits of pretrained encoding models. For preference optimization, we provide decomposed CLIP-based preference models to fine-tune diffusion models. Specifically, to utilize CLIP-based preference models for T2I alignment, we delve into their scoring mechanisms and find that the preference scores can be decomposed into two components: a text-relevant part that measures T2I alignment and a text-irrelevant part that assesses other visual aspects of human preference. Additionally, we find that the text-irrelevant part contributes to a common overfitting problem during fine-tuning. To address this, we propose a reweighting strategy that assigns different weights to these two components, thereby reducing overfitting and enhancing alignment. After fine-tuning $512 \\times 512$ Stable Diffusion (SD) v1.5 for about 20 hours using our method, the fine-tuned SD outperforms stronger foundation models in T2I alignment, such as PixArt-$\\alpha$ and Kandinsky v2.2. The code is available at https://github.com/luping-liu/LongAlign.

EAAI Journal 2025 Journal Article

OralTransNet: A novel hybrid model integrating transformer attention and CNN features for accurate diagnosis of mouth and oral diseases

• Sohaib Asif
• Vicky Yang Wang
• Dong Xu

The rising prevalence of mouth and oral diseases (MOD), including gum disease and oral cancer, presents a significant global health challenge. Early detection is crucial for effective intervention. However, existing models often rely on complex preprocessing, computationally expensive operations, and specialized resources, leading to inefficiency and limited practicality. This paper presents a novel lightweight hybrid model that combines the local feature extraction strengths of CNNs with the global contextual power of Transformer attention mechanisms, contributing to the advancement of artificial intelligence (AI) in medical image analysis. The proposed architecture integrates the local feature extraction efficiency of convolutional neural networks (CNNs) with the global context modeling strength of Transformers. This combination enables the model to effectively capture both fine-grained details and broader spatial patterns, while maintaining low computational complexity. By leveraging CNNs' weight-sharing properties for efficient feature extraction and Transformers' ability to model global patterns, the proposed model performs well across datasets of varying sizes and complexities. Its lightweight design emphasizes efficiency, with fewer parameters, reduced floating-point operations (FLOPs), and shorter inference times, making it ideal for real-time AI applications, particularly in resource-constrained environments. The proposed model is also well-suited for deployment on mobile devices and in regions with limited medical infrastructure, providing a scalable solution for early diagnosis in diverse healthcare settings. In the context of medical engineering, the proposed model is applied to the automated detection of mouth and oral diseases (MOD) using both clinical and histopathological images. This approach aims to enhance diagnostic capabilities in resource-constrained clinical environments. The model is rigorously evaluated on three datasets: the MOD dataset (5143 images, 7 classes), the Oral Cancer dataset (241 images, 2 classes), and the Histopathological Oral Cancer dataset (5192 images, 2 classes). The proposed model achieves accuracies of 99. 03 %, 97. 83 %, and 94. 23 %, respectively—surpassing several state-of-the-art (SOTA) models. Its strong performance, lightweight design, and enhanced interpretability position it as a practical and scalable solution for early and reliable oral disease detection in diverse clinical settings.

NeurIPS Conference 2025 Conference Paper

SAFEx: Analyzing Vulnerabilities of MoE-Based LLMs via Stable Safety-critical Expert Identification

• Zhenglin Lai
• Mengyao Liao
• Bingzhe Wu
• Dong Xu
• Zebin Zhao
• Zhihang Yuan
• Chao Fan
• Jianqiang Li

Large language models with Mixture-of-Experts (MoE) architectures achieve efficiency and scalability, yet their routing mechanisms introduce safety alignment challenges insufficiently addressed by techniques developed for dense models. In this work, the MoE-specific safety risk of positional vulnerability—that safety-aligned behaviors rely on specific expert modules—is formalized and systematically analyzed. An analytical framework, SAFEx, is presented to robustly identify, characterize, and validate safety-critical experts via a stability-based expert selection procedure, and to decompose them into two functional groups: the Harmful Content Detection Group (HCDG), which specializes in identifying and recognizing harmful content within user inputs, and the Harmful Response Control Group (HRCG), which specializes in controlling and enforcing model behaviors to generate appropriate safety responses. Expert-level interventions are conducted to probe causality and to test mitigation. Targeted masking of SAFEx-selected experts reveals that safety behavior is highly concentrated. On Qwen3-30B-A3B, configured with 48 MoE-FFN layers and 128 experts per layer under top-8 routing (48×128=6, 144 experts in total), disabling 12 selected experts reduces the refusal rate by 22%. In addition, lightweight adaptation is performed using LoRA under three configurations—the HRCG, the union of HCDG and HRCG, and all experts—and the resulting updates are composed through negative weight merging targeted at the HRCG, leading to improved refusal under adversarial prompts without full-model retraining. These results establish positional vulnerability as a distinct MoE-specific safety challenge and provide a practical, compute-efficient pathway for expert-level safety interventions within routed architectures.

IROS Conference 2025 Conference Paper

Scalable MARL for Cooperative Exploration with Dynamic Robot Populations via Graph-Based Information Aggregation

• Xiaoqi Ren
• Guanglong Du
• Zhuoyao Wang 0001
• Dong Xu
• Xueqian Wang
• Quanlong Guan
• Xiaojian Qiu

This study addresses the challenge of multi-robot cooperative exploration under limited local observations in environments with dynamic robot populations. To achieve efficient area coverage within constrained timeframes, we propose the Multi-Robot Informative Planner (MIP), a novel reinforcement learning (RL)-based planning module. The core component of MIP is the Neighborhood Information Aggregator, which employs a graph neural network (GNN) to integrate local neighborhood information for each robot. Our design enhances sample efficiency by minimizing information requirements while ensuring scalability across environments with varying robot numbers. To generate high-quality, expressive neighborhood feature representations, we utilize Graphical Mutual Information (GMI) to maximize the correlation between neighboring robots’ input features and their high-level hidden representations. Furthermore, MIP incorporates the Spatial-Neighborhood Transformer, which captures spatial features and inter-robot interactions through spatial self-attention mechanisms. These components collectively form the Multi-Robot Neural Informative Mapping (MRNIM) framework, outperforming traditional benchmarks in Habitat simulator.

NeurIPS Conference 2025 Conference Paper

TCM-Ladder: A Benchmark for Multimodal Question Answering on Traditional Chinese Medicine

• Jiacheng Xie
• Yang Yu
• Ziyang Zhang
• Shuai Zeng
• Jiaxuan He
• Ayush Vasireddy
• Xiaoting tang
• Congyu Guo

Traditional Chinese Medicine (TCM), as an effective alternative medicine, has been receiving increasing attention. In recent years, the rapid development of large language models (LLMs) tailored for TCM has highlighted the urgent need for an objective and comprehensive evaluation framework to assess their performance on real-world tasks. However, existing evaluation datasets are limited in scope and primarily text-based, lacking a unified and standardized multimodal question-answering (QA) benchmark. To address this issue, we introduce TCM-Ladder, the first comprehensive multimodal QA dataset specifically designed for evaluating large TCM language models. The dataset covers multiple core disciplines of TCM, including fundamental theory, diagnostics, herbal formulas, internal medicine, surgery, pharmacognosy, and pediatrics. In addition to textual content, TCM-Ladder incorporates various modalities such as images and videos. The dataset was constructed using a combination of automated and manual filtering processes and comprises over 52, 000 questions. These questions include single-choice, multiple-choice, fill-in-the-blank, diagnostic dialogue, and visual comprehension tasks. We trained a reasoning model on TCM-Ladder and conducted comparative experiments against nine state-of-the-art general domain and five leading TCM-specific LLMs to evaluate their performance on the dataset. Moreover, we propose Ladder-Score, an evaluation method specifically designed for TCM question answering that effectively assesses answer quality in terms of terminology usage and semantic expression. To the best of our knowledge, this is the first work to systematically evaluate mainstream general domain and TCM-specific LLMs on a unified multimodal benchmark. The datasets and leaderboard are publicly available at https: //tcmladder. com and will be continuously updated. The source code is available at https: //github. com/orangeshushu/TCM-Ladder.

ICML Conference 2024 Conference Paper

Adaptive Conformal Inference by Betting

• Aleksandr Podkopaev
• Dong Xu
• Kuang-chih Lee

Conformal prediction is a valuable tool for quantifying predictive uncertainty of machine learning models. However, its applicability relies on the assumption of data exchangeability, a condition which is often not met in real-world scenarios. In this paper, we consider the problem of adaptive conformal inference without any assumptions about the data generating process. Existing approaches for adaptive conformal inference are based on optimizing the pinball loss using variants of online gradient descent. A notable shortcoming of such approaches is in their explicit dependence on and sensitivity to the choice of the learning rates. In this paper, we propose a different approach for adaptive conformal inference that leverages parameter-free online convex optimization techniques. We prove that our method controls long-term miscoverage frequency at a nominal level and demonstrate its convincing empirical performance without any need of performing cumbersome parameter tuning.

NeurIPS Conference 2024 Conference Paper

An In-depth Investigation of Sparse Rate Reduction in Transformer-like Models

• Yunzhe Hu
• Difan Zou
• Dong Xu

Deep neural networks have long been criticized for being black-box. To unveil the inner workings of modern neural architectures, a recent work proposed an information-theoretic objective function called Sparse Rate Reduction (SRR) and interpreted its unrolled optimization as a Transformer-like model called Coding Rate Reduction Transformer (CRATE). However, the focus of the study was primarily on the basic implementation, and whether this objective is optimized in practice and its causal relationship to generalization remain elusive. Going beyond this study, we derive different implementations by analyzing layer-wise behaviors of CRATE, both theoretically and empirically. To reveal the predictive power of SRR on generalization, we collect a set of model variants induced by varied implementations and hyperparameters and evaluate SRR as a complexity measure based on its correlation with generalization. Surprisingly, we find out that SRR has a positive correlation coefficient and outperforms other baseline measures, such as path-norm and sharpness-based ones. Furthermore, we show that generalization can be improved using SRR as regularization on benchmark image classification datasets. We hope this paper can shed light on leveraging SRR to design principled models and study their generalization ability.

EAAI Journal 2024 Journal Article

Coal allocation optimization based on a hybrid residual prediction model with an improved genetic algorithm

• Ming Liu
• Ziqi Yu
• Boran Li
• Qingjie Wang
• Huawei Ren
• Dong Xu

The objective of the coal blending optimization problem is to find an optimal coal blending in the feasible domain such that the blended coal meets the quality requirements at the end of the coking process and the cost of coal blending is minimized. This paper proposes a hybrid residual prediction model and an improved genetic algorithm to solve this problem and predict coke quality. For this purpose, a hybrid residual prediction model is used to predict coke quality. The model first uses a random forest feature extraction method to reduce the dimensionality of the data, and then trains several prediction models such as eXtreme Gradient Boosting (XGBoost), Adaboost and Light Gradient-Boosting Machine (lightGBM) for different coke indicators an improved genetic algorithm based on the adaptive weighted genetic algorithm (awGA) and another improved genetic algorithm based on a priori knowledge and adaptive random initialization method were designed and implemented to solve the optimization problem under strict constraints (P-awGA). The experimental results show that using the hybrid residual prediction model and the improved genetic algorithm can accurately predict the coke quality and use less time to obtain a lower-cost coal blending solution.

AAAI Conference 2024 Conference Paper

Data-Free Generalized Zero-Shot Learning

• Bowen Tang
• Jing Zhang
• Long Yan
• Qian Yu
• Lu Sheng
• Dong Xu

Deep learning models have the ability to extract rich knowledge from large-scale datasets. However, the sharing of data has become increasingly challenging due to concerns regarding data copyright and privacy. Consequently, this hampers the effective transfer of knowledge from existing data to novel downstream tasks and concepts. Zero-shot learning (ZSL) approaches aim to recognize new classes by transferring semantic knowledge learned from base classes. However, traditional generative ZSL methods often require access to real images from base classes and rely on manually annotated attributes, which presents challenges in terms of data restrictions and model scalability. To this end, this paper tackles a challenging and practical problem dubbed as data-free zero-shot learning (DFZSL), where only the CLIP-based base classes data pre-trained classifier is available for zero-shot classification. Specifically, we propose a generic framework for DFZSL, which consists of three main components. Firstly, to recover the virtual features of the base data, we model the CLIP features of base class images as samples from a von Mises-Fisher (vMF) distribution based on the pre-trained classifier. Secondly, we leverage the text features of CLIP as low-cost semantic information and propose a feature-language prompt tuning (FLPT) method to further align the virtual image features and textual features. Thirdly, we train a conditional generative model using the well-aligned virtual image features and corresponding semantic text features, enabling the generation of new classes features and achieve better zero-shot generalization. Our framework has been evaluated on five commonly used benchmarks for generalized ZSL, as well as 11 benchmarks for the base-to-new ZSL. The results demonstrate the superiority and effectiveness of our approach. Our code is available in https://github.com/ylong4/DFZSL.

AAAI Conference 2024 Conference Paper

Multi-Modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation

• Xiawei Li
• Qingyuan Xu
• Jing Zhang
• Tianyi Zhang
• Qian Yu
• Lu Sheng
• Dong Xu

3D point cloud semantic segmentation has a wide range of applications. Recently, weakly supervised point cloud segmentation methods have been proposed, aiming to alleviate the expensive and laborious manual annotation process by leveraging scene-level labels. However, these methods have not effectively exploited the rich geometric information (such as shape and scale) and appearance information (such as color and texture) present in RGB-D scans. Furthermore, current approaches fail to fully leverage the point affinity that can be inferred from the feature extraction network, which is crucial for learning from weak scene-level labels. Additionally, previous work overlooks the detrimental effects of the long-tailed distribution of point cloud data in weakly supervised 3D semantic segmentation. To this end, this paper proposes a simple yet effective scene-level weakly supervised point cloud segmentation method with a newly introduced multi-modality point affinity inference module. The point affinity proposed in this paper is characterized by features from multiple modalities (e.g., point cloud and RGB), and is further refined by normalizing the classifier weights to alleviate the detrimental effects of long-tailed distribution without the need of the prior of category distribution. Extensive experiments on the ScanNet and S3DIS benchmarks verify the effectiveness of our proposed method, which outperforms the state-of-the-art by ~4% to ~ 6% mIoU. Codes are released at https://github.com/Sunny599/AAAI24-3DWSSG-MMA.

JBHI Journal 2024 Journal Article

pathCLIP: Detection of Genes and Gene Relations From Biological Pathway Figures Through Image-Text Contrastive Learning

• Fei He
• Kai Liu
• Zhiyuan Yang
• Yibo Chen
• Richard D. Hammer
• Dong Xu
• Mihail Popescu

In biomedical literature, biological pathways are commonly described through a combination of images and text. These pathways contain valuable information, including genes and their relationships, which provide insight into biological mechanisms and precision medicine. Curating pathway information across the literature enables the integration of this information to build a comprehensive knowledge base. While some studies have extracted pathway information from images and text independently, they often overlook the correspondence between the two modalities. In this paper, we present a pathway figure curation system named pathCLIP for identifying genes and gene relations from pathway figures. Our key innovation is the use of an image-text contrastive learning model to learn coordinated embeddings of image snippets and text descriptions of genes and gene relations, thereby improving curation. Our validation results, using pathway figures from PubMed, showed that our multimodal model outperforms models using only a single modality. Additionally, our system effectively curates genes and gene relations from multiple literature sources. Two case studies on extracting pathway information from literature of non-small cell lung cancer and Alzheimer's disease further demonstrate the usefulness of our curated pathway information in enhancing related pathways in the KEGG database.

AAAI Conference 2024 Conference Paper

UFDA: Universal Federated Domain Adaptation with Practical Assumptions

• Xinhui Liu
• Zhenghao Chen
• Luping Zhou
• Dong Xu
• Wei Xi
• Gairui Bai
• Yihan Zhao
• Jizhong Zhao

Conventional Federated Domain Adaptation (FDA) approaches usually demand an abundance of assumptions, which makes them significantly less feasible for real-world situations and introduces security hazards. This paper relaxes the assumptions from previous FDAs and studies a more practical scenario named Universal Federated Domain Adaptation (UFDA). It only requires the black-box model and the label set information of each source domain, while the label sets of different source domains could be inconsistent, and the target-domain label set is totally blind. Towards a more effective solution for our newly proposed UFDA scenario, we propose a corresponding methodology called Hot-Learning with Contrastive Label Disambiguation (HCLD). It particularly tackles UFDA's domain shifts and category gaps problems by using one-hot outputs from the black-box models of various source domains. Moreover, to better distinguish the shared and unknown classes, we further present a cluster-level strategy named Mutual-Voting Decision (MVD) to extract robust consensus knowledge across peer classes from both source and target domains. Extensive experiments on three benchmark datasets demonstrate that our method achieves comparable performance for our UFDA scenario with much fewer assumptions, compared to previous methodologies with comprehensive additional assumptions.

NeurIPS Conference 2023 Conference Paper

CS-Isolate: Extracting Hard Confident Examples by Content and Style Isolation

• Yexiong Lin
• Yu Yao
• Xiaolong Shi
• Mingming Gong
• Xu Shen
• Dong Xu
• Tongliang Liu

Label noise widely exists in large-scale image datasets. To mitigate the side effects of label noise, state-of-the-art methods focus on selecting confident examples by leveraging semi-supervised learning. Existing research shows that the ability to extract hard confident examples, which are close to the decision boundary, significantly influences the generalization ability of the learned classifier. In this paper, we find that a key reason for some hard examples being close to the decision boundary is due to the entanglement of style factors with content factors. The hard examples become more discriminative when we focus solely on content factors, such as semantic information, while ignoring style factors. Nonetheless, given only noisy data, content factors are not directly observed and have to be inferred. To tackle the problem of inferring content factors for classification when learning with noisy labels, our objective is to ensure that the content factors of all examples in the same underlying clean class remain unchanged as their style information changes. To achieve this, we utilize different data augmentation techniques to alter the styles while regularizing content factors based on some confident examples. By training existing methods with our inferred content factors, CS-Isolate proves their effectiveness in learning hard examples on benchmark datasets. The implementation is available at https: //github. com/tmllab/2023 NeurIPS CS-isolate.

NeurIPS Conference 2023 Conference Paper

DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models

• XiMing Xing
• Chuang Wang
• Haitao Zhou
• Jing Zhang
• Qian Yu
• Dong Xu

Even though trained mainly on images, we discover that pretrained diffusion models show impressive power in guiding sketch synthesis. In this paper, we present DiffSketcher, an innovative algorithm that creates \textit{vectorized} free-hand sketches using natural language input. DiffSketcher is developed based on a pre-trained text-to-image diffusion model. It performs the task by directly optimizing a set of Bézier curves with an extended version of the score distillation sampling (SDS) loss, which allows us to use a raster-level diffusion model as a prior for optimizing a parametric vectorized sketch generator. Furthermore, we explore attention maps embedded in the diffusion model for effective stroke initialization to speed up the generation process. The generated sketches demonstrate multiple levels of abstraction while maintaining recognizability, underlying structure, and essential visual details of the subject drawn. Our experiments show that DiffSketcher achieves greater quality than prior work. The code and demo of DiffSketcher can be found at https: //ximinng. github. io/DiffSketcher-project/.

JBHI Journal 2023 Journal Article

Large AI Models in Health Informatics: Applications, Challenges, and the Future

• Jianing Qiu
• Lin Li
• Jiankai Sun
• Jiachuan Peng
• Peilun Shi
• Ruiyang Zhang
• Yinzhao Dong
• Kyle Lam

Large AI models, or foundation models, are models recently emerging with massive scales both parameter-wise and data-wise, the magnitudes of which can reach beyond billions. Once pretrained, large AI models demonstrate impressive performance in various downstream tasks. A prime example is ChatGPT, whose capability has compelled people's imagination about the far-reaching influence that large AI models can have and their potential to transform different domains of our lives. In health informatics, the advent of large AI models has brought new paradigms for the design of methodologies. The scale of multi-modal data in the biomedical and health domain has been ever-expanding especially since the community embraced the era of deep learning, which provides the ground to develop, validate, and advance large AI models for breakthroughs in health-related areas. This article presents a comprehensive review of large AI models, from background to their applications. We identify seven key sectors in which large AI models are applicable and might have substantial influence, including: 1) bioinformatics; 2) medical diagnosis; 3) medical imaging; 4) medical informatics; 5) medical education; 6) public health; and 7) medical robotics. We examine their challenges, followed by a critical discussion about potential future directions and pitfalls of large AI models in transforming the field of health informatics.

EAAI Journal 2023 Journal Article

Semi-supervised active learning hypothesis verification for improved geometric expression in three-dimensional object recognition

• Zhenhao Wang
• Rui Xu
• Tingyuan Nie
• Dong Xu

Efficient three-dimensional (3D) object recognition plays an important role in the 3D reconstruction of light-field displays. However, presently, the error rate of 3D implicit shape object recognition remains high, because the local features are sparse in the geometric expression of 3D reconstruction. To address this issue, a hypothesis verification method based on semi-supervised active learning-based K-means++ combined with 3D feature extraction is proposed. The proposed approach consists of the offline and online phases. The algorithm time complexity is O ( n ) and O ( n 2 ), respectively. The offline phase includes keypoint detection, normal estimation, fast point feature histograms (FPFH) descriptor extraction, geometric word weight saving, and indexing structure construction. In addition to the FPFH extraction, the online phase includes nearest geometric word searching, corresponding direction and center voting, and non-maximum suppression. Comparative experiments were conducted in which the models and scenes were tested on the 3D datasets Mian and Tosca that is high-resolution. The experimental results demonstrate that the proposed method resolves the low recognition rate problem of 3D implicit objects, with the highest 3D intersection over union (IoU) reaching 88. 89%.

AAAI Conference 2020 Conference Paper

Channel Pruning Guided by Classification Loss and Feature Importance

• Jinyang Guo
• Wanli Ouyang
• Dong Xu

In this work, we propose a new layer-by-layer channel pruning method called Channel Pruning guided by classification Loss and feature Importance (CPLI). In contrast to the existing layer-by-layer channel pruning approaches that only consider how to reconstruct the features from the next layer, our approach additionally take the classification loss into account in the channel pruning process. We also observe that some reconstructed features will be removed at the next pruning stage. So it is unnecessary to reconstruct these features. To this end, we propose a new strategy to suppress the influence of unimportant features (i. e. , the features will be removed at the next pruning stage). Our comprehensive experiments on three benchmark datasets, i. e. , CIFAR-10, ImageNet, and UCF-101, demonstrate the effectiveness of our CPLI method.

AAAI Conference 2020 Conference Paper

Hashing Based Answer Selection

• Dong Xu
• Wu-Jun Li

Answer selection is an important subtask of question answering (QA), in which deep models usually achieve better performance than non-deep models. Most deep models adopt question-answer interaction mechanisms, such as attention, to get vector representations for answers. When these interaction based deep models are deployed for online prediction, the representations of all answers need to be recalculated for each question. This procedure is time-consuming for deep models with complex encoders like BERT which usually have better accuracy than simple encoders. One possible solution is to store the matrix representation (encoder output) of each answer in memory to avoid recalculation. But this will bring large memory cost. In this paper, we propose a novel method, called hashing based answer selection (HAS), to tackle this problem. HAS adopts a hashing strategy to learn a binary matrix representation for each answer, which can dramatically reduce the memory cost for storing the matrix representations of answers. Hence, HAS can adopt complex encoders like BERT in the model, but the online prediction of HAS is still fast with a low memory cost. Experimental results on three popular answer selection datasets show that HAS can outperform existing models to achieve state-of-the-art performance.

IJCAI Conference 2017 Conference Paper

Dependency Exploitation: A Unified CNN-RNN Approach for Visual Emotion Recognition

• Xinge Zhu
• Liang Li
• Weigang Zhang
• Tianrong Rao
• Min Xu
• Qingming Huang
• Dong Xu

Visual emotion recognition aims to associate images with appropriate emotions. There are different visual stimuli that can affect human emotion from low-level to high-level, such as color, texture, part, object, etc. However, most existing methods treat different levels of features as independent entity without having effective method for feature fusion. In this paper, we propose a unified CNN-RNN model to predict the emotion based on the fused features from different levels by exploiting the dependency among them. Our proposed architecture leverages convolutional neural network (CNN) with multiple layers to extract different levels of features with in a multi-task learning framework, in which two related loss functions are introduced to learn the feature representation. Considering the dependencies within the low-level and high-level features, a new bidirectional recurrent neural network (RNN) is proposed to integrate the learned features from different layers in the CNN model. Extensive experiments on both Internet images and art photo datasets demonstrate that our method outperforms the state-of-the-art methods with at least 7% performance improvement.

AIIM Journal 2010 Journal Article

PMirP: A pre-microRNA prediction method based on structure–sequence hybrid features

• Dongyu Zhao
• Yan Wang
• Di Luo
• Xiaohu Shi
• Liupu Wang
• Dong Xu
• Jun Yu
• Yanchun Liang

Objective MicroRNA is a type of small non-coding RNAs, which usually has a stem-loop structure. As an important stage of microRNA, the pre-microRNA is transported from nuclear to cytoplasm by exportin5 and finally cleaved into mature microRNA. Structure–sequence features and minimum of free energy of secondary structure have been used for predicting pre-microRNA. Meanwhile, the double helix structure with free nucleotides and base-pairing features is used to identify pre-miRNA for the first time. Methods We applied support vector machine for a novel hybrid coding scheme using left-triplet method, the free nucleotides, the minimum of free energy of secondary structure and base-pairings features. Data sets of human pre-microRNA, other 11 species and the latest pre-microRNA sequences were used for testing. Results In this study we developed an improved method for pre-microRNA prediction using a combination of various features and a web server called PMirP. The prediction specificity and sensitivity for real and pseudo human pre-microRNAs are as high as 98. 4% and 94. 9%, respectively. The web server is freely available to the public at http: //ccst. jlu. edu. cn/ci/bioinformatics/MiRNA (accessed: 26 February 2010). Conclusions Experimental results show that the proposed method improves the prediction efficiency and accuracy over existing methods. In addition, the PMirP has lower computational complexity and higher throughput prediction capacity than Mipred web server.

IJCAI Conference 2009 Conference Paper

• Feiping Nie
• Dong Xu
• Ivor W. Tsang
• Changshui Zhang

In this paper, we propose a new spectral clustering method, referred to as Spectral Embedded Clustering (SEC), to minimize the normalized cut criterion in spectral clustering as well as control the mismatch between the cluster assignment matrix and the low dimensional embedded representation of the data. SEC is based on the observation that the cluster assignment matrix of high dimensional data can be represented by a low dimensional linear mapping of data. We also discover the connection between SEC and other clustering methods, such as spectral clustering, Clustering with local and global regularization, K-means and Discriminative K-means. The experiments on many realworld data sets show that SEC significantly outperforms the existing spectral clustering methods as well as K-means clustering related methods.

ICRA Conference 2008 Conference Paper

Trajectory tracking control of omnidirecitonal wheeled mobile manipulators: Robust neural network based sliding mode approach

• Dong Xu
• Dongbin Zhao
• Jianqiang Yi
• Xiang-min Tan
• Zonghai Chen

This paper focuses on developing a robust neural network (NN) based sliding mode controller (NNSMC) to solve the trajectory tracking problem of a redundantly-actuated omnidirectional mobile manipulator. The SMC is designed to be robust to disturbances assuring the stability of the system. The NN is used to identify the unstructured uncertainty of system dynamics. The stability of the closed-loop system, the convergence of the NN weight-updating process, and the boundedness of the NN weight estimation errors are all strictly guaranteed. Through theories analysis, we know the controller is also capable of disturbance-rejection in the presence of time varying disturbances. Finally, simulation results demonstrate the proposed NNSMC approach can guarantee the whole system’s convergence to the desired manifold with prescribed performance.

IROS Conference 2007 Conference Paper

Robust adaptive tracking control of omnidirecitonal wheeled mobile manipulators

• Dong Xu
• Dongbin Zhao
• Jianqiang Yi
• Xiang-min Tan

This paper addresses the trajectory tracking problem for an redundantly-actuated omnidirectional mobile manipulator system with uncertainties and disturbances. The proposed algorithm is robust adaptive control strategy and the parameter estimates are tuned online. First, for designing controller, the conservative upper-bounded function of dynamic model of omnidirectional mobile manipulator system is derived based on the dynamic structure properties. Then, a robust adaptive control scheme is presented to ensure trajectory tracking effect of this closed-loop system. The asymptotical stability is verified a Lyapunov method. Finally, simulation examples are given to demonstrate the proposed approach can guarantee the whole system converge to the desired manifold with prescribed performance.

TCS Journal 2003 Journal Article

Approximation algorithms for NMR spectral peak assignment

• Zhi-Zhong Chen
• Tao Jiang
• Guohui Lin
• Jianjun Wen
• Dong Xu
• Jinbo Xu
• Ying Xu

We study a constrained bipartite matching problem where the input is a weighted bipartite graph G=(U, V, E), U is a set of vertices following a sequential order, V is another set of vertices partitioned into a collection of disjoint subsets, each following a sequential order, and E is a set of edges between U and V with non-negative weights. The objective is to find a matching in G with the maximum weight that satisfies the given sequential orders on both U and V, i. e. if u i+1 follows u i in U and if v j+1 follows v j in V, then u i is matched with v j if and only if u i+1 is matched with v j+1. The problem has recently been formulated as a crucial step in an algorithmic approach for interpreting NMR spectral data (IEEE Comput. Sci. Eng. 4 (2002) 50–62). The interpretation of NMR spectral data is known as a key problem in protein structure determination via NMR spectroscopy. Unfortunately, the constrained bipartite matching problem is NP-hard (IEEE Comput. Sci. Eng. 4 (2002) 50–62). We first propose a 2-approximation algorithm for the problem, which follows directly from the recent result of Bar-Noy et al. (Proc. 32nd ACM Symp. on Theory of Computing (STOC’00), 2000, pp. 735–744) on interval scheduling. However, our extensive experimental results on real NMR spectral data illustrate that the algorithm performs poorly in terms of recovering target-matching edges. We then propose another approximation algorithm that tries to take advantage of the “density” of the sequential order information in V. Although we are only able to prove an approximation ratio of 3 log 2 D for this algorithm, where D is the length of a longest string in V, the experimental results demonstrate that this new algorithm performs much better on real data, i. e. it is able to recover a large fraction of target-matching edges and the weight of its output matching is often in fact close to the maximum. We also prove that the problem is MAX SNP-hard, even if the input bipartite graph is unweighted. We further present an approximation algorithm for a nontrivial special case that breaks the ratio 2 barrier.