Author name cluster

Lei Liu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

41 papers

2 author rows

EAAI Journal 2026 Journal Article

A knowledge prompt augmented lightweight multimodal language assistant for biomedicine

Lei Liu
Xiangdong Su
Xingxiang Zhou
Guanglai Gao

Hallucinations caused by uncertain reasoning are critical in multimodal large language models, where factual correctness is vital for clinical applications. This challenge is further exacerbated in resource-constrained lightweight models due to their limited capacity and lack of domain-specific knowledge. To address this, we propose a knowledge prompt-augmented, lightweight multimodal language assistant for biomedicine (BioMiLA-K). BioMiLA-K incorporates relevant entities from a pre-built knowledge base as prompts, mitigating hallucinations and improving reasoning. We introduce a Knowledge Match Mechanism module for multimodal entity selection: a biomedical vision-language pretraining model encodes medical images, retrieves similar embeddings from the knowledge base, and links them to corresponding textual entities. A multilayer perceptron model, trained with cross-entropy and ranking losses, selects the most relevant entities for the prompt. Next, we propose an Entity Attention Optimization module, which enables the language model to dynamically assess entity importance by computing cross-attention among entities, images, and questions. A self-supervised attention loss guides BioMiLA-K to focus on critical knowledge and generate accurate responses. Using 80, 080 medical image-text pairs, we build a multimodal knowledge base and generate 90, 000 visual instruction samples. BioMiLA-K adopts a two-stage instruction tuning: image-caption alignment tuning, followed by low-rank adaptation fine-tuning on the generated data. With just 0. 79 billion parameters, it achieves competitive performance on multiple medical visual question answering benchmarks. Its sub-billion scale enables training and deployment on a single graphics processing unit with 24 GB of memory, demonstrating that explicit knowledge grounding can effectively substitute for large model size in biomedical artificial intelligence-based multimodal language systems.

JBHI Journal 2026 Journal Article

A Prompt-Guided Vision-Language Framework for Interpretable and Region-Aware Disease Diagnosis in Chest X-rays

Lei Liu
Shutong Luo
Xiukun Li
Fuzhe Ma

Effective interpretation of chest X-rays requires a tightly integrated process of visual analysis, diagnostic reasoning, and structured reporting. Yet, most machine learning systems handle these steps in isolation. Visual encoders are typically trained without diagnostic context, and language outputs often lack spatial grounding. To address this gap, we propose an interactive vision-language framework that supports prompt-guided reasoning over both textual and spatial queries, enabling region-aware, clinically aligned interpretations. The framework comprises three functional modules: Prompt-Guided Localization (PGL) for identifying relevant regions, Region-Level Diagnosis (RLD) for structured classification, and Region-Aware Explanation (RAE) for generating localized descriptions. These modules are unified through a regional alignment mechanism built on a multi-task Detection Transformer (DETR) backbone, which maps prompts and image regions into a shared semantic space. To train the system under limited supervision, we adopt a two-stage strategy: contrastive pretraining to establish cross-modal alignment, followed by multi-task fine-tuning to support downstream tasks including disease classification and report generation. Experiments across the publicly available chest X-ray datasets MIMIC-CXR, VinDr-CXR, and MS-CXR demonstrate consistent gains compared with state-of-the-art methods. Module-wise ablations further validate the contribution of each component and highlight the framework's potential for transparent, clinically applicable diagnostic support.

JBHI Journal 2026 Journal Article

Generation and Selection: A Self-Iterative Two-Stage Data Augmentation Method for Automated ECG Classification

Chaoying Jiang
Yujing Xin
Ning Liu
Yonghui Xu
Lei Liu
Lizhen Cui
Jianyong Wang

Automated electrocardiogram (ECG) classification tasks play a crucial role in clinical but face challenges due to the scarcity of accessible and well-labeled data. ECG data augmentation is an effective way to address these challenges, either by altering the characteristics of real ECG data or using statistical and generative models to generate labeled data. However, the generated data often suffer from noise in both the data and label, which can reduce the performance of classification models. To address this, we propose a novel self-iterative two-stage data augmentation method for automated ECG classification, called SiTs-ECG. In the generation stage, an unconditional diffusion model, guided by a Transformer encoder, is trained to capture the complex characteristics of long-term ECG signals, generating high-quality ECG-like samples. In the selection stage, the generated samples are assigned pseudo-labels by a well-trained base classification model, and those generated samples for which the model can confidently predict the pseudo-labels are selected. We then integrate these stages into a self-iterative training process to continually improve the performance of base classification model. Extensive experiments on three real-world datasets demonstrate the effectiveness of our method. Notably, on the Apnea-ECG dataset, using ECG-Transformer as the downstream classification model, Precision, Recall, F1, and Accuracy are improved by 7. 9, 9. 1, 9. 2, and 7. 3 percentage points, respectively. Furthermore, our method is versatile and compatible with various generative and downstream classification models, showing promising applications in automated ECG classification in the clinical field.

YNIMG Journal 2026 Journal Article

Hierarchical neurobehavioral model reveals that shared flexibility, not individual stability, supports rhythmic coordination

Ruoyu Niu
Yanan Li
Lei Liu
Yafeng Pan
Ying Liu

Interpersonal coordination requires balancing individual control with interaction-derived synergy, yet it remains unclear when neural coupling contributes beyond behavior. Using an fNIRS hyperscanning paradigm, we examined dyadic rhythmic coordination and jointly modeled behavioral stability, dispositional structure, and interbrain synchrony within a hierarchical neurobehavioral framework. Across models, mean individual stability was negatively associated with dyadic performance, whereas interaction-derived shared flexibility (i.e., dyad-level behavioral stability synergy) was the most robust positive predictor. Incorporating dispositional structure showed that larger within-dyad differences in figure-embedding performance impaired coordination, whereas higher dyad-level self-esteem facilitated coordination. The neural coupling index showed no reliable main effect after accounting for behavioral and trait factors, but moderation analyses indicated a conditional contribution: interbrain synchrony compensated when shared flexibility was low, with diminishing benefit as synergy increased. Together, these findings support a hierarchical neurobehavioral architecture in which shared flexibility provides the primary foundation of coordination, dispositional structure shapes the conditions for synergy, and interbrain synchrony contributes in a context-dependent manner.

JBHI Journal 2026 Journal Article

KidMesh: Computational Mesh Reconstruction for Pediatric Congenital Hydronephrosis Using Deep Neural Networks

Haoran Sun
Zhanpeng Zhu
Anguo Zhang
Bo Liu
Zhaohua Lin
Liqin Huang
Mingjing Yang
Lei Liu

Pediatric congenital hydronephrosis (CH) is a common urinary tract disorder, primarily caused by obstruction at the renal pelvis-ureter junction. Magnetic resonance urography (MRU) can visualize hydronephrosis, including renal pelvis and calyces, by utilizing the natural contrast provided by water. Existing voxel-based segmentation approaches can extract CH regions from MRU, facilitating disease diagnosis and prognosis. However, these segmentation methods predominantly focus on morphological features, such as size, shape, and structure. To enable functional assessments, such as urodynamic simulations, external complex post-processing steps are required to convert these results into mesh-level representations. To address this limitation, we propose an end-to-end method based on deep neural networks, namely KidMesh, which could automatically reconstruct CH meshes directly from MRU. Generally, KidMesh extracts feature maps from MRU images and converts them into feature vertices through grid sampling. It then deforms a template mesh according to these feature vertices to generate the specific CH meshes of MRU images. Meanwhile, we develop a novel schema to train KidMesh without relying on accurate mesh-level annotations, which are difficult to obtain due to the sparsely sampled MRU slices. Experimental results show that KidMesh reconstructs CH meshes in an average of 0. 4 seconds, and achieve comparable performance to conventional methods without requiring post-processing. The reconstructed meshes exhibited no self-intersections, with only 3. 7% and 0. 2% of the vertices having error distances exceeding 3. 2mm and 6. 4mm, respectively. After rasterization, these meshes achieved a Dice score of 0. 86 against manually delineated CH masks. Furthermore, these meshes could be used in renal urine flow simulations, providing valuable urodynamic information for clinical practice.

JBHI Journal 2026 Journal Article

MoACNN-XGNet: Interpretable Multi-Omics Convolutional Network for Breast Cancer Subtyping and Prognostic Genes Identification

Qian Li
Lei Liu
Qing Zhang
Xiaobin Zhang
Na Li
Yaoyao Zhao
Jiayi Teng
Fuzhong Xue

Breast cancer, a highly heterogeneous disease at both the phenotypic and molecular levels, presents significant challenges for prognosis and treatment. Accurate subtyping of breast cancer is critical due to its complex biological characteristics, which directly influence disease progression and therapeutic outcomes. In this study, we integrate multi-omics data, including copy number variation, RNA sequencing, and DNA methylation, to generate two-dimensional representations of each sample using Uniform Manifold Approximation and Projection. This transformation enhances data interpretability and supports subsequent learning tasks. Traditional convolutional neural networks have demonstrated potential in medical image analysis but often struggle with high-dimensional omics data. To address this limitation, we propose MoACNN-XGNet, an attention-based convolutional neural network framework that prioritizes key features within image-transformed multi-omics data. Our method significantly improves the precision of subtype classification and effectively overcomes the challenges posed by the high dimensionality and structural complexity of multi-omics data. Furthermore, we employ the Guided Grad-CAM method to enhance model interpretability, enabling the identification of subtype-specific explainable genes. Subsequent enrichment and survival analyses of these genes reveal critical biological pathways and potential therapeutic targets. This study offers a novel approach to refining breast cancer subtyping and highlights the potential for personalized treatment strategies, ultimately aiming to improve patient survival outcomes.

EAAI Journal 2026 Journal Article

Monocular depth estimation in adverse weather via cross-domain data fusion and hybrid supervision

Jia Yu
Xiaxu Huang
Lei Liu
Zhonghao Zuo

Monocular depth estimation is vital for autonomous driving perception, yet existing models suffer substantial performance degradation under adverse weather conditions such as rain and snow. This limitation arises mainly from: (1) the scarcity of annotated real-world adverse-weather data, (2) severe domain gaps between synthetic and real imagery, and (3) motion-induced scale drift in self-supervised learning. To address these challenges, we propose HybridDepth, a unified framework that jointly leverages real, generative adversarial network augmented, and synthetic data while tightly coupling visual and inertial measurement unit cues to achieve robust all-weather depth estimation. HybridDepth integrates three complementary data domains: real-world data, generative adversarial network augmented real data (optimized via self-supervised photometric consistency), and synthetic data with ground-truth depth. To mitigate scale drift, we enforce kinematic consistency by fusing visual features with inertial measurement unit measurements. For domain adaptation, we incorporate gradient reversal layers to learn domain-invariant representations, and we introduce an exponential moving average based dynamic loss weighting strategy to balance supervised and self-supervised objectives. HybridDepth achieves up to a 5% reduction in Absolute Relative Error relative to the MonoViT baseline across multiple adverse-weather benchmarks, consistently outperforming state-of-the-art methods under rain, fog, and snow conditions. These results highlight HybridDepth as a promising paradigm for robust, scalable, and reliable all-weather depth perception in autonomous driving.

AAAI Conference 2026 Conference Paper

PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation

Zhehao Tan
Yihan Jiao
Dan Yang
Junwei Liu
Lei Liu
Jie Feng
Duolin Sun
Yue Shen

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by integrating external knowledge, where the LLM's ability to generate responses based on the combination of a given query and retrieved documents is crucial. However, most benchmarks focus on overall RAG system performance, rarely assessing LLM-specific capabilities. Current benchmarks emphasize broad aspects such as noise robustness, but lack a systematic and granular evaluation framework on document utilization. To this end, we introduce Placeholder-RAG-Benchmark, a multi-level fine-grained benchmark, emphasizing the following progressive dimensions: (1) multi-level filtering abilities, (2) combination abilities, and (3) reference reasoning. To provide a more nuanced understanding of LLMs' roles in RAG systems, we formulate an innovative placeholder-based approach to decouple the contributions of the LLM's parametric knowledge and the external knowledge. Experiments demonstrate the limitations of representative LLMs in the RAG system's generation capabilities, particularly in error resilience and context faithfulness. Our benchmark provides a reproducible framework for developing more reliable and efficient RAG systems.

PDF Details DOI

EAAI Journal 2026 Journal Article

Progressive information integration in lightweight image super-resolution

Longfeng Shen
Jiacong Chen
Liangjin Diao
Lei Liu
Fenglan Qin
Fangzhen Ge

Transformer-based models have considerably improved image super-resolution (SR). However, these networks fail to design multi-operations aggregation architecture with guidance of SR knowledge. To tackle these drawbacks, we propose a novel progressive information integration (PII) module. It extracts the input features from progressive perspectives: a dense region, sparse region, and three-dimensional space. We employ a local convolution block to access pixels in the dense region and window-based self-attention for those in the sparse region. To leverage the advantages of both channel-attention and spatial-attention schemes, we introduce a Hybrid Attention block (HAB). This block enables the effective use of pixels in three-dimensional space by combining the complementary benefits of the two attention schemes. As an artificial intelligence (AI) -driven approach, our method is applied to lightweight image super-resolution tasks, aiming to balance performance and computational efficiency. Extensive experiments demonstrate that our PII-based super-resolution (PII-SR) achieves the superior results on lightweight SR benchmarks with fewer parameters (e. g. , 26. 81 dB (dB)@Urban100 × 4 with only 652 thousand (K) parameters).

NeurIPS Conference 2025 Conference Paper

Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search

Haoran Sun
Yankai Jiang
Wenjie Lou
Yujie Zhang
Wenjie Li
Lilong Wang
Mianxin Liu
Lei Liu

Multimodal large language models (MLLMs) have begun to demonstrate robust reasoning capabilities on general tasks, yet their application in the medical domain remains in its early stages. Constructing chain-of-thought (CoT) training data is essential for bolstering the reasoning abilities of medical MLLMs. However, existing approaches exhibit a deficiency in offering a comprehensive framework for searching and evaluating effective reasoning paths towards critical diagnosis. To address this challenge, we propose Mentor-Intern Collaborative Search (MICS), a novel reasoning-path searching scheme to generate rigorous and effective medical CoT data. MICS first leverages mentor models to initialize the reasoning, one step at a time, then prompts each intern model to continue the thinking along those initiated paths, and finally selects the optimal reasoning path according to the overall reasoning performance of multiple intern models. The reasoning performance is determined by an MICS-Score, which assesses the quality of generated reasoning paths. Eventually, we construct MMRP, a multi-task medical reasoning dataset with ranked difficulty, and Chiron-o1, a new medical MLLM devised via a curriculum learning strategy, with robust visual question-answering and generalizable reasoning capabilities. Extensive experiments demonstrate that Chiron-o1, trained on our CoT dataset constructed using MICS, achieves state-of-the-art performance across a list of medical visual question answering and reasoning benchmarks. Codes are available at https: //github. com/Yankai96/Chiron-o1

NeurIPS Conference 2025 Conference Paper

CPSea: Large-scale cyclic peptide-protein complex dataset for machine learning in cyclic peptide design

Ziyi Yang
Hanyuan Xie
Yinjun Jia
Xiangzhe Kong
Jiqing Zheng
Ziting Zhang
Yang Liu
Lei Liu

Cyclic peptides exhibit better binding affinity and proteolytic stability compared to their linear counterparts. However, the development of cyclic peptide design models is hindered by the scarcity of data. To address this, we introduce **CPSea**(**C**yclic **P**eptide **Sea**), a dataset of 2. 71 million cyclic peptide-receptor complexes, curated through systematic mining of the AlphaFold Database (AFDB). Our pipeline extracts compact domains from AFDB, identifies cyclization sites using the $\beta$-carbon (C$_\beta$) distance thresholds, and applies multi-stage filtering to ensure structure fidelity and binding compatibility. Compared with experimental data of cyclic peptides, CPSea shows similar distributions in metrics on structure fidelity and wet-lab compatibility. To our knowledge, CPSea is the largest cyclic peptide-receptor dataset to date, enabling end-to-end model training for the first time. The dataset also showcases the feasibility of simulating inter-chain interactions using intra-chain interactions, expanding available resources for machine-learning models on protein-protein interactions. The dataset and relevant scripts are accessible on GitHub ([https: //github. com/YZY010418/CPSea](https: //github. com/YZY010418/CPSea)).

IJCAI Conference 2025 Conference Paper

DiffSQL: Leveraging Diffusion Model for Zero-Shot Self-Supervised Monocular Depth Estimation

Heyuan Zheng
Yunji Liang
Lei Liu
Zhiwen Yu

Self-supervised monocular depth estimation has attracted significant attention due to its broad applications in autonomous driving and robotics. Although significant performance improvements have been achieved by learning the relative distance of objects with the introduction of Self Query Layer (SQL), it struggles with zero-shot generalization due to the lack of geometric features and the fixed number of query sizes. To address these problems, we propose a diffusion-augmented self-supervised depth estimation framework, named DiffSQL, to learn geometric priors for feature augmentation. Additionally, we introduce a dynamic self-query layer that implicitly computes the relative distances between objects by adjusting the query size according to the feature distribution. Experimental results on the KITTI dataset show that DiffSQL outperforms SQLdepth by 1. 03% in terms of AbsRel and 2. 79% in terms of SqRel. Furthermore, our experiments demonstrate that DiffSQL is superior in zero-shot generalization.

PDF Details DOI

ICML Conference 2025 Conference Paper

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment

Chengting Yu
Xiaochen Zhao
Lei Liu
Shu Yang
Gaoang Wang
Erping Li 0001
Aili Wang 0002

Spiking Neural Networks (SNNs) are emerging as a brain-inspired alternative to traditional Artificial Neural Networks (ANNs), prized for their potential energy efficiency on neuromorphic hardware. Despite this, SNNs often suffer from accuracy degradation compared to ANNs and face deployment challenges due to fixed inference timesteps, which require retraining for adjustments, limiting operational flexibility. To address these issues, our work considers the spatio-temporal property inherent in SNNs, and proposes a novel distillation framework for deep SNNs that optimizes performance across full-range timesteps without specific retraining, enhancing both efficacy and deployment adaptability. We provide both theoretical analysis and empirical validations to illustrate that training guarantees the convergence of all implicit models across full-range timesteps. Experimental results on CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet demonstrate state-of-the-art performance among distillation-based SNNs training methods. Our code is available at https: //github. com/Intelli-Chip-Lab/snn_temporal_decoupling_distillation.

NeurIPS Conference 2025 Conference Paper

Enhanced Expert Merging for Mixture-of-Experts in Graph Foundation Models

Lei Liu
Xingyu Xia
Qianqian Xie
Ben Liu
Wenjie Xu
Min Peng

Graph foundation models (GFMs) have emerged as a promising paradigm for learning transferable knowledge across diverse graph-structured data. The inherent heterogeneity in features and graph structures poses significant challenges for building scalable and generalizable GFMs. Existing research has employed mixture-of-experts (MoE) models to handle the challenges, assigning the most suitable expert to each graph. Despite this, the underlying mechanisms of MoE within the context of GFMs remain insufficiently explored. In this work, we conduct an in-depth experimental study on an MoE-based GFM and uncover an intriguing finding: the experts ranked second and third assigned by the router perform better than the top-ranked expert. This insight motivates us to investigate the potential of leveraging knowledge embedded across multiple experts. However, directly ensembling the outputs of multiple experts would incur substantial computational overhead, while applying a standard expert merging strategy risks suboptimal performance. To address these challenges, we introduce two enhanced expert merging strategies that retain the computational efficiency of expert merging, while improving performance to approach the effectiveness of expert ensembling. Specifically, we propose (i) a knowledge distillation-inspired expert merging method that aligns the behavior of parameter-fused experts with expert ensembles, and (ii) a theoretical parameter proximity approach that leverages the similarity of expert parameters to approximate ensemble outputs while preserving diversity. Extensive experiments demonstrate that our methods effectively enhance model performance.

NeurIPS Conference 2025 Conference Paper

Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training

Xiaochen Zhao
Chengting Yu
Kairong Yu
Lei Liu
Aili Wang

Spiking Neural Networks (SNNs) exhibit exceptional energy efficiency on neuromorphic hardware due to their sparse activation patterns. However, conventional training methods based on surrogate gradients and Backpropagation Through Time (BPTT) not only lag behind Artificial Neural Networks (ANNs) in performance, but also incur significant computational and memory overheads that grow linearly with the temporal dimension. To enable high-performance SNN training under limited computational resources, we propose an enhanced self-distillation framework, jointly optimized with rate-based backpropagation. Specifically, the firing rates of intermediate SNN layers are projected onto lightweight ANN branches, and high-quality knowledge generated by the model itself is used to optimize substructures through the ANN pathways. Unlike traditional self-distillation paradigms, we observe that low-quality self-generated knowledge may hinder convergence. To address this, we decouple the teacher signal into reliable and unreliable components, ensuring that only reliable knowledge is used to guide the optimization of the model. Extensive experiments on CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet demonstrate that our method reduces training complexity while achieving high-performance SNN training. Our code is available at https: //github. com/Intelli-Chip-Lab/enhanced-self-distillation-framework-for-snn.

ECAI Conference 2025 Conference Paper

Enhancing Learning of Spiking Neural Networks Through Normalization with Time-Based Statistics Estimation

Lei Liu
Chengting Yu
Kainan Wang
Aili Wang

Spiking Neural Networks (SNNs) represent a promising avenue for energy-efficient neuromorphic computing. Despite their potential, SNNs typically underperform compared to Artificial Neural Networks (ANNs) due to their complex spatio-temporal dynamics. To improve learning in these networks, researchers have developed various approaches that account for their unique characteristics—among them, normalization techniques have proven especially important. Recently, online learning algorithms have been explored for SNN training as they update network weights using only temporally local information, avoiding the high memory demands associated with Backpropagation Through Time (BPTT). However, the computational mechanism of online learning, which relies on temporally local information to update weights, hinders the application of integrating effective normalization techniques tailored for SNNs. In this work, we propose a Time-based Statistics Estimation (TSE) method to address limitations in existing normalization strategies for SNNs. We begin by establishing a systematic link between overall statistics and time-step-specific ones, leveraging the decomposability of key statistical measures. This insight allows our proposed TSE method to reliably estimate overall statistics using only recent iterations. Furthermore, the proposed method is compatible with both BPTT and online learning, consistently yielding strong performance across learning paradigms. Experiments on CIFAR-10, CIFAR-100, ImageNet, and DVS-CIFAR10 datasets demonstrate the superior performance of our method on both static and neuromorphic datasets. In particular, our method achieves state-of-the-art performance in online learning for SNN training.

EAAI Journal 2025 Journal Article

Hierarchical semantics guided multi-scale correlation network for alignment-free red-green-blue and thermal salient object detection

Chengmei Han
Lei Liu
Kunpeng Wang
Fei Xie
Bing Wei

RGBT (red-green-blue and thermal) salient object detection (SOD) aims to identify and highlight the most visually salient objects in an image by leveraging the complementary information from both RGB and thermal (TIR) modalities. It is particularly effective for 24/7 intelligent surveillance and autonomous perception in smart city security and traffic monitoring, especially under low light and adverse weather. However, existing methods primarily rely on manually aligned datasets, which are limited in handling the challenges posed by unaligned multi-modal data in real-world applications. Furthermore, these methods usually extract complementary information from both modalities using fixed-size windows (Liuet al. , 2022, Wanget al. , 2024b). However, such fixed-size windows are not effective in dealing with unaligned multi-modal images due to spatial inconsistencies. Additionally, existing methods often use single-layer high-level feature to represent semantic information, which fails to fully exploit the complementary benefits of multi-level features, thereby reducing the effectiveness of semantic guidance. To address these challenges, we propose a Hierarchical Semantics guided Multi-scale correlation Network (HSMNet) for alignment-free RGBT SOD. A Hierarchical Semantic Fusion Module (HSFM) dynamically assigns weights to features from multiple levels, enabling adaptive fusion of multi-level semantic information. A Multi-scale Asymmetric Correlation Module (MACM) employs windows of various sizes to capture asymmetric correlations between unaligned multi-modal data, enhancing cross-modal complementary information extraction even when data are not perfectly aligned. We conduct extensive experiments on unaligned, weakly aligned and aligned RGBT SOD datasets, with results demonstrating that our method outperforms state-of-the-art algorithms, achieving superior accuracy and robustness in both unaligned and weakly aligned RGBT SOD scenarios.

AIIM Journal 2025 Journal Article

Medical multimodal foundation models in clinical diagnosis and treatment: Applications, challenges, and future directions

Kai Sun
Siyan Xue
Fuchun Sun
Haoran Sun
Yu Luo
Ling Wang
Siyuan Wang
Na Guo

Recent advancements in deep learning have significantly revolutionized the field of clinical diagnosis and treatment, offering novel approaches to improve diagnostic precision and treatment efficacy across diverse clinical domains, thus driving the pursuit of precision medicine. The growing availability of multi-organ and multimodal datasets has accelerated the development of large-scale Medical Multimodal Foundation Models (MMFMs). These models, known for their strong generalization capabilities and rich representational power, are increasingly being adapted to address a wide range of clinical tasks, from early diagnosis to personalized treatment strategies. This review offers a comprehensive analysis of recent developments in MMFMs, focusing on three key aspects: datasets, model architectures, and clinical applications. We also explore the challenges and opportunities in optimizing multimodal representations and discuss how these advancements are shaping the future of healthcare by enabling improved patient outcomes and more efficient clinical workflows.

FLAP Journal 2025 Journal Article

Study of Residuated Lattices via Subtractive Derivations

Mei Wang
Xiaoyun Cheng
Lei Liu

In this paper, we introduce the concept of subtractive derivations and study some of their related algebraic properties. Also, we give some characterizations of subtractive derivations on its idempotent center. In particular, we prove that the set of all subtractive derivations forms a Heyting algebras with respect to the set inclusion, and show that every Heyting algebra is isomorphic to the alge- bra of all subtractive derivations. Moreover, Heyting algebras by the fixed set of subtractive derivations are characterized. Finally, we discuss the relationship between subtractive derivations and other derivations for residuated lattices. These results of the paper can provide the common properties of subtractive derivations on the t-norm based fuzzy logical algebras.

AAAI Conference 2025 Conference Paper

VQLTI: Long-Term Tropical Cyclone Intensity Forecasting with Physical Constraints

Xinyu Wang
Lei Liu
Kang Chen
Tao Han
Bin Li
Lei Bai

Tropical cyclone (TC) intensity forecasting is crucial for early disaster warning and emergency decision-making. Numerous researchers have explored deep-learning methods to address computational and post-processing issues in operational forecasting. Regrettably, they exhibit subpar long-term forecasting capabilities. We use two strategies to enhance long-term forecasting. (1) By enhancing the matching between TC intensity and spatial information, we can improve long-term forecasting performance. (2) Incorporating physical knowledge and physical constraints can help mitigate the accumulation of forecasting errors. To achieve the above strategies, we propose the VQLTI framework. VQLTI transfers the TC intensity information to a discrete latent space while retaining the spatial information differences, using large-scale spatial meteorological data as conditions. Furthermore, we leverage the forecast from the weather prediction model FengWu to provide additional physical knowledge for VQLTI. Additionally, we calculate the potential intensity (PI) to impose physical constraints on the latent variables. In the global long-term TC intensity forecasting, VQLTI achieves state-of-the-art results for the 24h to 120h, with the MSW (Maximum Sustained Wind) forecast error reduced by 35.65%-42.51% compared to ECMWF-IFS.

PDF Details DOI

JBHI Journal 2024 Journal Article

A Comprehensive Privacy-Preserving Federated Learning Scheme With Secure Authentication and Aggregation for Internet of Medical Things

Jingwei Liu
Jin Zhang
Mian Ahmad Jan
Rong Sun
Lei Liu
Sahil Verma
Pushpita Chatterjee

Data mining, integration, and utilization are the inevitable trend of the Internet of Medical Things (IoMT) in the context of Big Data. With the increasing demand for data privacy, federated learning has emerged as a new paradigm, which enables distributed joint training of medical data sources without leaving the private domain. However, federated learning is suffering from security threats as the shared local model will reveal original datasets. Privacy leakage is even more fatal in healthcare because medical data contains critically sensitive information. In addition, open wireless channels are susceptible to malicious attacks. To further safeguard the privacy of IoMT, we propose a comprehensive privacy-preserving federated learning scheme with a tactful dropout handling mechanism. The proposed scheme leverages blind masking and certificateless proxy re-encryption (CL-PRE) for secure aggregation, ensuring the confidentiality of the local model and rendering the global model invisible to any parties other than clients. It also provides authentication of uploaded models while protecting identity privacy. Compared with other relevant schemes, our solution has better performance on functional features and efficiency, and is more applicable to IoMT systems with many devices.

EAAI Journal 2024 Journal Article

A parallel ensemble optimization and transfer learning based intelligent fault diagnosis framework for bearings

Guiting Tang
Cai Yi
Lei Liu
Du Xu
Qiuyang Zhou
Yongxu Hu
Pengcheng Zhou
Jianhui Lin

Transfer learning (TL) is an important method to accurately identify the bearing health status in cross-domain and ensure the safe operation of machinery. With the advancement in research, it will become a trend to choose different neural networks or optimization functions to improve and re-model fault diagnosis methods. However, the variants of these fault diagnostic methods are less capable of generalizing input dimensions and do not significantly increase demand for machinery expertise. The idea of ensemble learning solves the problem of low generalization. In this research, a parallel ensemble optimization loss function and multi-source TL based model are proposed to solve the problem of unknown distribution difference between source domain and target domain, thus improving the generalization of optimization objectives. Firstly, based on the signal demodulation method, an adaptive input module is constructed to automatically select the input length from the original vibration signal. Secondly, a TL network with low-dimensional features reuse is constructed to achieve weight and bias sharing. Thirdly, a parallel ensemble optimization loss function is developed to align the data whose distribution is unknown between source and target domains. Finally, two cases with multi-source, unsupervised, and cross-domain TL are used to verify the performance of the proposed method. The average accuracy in case 1 and case 2 is 99. 81 % and 99. 17 % respectively. It is proved that the proposed method can not only get rid of the limitation of manual input length setting, but also overcome the limitation of optimization function, which is more effective than the existing intelligent fault diagnosis models.

NeurIPS Conference 2024 Conference Paper

Advancing Training Efficiency of Deep Spiking Neural Networks through Rate-based Backpropagation

Chengting Yu
Lei Liu
Gaoang Wang
Erping Li
Aili Wang

Recent insights have revealed that rate-coding is a primary form of information representation captured by surrogate-gradient-based Backpropagation Through Time (BPTT) in training deep Spiking Neural Networks (SNNs). Motivated by these findings, we propose rate-based backpropagation, a training strategy specifically designed to exploit rate-based representations to reduce the complexity of BPTT. Our method minimizes reliance on detailed temporal derivatives by focusing on averaged dynamics, streamlining the computational graph to reduce memory and computational demands of SNNs training. We substantiate the rationality of the gradient approximation between BPTT and the proposed method through both theoretical analysis and empirical observations. Comprehensive experiments on CIFAR-10, CIFAR-100, ImageNet, and CIFAR10-DVS validate that our method achieves comparable performance to BPTT counterparts, and surpasses state-of-the-art efficient training techniques. By leveraging the inherent benefits of rate-coding, this work sets the stage for more scalable and efficient SNNs training within resource-constrained environments.

PDF Details DOI

JBHI Journal 2024 Journal Article

CiGNN: A Causality-Informed and Graph Neural Network Based Framework for Cuffless Continuous Blood Pressure Estimation

Lei Liu
Huiqi Lu
Maxine Whelan
Yifan Chen
Xiaorong Ding

Causalityholds profound potentials to dissipate confusion and improve accuracy in cuffless continuous blood pressure (BP) estimation, an area often neglected in current research. In this study, we propose a two-stage framework, CiGNN, that seamlessly integrates causality and graph neural network (GNN) for cuffless continuous BP estimation. The first stage concentrates on the generation of a causal graph between BP and wearable features from the the perspective of causal inference, so as to identify features that are causally related to BP variations. This stage is pivotal for the identification of novel causal features from the causal graph beyond pulse transit time (PTT). We found these causal features empower better tracking in BP changes compared to PTT. For the second stage, a spatio-temporal GNN (STGNN) is utilized to learn from the causal graph obtained from the first stage. The STGNN can exploit both the spatial information within the causal graph and temporal information from beat-by-beat cardiac signals for refined cuffless continuous BP estimation. We evaluated the proposed method with three datasets that include 305 subjects (102 hypertensive patients) with age ranging from 20–90 and BP at different levels, with the continuous Finapres BP as references. The mean absolute difference (MAD) for estimated systolic blood pressure (SBP) and diastolic blood pressure (DBP) were 3. 77 mmHg and 2. 52 mmHg, respectively, which outperformed comparison methods. In all cases including subjects with different age groups, while doing various maneuvers that induces BP changes at different levels and with or without hypertension, the proposed CiGNN method demonstrates superior performance for cuffless continuous BP estimation. These findings suggest that the proposed CiGNN is a promising approach in elucidating the causal mechanisms of cuffless BP estimation and can substantially enhance the precision of BP measurement.

JBHI Journal 2024 Journal Article

Elimination of Random Mixed Noise in ECG Using Convolutional Denoising Autoencoder With Transformer Encoder

Meng Chen
Yongjian Li
Liting Zhang
Lei Liu
Baokun Han
Wenzhuo Shi
Shoushui Wei

Electrocardiogram (ECG) signals frequently encounter diverse types of noise, such as baseline wander (BW), electrode motion (EM) artifacts, muscle artifact (MA), and others. These noises often occur in combination during the actual data acquisition process, resulting in erroneous or perplexing interpretations for cardiologists. To suppress random mixed noise (RMN) in ECG with less distortion, we propose a Transformer-based Convolutional Denoising AutoEncoder model (TCDAE) in this study. The encoder of TCDAE is composed of three stacked gated convolutional layers and a Transformer encoder block with a point-wise multi-head self-attention module. To obtain minimal distortion in both time and frequency domains, we also propose a frequency weighted Huber loss function in training phase to better approximate the original signals. The TCDAE model is trained and tested on the QT Database (QTDB) and MIT-BIH Noise Stress Test Database (NSTDB), with the training data and testing data coming from different records. All the metrics perform the most robust in overall noise and separate noise intervals for RMN removal compared with the baseline methods. We also conduct generalization tests on the Icentia11k database where the TCDAE outperforms the state-of-the-art models, with a 55% reduction of the false positives in R peak detection after denoising. The TCDAE model approximates the short-term and long-term characteristics of ECG signals and has higher stability even under extreme RMN corruption. The memory consumption and inference speed of TCDAE are also feasible for its deployment in clinical applications.

NeurIPS Conference 2024 Conference Paper

One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos

Zechen Bai
Tong He
Haiyang Mei
Pichao Wang
Ziteng Gao
Joya Chen
Lei Liu
Zheng Zhang

We introduce VideoLISA, a video-based multimodal large language model designed to tackle the problem of language-instructed reasoning segmentation in videos. Leveraging the reasoning capabilities and world knowledge of large language models, and augmented by the Segment Anything Model, VideoLISA generates temporally consistent segmentation masks in videos based on language instructions. Existing image-based methods, such as LISA, struggle with video tasks due to the additional temporal dimension, which requires temporal dynamic understanding and consistent segmentation across frames. VideoLISA addresses these challenges by integrating a Sparse Dense Sampling strategy into the video-LLM, which balances temporal context and spatial detail within computational constraints. Additionally, we propose a One-Token-Seg-All approach using a specially designed token, enabling the model to segment and track objects across multiple frames. Extensive evaluations on diverse benchmarks, including our newly introduced ReasonVOS benchmark, demonstrate VideoLISA's superior performance in video object segmentation tasks involving complex reasoning, temporal understanding, and object tracking. While optimized for videos, VideoLISA also shows promising generalization to image segmentation, revealing its potential as a unified foundation model for language-instructed object segmentation. Code and model will be available at: https: //github. com/showlab/VideoLISA.

PDF Details DOI

EAAI Journal 2024 Journal Article

Optimized single-image super-resolution reconstruction: A multimodal approach based on reversible guidance and cyclical knowledge distillation

JingKe Yan
Qin Wang
Yao Cheng
ZhaoYu Su
Fan Zhang
MeiLing Zhong
Lei Liu
Bo Jin

This paper proposes a new approach for reconstructing high-resolution images from low-resolution inputs using Denoising Diffusion Probabilistic Models (DDPMs). Existing DDPMs, while promising, face two issues: one is detail discrepancies due to the uncertain degradation factors in low-resolution images, the other is slow sampling speeds. To address these, a multimodal approach based on reversible guidance and cyclical knowledge distillation (MRKD) is introduced. This method is based on the concept where prior and posterior probabilities can assist in comprehending and predicting future events from available data and information. In the MRKD method, text and image information are separately encoded, and novel constraints are applied on prior and posterior distributions, optimizing the detailed features of the reconstructed image. In addition, due to the uncertainty of degradation factors in low-resolution images, a ‘one-to-many’ mapping issue arises in single-image super-resolution tasks. In response to this, the paper redefines constraints on the posterior distribution using the log-likelihood. Specifically, the Bayesian transformation of the input and output of the observation model is employed to effectively guide the diffusion process. To boost the slow sampling speed of DDPM, a cyclical knowledge distillation strategy is proposed, allowing iterative transfer of learned parameters from a high-step DDPM to a low-step model, thereby accelerating the sampling process while preserving image quality. The experimental results demonstrate that these strategies enable the model to effectively comprehend the high-level semantics and contextual information within images. Additionally, they address challenges associated with mode collapse, the loss of high-frequency details, and the complexities of long-tail data.

AAMAS Conference 2024 Conference Paper

Towards Efficient Auction Design with ROI Constraints

Xinyu Tang
Hongtao Lv
Yingjie Gao
Fan Wu
Lei Liu
Lizhen Cui

Online advertising stands as a significant revenue source of the Internet. Recently, the trend among advertisers tilting towards the use of auto-bidding tools has heralded the emergence of a new model of bidders operating with constraints related to return on investment (ROI). However, most of the current research on ROIconstrained bidders in auction design only focuses on either the ROI constraints or values of bidders being private, while it is more practical to keep them both private in reality. Designing a truthful mechanism for bidders with both private values and ROI constraints introduces complexities because of the characteristics of designing mechanisms with multiple parameters. To remedy this, we divide bidders into binary classes: the traditional utility maximizers (UMs) who can be viewed as having an ROI constraint of 1, and the ROIconstrained bidders (RBs) who share a fixed ROI constraint denoted as 𝛾. This framework retains the essence of multi-parameter mechanism design but transitions this into a more tractable form. Then we introduce a novel auction mechanism, cleverly combining the conventional VCG mechanism and an existing mechanism for public ROI-constrained bidders which is called Cavallo’s mechanism. Our mechanism can achieve an approximation ratio of 3 2 on social welfare. Additionally, we unearth new insights into the limitations posed by ROI constraints. When the ROI constraint 𝛾 exceeds 2, the lower bound of social welfare is 5 4; when it falls below 2, the lower bound becomes 3+𝛾 2+3𝛾−𝛾2.

EAAI Journal 2023 Journal Article

A hybrid Gaussian mutation PSO with search space reduction and its application to intelligent selection of piston seal grooves for homemade pneumatic cylinders

Pengfei Qian
Hui Luo
Lei Liu
Pansong Lv
Chenwei Pu
Deyuan Meng
Luis Miguel Ruiz Páez

To make the motion tracking control of the homemade pneumatic cylinder as accurate as possible, it is necessary to properly match the seal groove and seal ring on the piston to generate the appropriate friction. However, since the relationship between friction and motion control accuracy is not yet clear, and the friction affected by many factors cannot be accurately modeled, it is impossible to design the optimal seal groove with the highest possible motion control accuracy for pneumatic cylinder through theoretical calculation or simulation optimization. For this reason, an experimental optimization method is considered to select the optimal one from the six empirically designed seal grooves through a particle swarm optimization (PSO) algorithm. To cope with the shortcomings of slow convergence rate and the tendency to fall into local optimum of the basic PSO algorithm (BPSO), three improved PSO algorithms (HGMPSO-0, HGMPSO and HGMPSO-SSR) are successively proposed in this study. The first two improved algorithms are compared with other PSO variants on 23 benchmark functions tested, and the results show that HGMPSO has better overall performance. To significantly improve the search space search efficiency and make the algorithm converge faster, the search space contraction mechanism is introduced into the HGMPSO algorithm to form the HGMPSO-SSR algorithm. The experimental results show that the HGMPSO-SSR algorithm significantly outperforms other PSO variants used for comparison and successfully achieves intelligent selection of piston seal grooves for the designed homemade pneumatic cylinder.

EAAI Journal 2023 Journal Article

A novel transfer learning network with adaptive input length selection and lightweight structure for bearing fault diagnosis

Guiting Tang
Cai Yi
Lei Liu
Xingguo Yang
Du Xu
Qiuyang Zhou
Jianhui Lin

In recent years, great progress has been made in intelligent bearing fault diagnosis based on transfer learning (TL). However, the huge number of parameters is ignored when using large convolutional neural network (CNN), and the input length of different bearings are almost not take into account. The high-energy hardware economic cost and time consumption caused by slow operation of large CNN have brought great difficulties to the engineering practice. Therefore, inspired by envelope demodulation and lightweight network signal processing methods, a novel lightweight TL network is proposed, which can adaptively select the input length (IL) and accurately identify the bearing health states under different work conditions. Firstly, an innovative adaptive IL selection strategy considering bearing differences is proposed to replace manually fixed IL. Secondly, a TL network containing group convolution and instance normalization is constructed to make the network lightweight and operate faster. Thirdly, maximum mean discrepancy is introduced to align the feature distribution between source domain and target domain. Lastly, 81 tasks are carried out on the across-domain datasets to validate the practicability of the proposed method. The results between accuracy and lightweight demonstrate that the proposed method is superior to other four state-of-the-art TL CNN, including three TL CNN and a lightweight model, under identical conditions.

TCS Journal 2023 Journal Article

Approximation algorithm for MinSum linear barrier coverage with sink-based mobile sensors on the plane

Wenjie Zou
Longkun Guo
Chunlin Hao
Lei Liu

Emerging wireless and mobile applications, such as border intrusion detection with station-based drones, brought a new barrier coverage problem of using sink-based mobile sensors to cover a given line barrier with minimum energy consumption. In this paper, we focus on the uniform sink-based line barrier coverage (SLBC) problem, in which we are given a line barrier and k sink stations distributed on the plane which can emit an infinite number of sensors with an identical sensing radius. The problem aims to find their final positions on the barrier for the sensors emitted by the stations, such that the total moving distance of the sensors is minimized and each point of the barrier is within the sensing area of at least one sensor. We first observe the geometric structure of an optimal solution that any optimal solution can be considered as a set of intersecting tangent (disk) segments, where a tangent (disk) segment is a sequence of tangent disks. Then, we devise an algorithm to calculate all possible tangent (disk) segments and another algorithm to calculate the near-optimal positions for each of such segments. After computing all tangent (disk) segments and their near-optimal positions, an algorithm is proposed to transform uniform SLBC into an instance of the shortest path problem. It is shown the whole algorithm deserves a runtime O ( k 2 log ⁡ k r ε ) and consumes at most ε more movement than an optimal solution, where ε is any given positive real number, and r and k are the sensor radius and the number of sink stations, respectively.

EAAI Journal 2023 Journal Article

Unsupervised transfer learning for intelligent health status identification of bearing in adaptive input length selection

Guiting Tang
Lei Liu
Yirong Liu
Cai Yi
Yongxu Hu
Du Xu
Qiuyang Zhou
Jianhui Lin

Input length (IL) is an important element in transfer learning (TL) network for intelligent health status identification of bearing (IHSIB). However, fixed IL are used in most studies. In this paper, a TL network via adaptive IL selection module for IHSIB (AILTLN) is proposed, which includes adaptive IL module, feature extractor module, health status identification module, and domain discriminator module. Firstly, an adaptive IL selection module based on envelope spectrum analysis is proposed. The module varies with bearing structure, motor speed, and sampling frequency. Secondly, group convolution, transposed convolution, and instant normalization are constructed in feature extractor. Thirdly, softmax cross-entropy loss function and maximum mean discrepancy are used for health status identification and domain alignment. The TL results of open bearing dataset and high-speed train bearing experiment show that AILTLN is better than the other existing methods in the TL of IHSIB. The ablation study shows the reuse of low-dimensional features and the adaptive IL help to improve to the accuracy of the proposed method.

AAAI Conference 2023 Conference Paper

Utility Maximizer or Value Maximizer: Mechanism Design for Mixed Bidders in Online Advertising

Hongtao Lv
Zhilin Zhang
Zhenzhe Zheng
Jinghan Liu
Chuan Yu
Lei Liu
Lizhen Cui
Fan Wu

Digital advertising constitutes one of the main revenue sources for online platforms. In recent years, some advertisers tend to adopt auto-bidding tools to facilitate advertising performance optimization, making the classical utility maximizer model in auction theory not fit well. Some recent studies proposed a new model, called value maximizer, for auto-bidding advertisers with return-on-investment (ROI) constraints. However, the model of either utility maximizer or value maximizer could only characterize partial advertisers in real-world advertising platforms. In a mixed environment where utility maximizers and value maximizers coexist, the truthful ad auction design would be challenging since bidders could manipulate both their values and affiliated classes, leading to a multi-parameter mechanism design problem. In this work, we address this issue by proposing a payment rule which combines the corresponding ones in classical VCG and GSP mechanisms in a novel way. Based on this payment rule, we propose a truthful auction mechanism with an approximation ratio of 2 on social welfare, which is close to the lower bound of at least 5/4 that we also prove. The designed auction mechanism is a generalization of VCG for utility maximizers and GSP for value maximizers.

PDF Details DOI

JBHI Journal 2022 Journal Article

A Privacy-Preserving Cross-Domain Healthcare Wearables Recommendation Algorithm Based on Domain-Dependent and Domain-Independent Feature Fusion

Xu Yu
Dingjia Zhan
Lei Liu
Hongwu Lv
Lingwei Xu
Junwei Du

Recently, recommender systems are applied to provide personalized recomendation for healthcare wearables. However, due to the sparsity problem, traditional recommendation algorithms are difficult to achieve desired performance. Considering that consumers often buy and rate other types of items on E-commerce platforms, we can leverage significant information in the auxiliary domains to improve the recommendation performance of healthcare wearables, which can be regarded as cross-domain recommendation. However, traditional cross-domain recommendation model cannot fully represent user’s characteristics and fail to consider the leaks of original auxiliary domain ratings during the information transfer process. To overcome the two shortcomings, this paper proposes a Privacy-Preserving Cross-Domain Healthcare Wearables Recommendation algorithm (PPCDHWRec). Firstly, user’s characteristics are divided into domain-dependent features and domain-independent features, which complement each other and fully depict the user’s characteristics. Secondly, inspired by the latent factor model, we factorize the original rating information of each auxiliary domain by Funk-SVD and Orthogonal Nonnegative Matrix Tri-Factorization (ONMTF) model, to obtain user’s domain-dependent and domain-independent features, respectively. Finally, the Factorization Machine algorithm is used to fuse the obtained user’s features with the target domain information to provide the recommendation results. By hiding the item latent factors obtained in the factorization process, PPCDHWRec ensures that the original information cannot be inferred from the transferred user hidden vector. Hence, PPCDHWRec is a privacy-preserving recommendation model. Experiments on two groups of auxiliary domains, having high and low correlations with target domain, show the effectiveness of PPCDHWRec.

AAAI Conference 2022 Conference Paper

Attribute-Based Progressive Fusion Network for RGBT Tracking

Yun Xiao
MengMeng Yang
Chenglong Li
Lei Liu
Jin Tang

RGBT tracking usually suffers from various challenging factors of fast motion, scale variation, illumination variation, thermal crossover and occlusion, to name a few. Existing works often study fusion models to solve all challenges simultaneously, which requires fusion models complex enough and training data large enough, and are usually difficult to be constructed in real-world scenarios. In this work, we disentangle the fusion process via the challenge attributes, and thus propose a novel Attribute-Based Progressive Fusion Network (APFNet) to increase the fusion capacity with a small number of parameters while reducing the dependence on large-scale training data. In particular, we design five attribute-specific fusion branches to integrate RGB and thermal features under the challenges of thermal crossover, illumination variation, scale variation, occlusion and fast motion respectively. By disentangling the fusion process, we can use a small number of parameters for each branch to achieve robust fusion of different modalities and train each branch using the small training subset with the corresponding attribute annotation. Then, to adaptive fuse features of all branches, we design an aggregation fusion module based on SKNet. Finally, we also design an enhancement fusion transformer to strengthen the aggregated feature and modality-specific features. Experimental results on benchmark datasets demonstrate the effectiveness of our APFNet against other state-of-the-art methods.

AAAI Conference 2022 Conference Paper

Cross-Modal Object Tracking: Modality-Aware Representations and a Unified Benchmark

Chenglong Li
Tianhao Zhu
Lei Liu
Xiaonan Si
Zilin Fan
Sulan Zhai

In many visual systems, visual tracking often bases on RGB image sequences, in which some targets are invalid in lowlight conditions, and tracking performance is thus affected significantly. Introducing other modalities such as depth and infrared data is an effective way to handle imaging limitations of individual sources, but multi-modal imaging platforms usually require elaborate designs and cannot be applied in many real-world applications at present. Near-infrared (NIR) imaging becomes an essential part of many surveillance cameras, whose imaging is switchable between RGB and NIR based on the light intensity. These two modalities are heterogeneous with very different visual properties and thus bring big challenges for visual tracking. However, existing works have not studied this challenging problem. In this work, we address the cross-modal object tracking problem and contribute a new video dataset, including 644 cross-modal image sequences with over 478K frames in total, and the average video length is more than 742 frames. To promote the research and development of cross-modal object tracking, we propose a new algorithm, which learns the modality-aware target representation to mitigate the appearance gap between RGB and NIR modalities in the tracking process. It is plugand-play and could thus be flexibly embedded into different tracking frameworks. Extensive experiments on the dataset are conducted, and we demonstrate the effectiveness of the proposed algorithm in two representative tracking frameworks against 19 state-of-the-art tracking methods.

IJCAI Conference 2021 Conference Paper

Few-Shot Partial-Label Learning

Yunfeng Zhao
Guoxian Yu
Lei Liu
Zhongmin Yan
Lizhen Cui
Carlotta Domeniconi

Partial-label learning (PLL) generally focuses on inducing a noise-tolerant multi-class classifier by training on overly-annotated samples, each of which is annotated with a set of labels, but only one is the valid label. A basic promise of existing PLL solutions is that there are sufficient partial-label (PL) samples for training. However, it is more common than not to have just few PL samples at hand when dealing with new tasks. Furthermore, existing few-shot learning algorithms assume precise labels of the support set; as such, irrelevant labels may seriously mislead the meta-learner and thus lead to a compromised performance. How to enable PLL under a few-shot learning setting is an important problem, but not yet well studied. In this paper, we introduce an approach called FsPLL (Few-shot PLL). FsPLL first performs adaptive distance metric learning by an embedding network and rectifying prototypes on the tasks previously encountered. Next, it calculates the prototype of each class of a new task in the embedding network. An unseen example can then be classified via its distance to each prototype. Experimental results on widely-used few-shot datasets demonstrate that our FsPLL can achieve a superior performance than the state-of-the-art methods, and it needs fewer samples for quickly adapting to new tasks.

PDF Details DOI

IJCAI Conference 2021 Conference Paper

IIAS: An Intelligent Insurance Assessment System through Online Real-time Conversation Analysis

Mengdi Zhou
Shuang Peng
Minghui Yang
Nan Li
Hongbin Wang
Li Qiao
Haitao Mi
Zujie Wen

With the development of Chinese medical insurance industry, the amount of claim cases is growing rapidly. Ultimately, more claims necessarily indicate that the insurance company has to spend much time assessing claims and decides how much compensation the claimant should receive, which is a highly professional process that involves many complex operations. Therefore, the insurance assessor's role is essential. However, for the junior assessor often lacking in practical experience, it is not easy to quickly handle such an online procedure. In order to alleviate assessors' cognitive workload, we propose an Intelligent Insurance Assessment System (IIAS) that helps effectively collect claimant information through online real-time conversation analysis. With the assistance of IIAS, the average time cost of the insurance assessment procedure is reduced from 55 minutes to 35 minutes.

PDF Details DOI

JBHI Journal 2020 Journal Article

Dehaze of Cataractous Retinal Images Using an Unpaired Generative Adversarial Network

Yuhao Luo
Kun Chen
Lei Liu
Jicheng Liu
Jianbo Mao
Genjie Ke
Mingzhai Sun

Cataracts are the leading cause of visual impairment worldwide. Examination of the retina through cataracts using a fundus camera is challenging and error-prone due to degraded image quality. We sought to develop an algorithm to dehaze such images to support diagnosis by either ophthalmologists or computer-aided diagnosis systems. Based on the generative adversarial network (GAN) concept, we designed two neural networks: CataractSimGAN and CataractDehazeNet. CataractSimGAN was intended for the synthesis of cataract-like images through unpaired clear retinal images and cataract images. CataractDehazeNet was trained using pairs of synthesized cataract-like images and the corresponding clear images through supervised learning. With two networks trained independently, the number of hyper-parameters was reduced, leading to better performance. We collected 400 retinal images without cataracts and 400 hazy images from cataract patients as the training dataset. Fifty cataract images and the corresponding clear images from the same patients after surgery comprised the test dataset. The clear images after surgery were used for reference to evaluate the performance of our method. CataractDehazeNet was able to enhance the degraded image from cataract patients substantially and to visualize blood vessels and the optic disc, while actively suppressing the artifacts common in application of similar methods. Thus, we developed an algorithm to improve the quality of the retinal images acquired from cataract patients. We achieved high structure similarity and fidelity between processed images and images from the same patients after cataract surgery.

ECAI Conference 2016 Conference Paper

A New Stochastic Local Search Approach for Computing Preferred Extensions of Abstract Argumentation

Dangdang Niu
Lei Liu
Shuai Lü 0001

In this paper, we proposed a new stochastic local search algorithm Inc-CCAEPfor computing the preferred extensions in (abstract) argumentation frameworks (AF). Inc-CCAEPrealizes an incremental version of Swcca, specially designed for computing the preferred extensions in AF. Experiments show that, Inc-CCAEPnotably outperforms the state-of-the-art solvers consistently on random benchmarks with non-empty preferred extensions.

TAAS Journal 2014 Journal Article

A Host-Based Approach for Unknown Fast-Spreading Worm Detection and Containment

Songqing Chen
Lei Liu
Xinyuan Wang
Xinwen Zhang
Zhao Zhang

The fast-spreading worm, which immediately propagates itself after a successful infection, is becoming one of the most serious threats to today’s networked information systems. In this article, we present WormTerminator, a host-based solution for fast Internet worm detection and containment with the assistance of virtual machine techniques based on the fast-worm defining characteristic. In WormTerminator, a virtual machine cloning the host OS runs in parallel to the host OS. Thus, the virtual machine has the same set of vulnerabilities as the host. Any outgoing traffic from the host is diverted through the virtual machine. If the outgoing traffic from the host is for fast worm propagation, the virtual machine should be infected and will exhibit worm propagation pattern very quickly because a fast-spreading worm will start to propagate as soon as it successfully infects a host. To prove the concept, we have implemented a prototype of WormTerminator and have examined its effectiveness against the real Internet worm Linux/Slapper. Our empirical results confirm that WormTerminator is able to completely contain worm propagation in real-time without blocking any non-worm traffic. The major performance cost of WormTerminator is a one-time delay to the start of each outgoing normal connection for worm detection. To reduce the performance overhead, caching is utilized, through which WormTerminator will delay no more than 6% normal outgoing traffic for such detection on average.