Arrow Research search

Author name cluster

Li Liu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

69 papers
2 author rows

Possible papers

69

EAAI Journal 2026 Journal Article

BeltTear-seg: A lightweight model for belt tear segmentation with multi-scale feature squeeze attention and enhanced classification decision

  • Li Yuan
  • Hebin Zhou
  • Yuqi Kong
  • Li Liu
  • Jiangyun Li

To address the challenges of small target tear omission and false detections, complex background interference, and strict real-time requirements in belt tear segmentation for industrial production, this study proposes an improved instance segmentation model based on You Only Look Once version 8 nano (YOLOv8n) for segmentation (YOLOv8n-seg), named BeltTear-seg. First, a Multi-scale Feature Squeeze Attention mechanism (MFSA) is introduced to enhance the model’s capability in capturing small target tear regions, effectively reducing the omission rate. Second, an Enhanced Classification Decision (ECD) layer is incorporated into the model head, working in conjunction with Bidirectional Feature Pyramid Network(BiFPN) to reduce the false detection rate. Finally, a lightweight designed Cross Stage Partial with 2 convolutions and feature fusion (Light-C2f) is introduced to significantly enhance computational efficiency while maintaining high segmentation accuracy. Experimental results demonstrate that the BeltTear-seg model performs exceptionally well on a custom-built belt tear instance segmentation dataset, which includes 4, 050 images (1, 800 negative and 2, 250 positive samples) of conveyor belt surfaces collected from real industrial sites, achieving a classification accuracy of 95. 9%, which represents a 5. 8% improvement over the original model YOLOv8n-seg on the custom dataset. Meanwhile the model’s detection time per frame is shortened by 15. 1%. When compared with other mainstream instance segmentation models, this model also demonstrates significant advantages. The proposed improvements not only significantly enhance the accuracy of the instance segmentation model but also simultaneously boost segmentation speed, thereby meeting the dual industrial requirements of high precision and real-time detection for belt tear inspection.

Details DOI

AAAI Conference 2026 Conference Paper

Boosting ASR Robustness via Test-Time Reinforcement Learning with Audio-Text Semantic Rewards

  • Linghan Fang
  • Tianxin Xie
  • Li Liu

Recently, Automatic Speech Recognition (ASR) systems (e.g., Whisper) have achieved remarkable accuracy improvements but remain highly sensitive to real-world unseen data (data with large distribution shifts), including noisy environments and diverse accents. To address this issue, test-time adaptation (TTA) has shown great potential in improving the model adaptability at inference time without ground-truth labels, and existing TTA methods often rely on pseudo-labeling or entropy minimization. However, by treating model confidence as a learning signal, these methods may reinforce high-confidence errors, leading to confirmation bias that undermines adaptation. To overcome these limitations, we present ASR-TRA, a novel Test-time Reinforcement Adaptation framework inspired by causal intervention. More precisely, our method introduces a learnable decoder prompt and utilizes temperature-controlled stochastic decoding to generate diverse transcription candidates. These are scored by a reward model that measures audio-text semantic alignment, and the resulting feedback is used to update both model and prompt parameters via reinforcement learning. Comprehensive experiments on LibriSpeech with synthetic noise and L2 Arctic accented English datasets demonstrate that our method significantly outperforms existing state-of-the-art (SOTA), including SUTA and SGEM, in both accuracy and inference speed. Ablation studies further confirm the effectiveness of combining audio and language-based rewards, highlighting our method's enhanced stability and interpretability. Overall, our approach provides a practical and robust solution for deploying ASR systems in challenging real-world conditions.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Cueing Without Gapping: Cuer-Independent Cued Speech Recognition Powered by Cross-Cuer Invariant Modeling

  • Fengji Ma
  • Chenxing Li
  • Li Liu

Automatic Cued Speech Recognition (ACSR) is a vital communication system designed to enhance spoken language accessibility for the hearing-impaired by combining lip movements and hand gestures to encode phonemes. Despite its effectiveness, current ACSR methods face significant challenges, including poor generalization to unseen cuers due to the limited scale of CS datasets, which restricts the ability of existing visual encoder to capture cuer-invariant CS visual features. Additionally, previous approaches relying on Connectionist Temporal Classification (CTC) decoding fail to incorporate prior linguistic sequence knowledge, further limiting their performance. To address these issues, we propose a novel Two Auxiliary Modalities guided Cross-cuer Invariant Adaptation method (TACIA), introducing pose and text modalities to help extract cuer-invariant motion and semantic features, thereby improving generalization. In addition, we introduce a Visual-guided Cued Token Prediction (VG-NTP) method, inspired by large language models. This method replaces CTC decoding by incorporating language modeling, leveraging rich linguistic knowledge, including semantics, to address the suboptimal issues present in the CTC decoding process. Extensive experiments demonstrate the superiority of our approach to the state-of-the-art (SOTA) on Chinese and British CS datasets, significantly advancing the accuracy and quality of ACSR systems.

PDF Details DOI

EAAI Journal 2026 Journal Article

Granular-ball based robust representation learning for social recommendation

  • Xiaofei Zhu
  • Shiyan Wu
  • Li Liu
  • Shuyin Xia
  • Yi Wang
  • Guoyin Wang

Social recommendation systems seek to leverage social relationships to mitigate data sparsity and cold-start issues by augmenting user–item interactions. However, existing methods encounter two critical limitations: (1) They predominantly model user–item interactions at a fine-grained granular level of user/item nodes, neglecting the potential coarse-grained collaborative patterns; and (2) They usually suppress noisy edges in social graphs from a single granular perspective, failing to adjust the denoising granularity according to the actual strength of relationships between users. To address these challenges, we propose GBRSR, a novel Granular-ball based Robust Representation Learning framework. Inspired by the “Global-first” cognitive principle, Granular-ball Computing (GBC), which represents data as granular-ball units with geometric significance, has garnered significant attention due to its outstanding performance in many fields. We leverage GBC theory for representation distillation, transferring coarse-grained knowledge to enhance fine-grained node-level representations. In addition, we employs a granular-ball based structure denoising strategy to prune noisy user relationships, while simultaneously alleviating noise in user representations through a diffusion process. Extensive experiments on three real-world benchmark datasets validate the superiority of GBRSR in recommendation accuracy and robustness, particularly under noisy and sparse conditions.

Details DOI

AAAI Conference 2026 Conference Paper

HSA-Net: Hierarchical and Structure-Aware Framework for Efficient and Scalable Molecular Language Modeling

  • Zihang Shao
  • Wentao Lei
  • Lei Wang
  • Wen-Cai Ye
  • Li Liu

Molecular representation learning, a cornerstone for downstream tasks like molecular captioning and molecular property prediction, heavily relies on Graph Neural Networks (GNN). However, GNN suffers from the over-smoothing problem, where node-level features collapse in deep GNN layers. While existing feature projection methods with cross-attention have been introduced to mitigate this issue, they still perform poorly in deep features. This motivated our exploration of using Mamba as an alternative projector for its ability to handle complex sequences. However, we observe that while Mamba excels at preserving global topological information from deep layers, it neglects fine-grained details in shallow layers. The capabilities of Mamba and cross-attention exhibit a global-local trade-off. To resolve this critical global-local trade-off, we propose Hierarchical and Structure-Aware Network (HSA-Net), a novel framework with two modules that enables a hierarchical feature projection and fusion. Firstly, a Hierarchical Adaptive Projector (HAP) module is introduced to process features from different graph layers. It learns to dynamically switch between a cross-attention projector for shallow layers and a structure-aware Graph-Mamba projector for deep layers, producing high-quality, multi-level features. Secondly, to adaptively merge these multi-level features, we design a Source-Aware Fusion (SAF) module, which flexibly selects fusion experts based on the characteristics of the aggregation features, ensuring a precise and effective final representation fusion. Extensive experiments demonstrate that our HSA-Net framework quantitatively and qualitatively outperforms current state-of-the-art (SOTA) methods.

PDF Details DOI

EAAI Journal 2026 Journal Article

Identifying non-small cell lung cancer subtypes by a hybrid representative causal network with computed tomography images

  • Li Liu
  • Xueying Wang
  • Shanshan Huang
  • Zhengqiao Deng
  • Shu Wang
  • Guang Wu
  • Donglai Yang
  • Sixi Zha

Identifying representative causal features from computed tomography (CT) images remains a significant challenge for the subtype classification of non-small cell lung cancer (NSCLC). Existing methods, whether based on radiomics or deep neural networks, often overlook the intricate causal relationships among features, thereby yielding suboptimal or even detrimental diagnostic outcomes. To bridge this gap, we propose a Hybrid Representative Causal Network (HRCL) for NSCLC subtype identification, which explicitly captures the local causal relationships inherent in the interaction between radiomics and features based on deep learning from a holistic perspective. Specifically, a causal network structure is learned to delineate the unique causal configuration of distinct NSCLC subtypes through a variable number of nodes and links. The resultant network adheres to the causal Markov property, thereby ensuring global consistency of all local cause–effect dependencies. Moreover, a hybrid representative feature selector is designed to identify the most salient causal features from the causal network for precise NSCLC subtype classification. Our method achieves an accuracy of 83. 7% on the publicly available P-NSCLC dataset and 90. 3% on the privately collected I-NSCLC dataset. The empirical evaluations demonstrate that our model significantly outperforms the state-of-the-art methods.

Details DOI

EAAI Journal 2026 Journal Article

Multi-feature recognition of weld defects with an ultrasonic signal-image-joint machine learning fusion model

  • Jiaxin Hu
  • Dengcai Zhao
  • Xiaochuan Liu
  • Li Liu
  • Siwen Lei
  • Changmao Li
  • Jingqi Yang
  • Jinggang Qin

The qualitative identification of weld defects is crucial for the safe use and life assessment of in-service welded components. Although intelligent defect recognition using machine learning has gained significant attention, existing approaches often suffer from limited defect types, small datasets, and reliance on single-feature sources. Consequently, the recognition accuracy remains insufficient for real-world industrial applications, especially when encountering new and complex defect patterns. To address these challenges, this study proposes a signal-image-joint machine learning fusion model (SIJ-MLF) that classifies weld defects using large-scale data collected from austenitic stainless steel weld samples. The method integrates multiple features extracted from both defect pulse signals and images, adaptive sliding window and feature selection techniques are introduced to further enhance recognition performance. Comparative analyses demonstrate that the proposed approach achieves a stable accuracy of up to 96% when tested on completely new practical samples. Moreover, an intelligent recognition software has been developed based on this work, illustrating its potential for reliable and efficient industrial inspection.

Details DOI

AAAI Conference 2026 Conference Paper

Multi-View Differential Mixing and Graph-Guided Structural Region Selection for Cross-Modal Alignment

  • Linlin Ji
  • Li Liu

Cross-modal alignment is a promising yet challenging task in multimodal learning. Existing methods typically assess it by measuring the cross-modal semantic similarity from both global and local perspectives. However, these methods often neglect their potential interdependence. Specifically, global matching methods suffer from the over-compression of local features, while local matching methods rarely consider the inherent spatial topology of image patches. To address these limitations, we propose MG-Net, a unified framework with two collaborative modules: Multi-View Differential Mixer (MDM) and Graph-Guided Structural Region Selector (GSRS). The MDM is designed to capture discriminative global representations. It generates a series of views by decomposing feature vectors through multi-order differential operations, and adaptively fuses them via a lightweight Mixture-of-Experts (MoE) network. Meanwhile, the GSRS organizes image patches as a spatial graph and employs text-guided contextual reasoning to select spatially coherent and semantically complete structural regions. Extensive experiments on the Flickr30K and MS-COCO benchmarks demonstrate that the proposed MG-Net outperforms state-of-the-art methods in most cases.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Strip R-CNN: Large Strip Convolution for Remote Sensing Object Detection

  • Xinbin Yuan
  • Zhaohui Zheng
  • Yuxuan Li
  • Xialei Liu
  • Li Liu
  • Xiang Li
  • Qibin Hou
  • Ming-Ming Cheng

In this paper, we show that current approaches using large square kernels or transformer-based global modeling aggregate contextual information uniformly across spatial dimensions, leading to feature dilution and localization errors for elongated targets. To mitigate this issue, we propose Strip R-CNN, the first work to systematically explore large strip convolutions for remote sensing object detection. Our key insight is that strip convolutions enable directional feature aggregation along the dominant spatial dimension of slender objects, reducing background interference while preserving essential geometric information. We design two core components: (i) StripNet, a backbone network employing sequential orthogonal large strip convolutions to capture anisotropic spatial patterns, and (ii) Strip Head, which enhances localization precision by incorporating strip convolutions into the detection head. Unlike previous large-kernel approaches that suffer from computational redundancy and isotropic limitations, our method achieves superior performance with remarkable efficiency. Extensive experiments on multiple benchmarks (DOTA, FAIR1M, HRSC2016, and DIOR) demonstrate significant improvements, with our 30M parameter model achieving 82.75% mAP on DOTA-v1.0, establishing a new state-of-the-art record while providing new insights into anisotropic feature learning for remote sensing applications.

PDF Details DOI

AAAI Conference 2026 Conference Paper

TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability

  • Fengji Ma
  • Hei Victor Cheng
  • Chenxing Li
  • Li Liu

Achieving zero-shot adversarial robustness without sacrificing generalization remains challenging for foundation models such as CLIP, especially under large adversarial perturbations. Through empirical analyses, we identify three critical yet overlooked issues: (1) Logit margins exhibit a stable offset between small and large adversarial perturbations, suggesting that explicitly adjusting margins could improve robustness against unseen large perturbations. (2) A significant negative correlation exists between logit margin and inter-class semantic similarity, indicating that semantic structures are insufficiently leveraged by existing methods. (3) Existing methods for adjusting text embeddings disrupt the intrinsic semantic consistency established by pre-trained models, undermining generalization capability. Motivated by these findings, we propose a novel Text-Image Mutual Awareness (TIMA) framework, including a Text-Aware Image (TAI) tuning module with an Adaptive Semantic-Aware Margin (ASAM) to explicitly calibrate logit margins, and an Image-Aware Text (IAT) tuning module with Semantic Consistent Minimum Hyperspherical Energy (SC-MHE) to preserve semantic consistency. Comprehensive experiments validate that TIMA significantly outperforms existing approaches by effectively addressing the identified limitations.

PDF Details DOI

AAAI Conference 2026 Conference Paper

UniCUE: Unified Recognition and Generation Framework for Chinese Cued Speech Video-to-Speech Generation

  • Jinting Wang
  • Shan Yang
  • Chenxing Li
  • Dong Yu
  • Li Liu

Cued Speech (CS) enhances lipreading via hand coding, offering visual phonemic cues that support precise speech perception for the hearing-impaired. The task of CS Video-to-Speech generation (CSV2S) aims to convert CS videos into intelligible speech signals. Most existing research focuses on CS Recognition (CSR), which transcribes video content into text. Consequently, a common solution for CSV2S is to integrate CSR with a text-to-speech (TTS) system. However, this pipeline relies on text as an intermediate medium, which may lead to error propagation and temporal misalignment between speech and CS video dynamics. In contrast, directly generating audio speech from CS video (direct CSV2S) often suffer from the inherent multimodal complexity and the limited availability of CS data. To address these challenges, we propose UniCUE, the first unified framework for CSV2S that directly generates speech from CS videos without relying on intermediate text. The core innovation of UniCUE lies in integrating a understanding task (CSR) that provides fine-grained CS visual-semantic cues to to guide the speech generation. Specifically, UniCUE incorporates a pose-aware visual processor, a semantic alignment pool that enables precise visual–semantic mapping, and a VisioPhonetic adapter to bridge the understanding and generation tasks within a unified architecture. To support this framework, we construct UniCUE-HI, a large-scale Mandarin CS dataset containing 11,282 videos from 14 cuers, including both hearing-impaired and normal-hearing individuals. Extensive experiments conducted on this dataset demonstrate that UniCUE achieves state-of-the-art (SOTA) performance across multiple evaluation metrics.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

BackdoorDM: A Comprehensive Benchmark for Backdoor Learning on Diffusion Model

  • Weilin Lin
  • Nanjun Zhou
  • Yanyun Wang
  • Jianze Li
  • Hui Xiong
  • Li Liu

Backdoor learning is a critical research topic for understanding the vulnerabilities of deep neural networks. While the diffusion model (DM) has been broadly deployed in public over the past few years, the understanding of its backdoor vulnerability is still in its infancy compared to the extensive studies in discriminative models. Recently, many different backdoor attack and defense methods have been proposed for DMs, but a comprehensive benchmark for backdoor learning on DMs is still lacking. This absence makes it difficult to conduct fair comparisons and thoroughly evaluate existing approaches, thus hindering future research progress. To address this issue, we propose BackdoorDM, the first comprehensive benchmark designed for backdoor learning on DMs. It comprises nine state-of-the-art (SOTA) attack methods, four SOTA defense strategies, and three useful visualization analysis tools. We first systematically classify and formulate the existing literature in a unified framework, focusing on three different backdoor attack types and five backdoor target types, which are restricted to a single type in discriminative models. Then, we systematically summarize the evaluation metrics for each type and propose a unified backdoor evaluation method based on multimodal large language model (MLLM). Finally, we conduct a comprehensive evaluation and highlight several important conclusions. We believe that BackdoorDM will help overcome current barriers and contribute to building a trustworthy artificial intelligence generated content (AIGC) community. The codes are released in https: //github. com/linweiii/BackdoorDM.

PDF Details

AAAI Conference 2025 Conference Paper

Decomposing and Fusing Intra- and Inter-Sensor Spatio-Temporal Signal for Multi-Sensor Wearable Human Activity Recognition

  • Haoyu Xie
  • Haoxuan Li
  • Chunyuan Zheng
  • Haonan Yuan
  • Guorui Liao
  • Jun Liao
  • Li Liu

Wearable Human Activity Recognition (WHAR) is a prominent research area within ubiquitous computing. Multi-sensor synchronous measurement has proven to be more effective for WHAR than using a single sensor. However, existing WHAR methods use shared convolutional kernels for indiscriminate temporal feature extraction across each sensor variable, which fails to effectively capture spatio-temporal relationships of intra-sensor and inter-sensor variables. We propose the DecomposeWHAR model consisting of a decomposition phase and a fusion phase to better model the relationships between modality variables. The decomposition creates high-dimensional representations of each intra-sensor variable through the improved Depth Separable Convolution to capture local temporal features while preserving their unique characteristics. The fusion phase begins by capturing relationships between intra-sensor variables and fusing their features at both the channel and variable levels. Long-range temporal dependencies are modeled using the State Space Model (SSM), and later cross-sensor interactions are dynamically captured through a self-attention mechanism, highlighting inter-sensor spatial correlations. Our model demonstrates superior performance on three widely used WHAR datasets, significantly outperforming state-of-the-art models while maintaining acceptable computational efficiency.

PDF Details DOI

YNIMG Journal 2025 Journal Article

Dynamic cycles between brain states during creative storytelling

  • Xitong Liang
  • Mingnan Cai
  • Gaohan Jing
  • Chengming Zhang
  • Emily Sophia Nichols
  • Li Liu

Many theories suggest that creative thinking involves a dynamic transition between different mental states, yet empirical evidence supporting this notion remains scarce. The dual process model proposes that spontaneous thinking and deliberate thinking drive the dwell in and the transitions between different mental states during creative thinking, but there is a debate over whether the two types of thinking operate in parallel or in sequence. To address these gaps, we conducted a functional magnetic resonance imaging (fMRI) study in 41 college students during a creative storytelling task. We then compared the dynamic brain states in creative versus uncreative storytelling to identify key brain states associated with creative thinking. And we further performed correlation analysis between these key brain states with performance of various creative tasks, trying to link the key brain states with different cognitive processes. The results showed that two key brain states are associated with creative thinking, with one involving whole-brain synchronization and the other involving the synchronization of four networks, including the default mode network and the control network. The transition patterns between the key brain states provide tentative evidence for dynamic circulation between different mental states during creative storytelling. Using a deep learning approach, we demonstrate an alternating interaction between spontaneous and deliberate thinking, driving dwelling in and the transitions between different brain states. These findings deepen our understanding of the cognitive and neural mechanisms underlying creative thinking.

Details DOI

EAAI Journal 2025 Journal Article

Fast and intelligent measurement of the ventilation resistance coefficient for the whole mine based on sparse measurement points

  • Dong Wang
  • Jian Liu
  • Lijun Deng
  • Peng Cao
  • Li Liu

Artificial intelligence is playing an important role in mine ventilation engineering, especially in ensuring safe mine production. The mine ventilation resistance coefficient (MVRC) is the core and basic parameter of a mine ventilation system. It is crucial to quickly and accurately obtain the ventilation resistance coefficient (VRC) of the whole mine for the scientific, safe, and intelligent management of the mine ventilation system. To solve the time-consuming and laborious problem of the traditional mine ventilation resistance measurement method, we propose a fast and intelligent measurement method to obtain the whole mine's VRC based on an artificial intelligence differential evolution algorithm and sparse measurement points. The VRC was experimentally measured to verify the validity of the intelligent measurement method and the reliability of the model. The relative error of the air volume at the observation points of the solved results is less than 6 %. The fast intelligent measurement of the MVRC of the Longshou mine was carried out. The results were applied to develop an emergency plan for addressing the insufficient air supply in the ventilation system caused by the collapse of the mine's main blind return shaft and validated through engineering practice. After field practice, the relative error between the predicted and tested air return volume of the 10-row inclined shaft was 3. 23 %. It is verified that the results obtained using this method can solve mine ventilation system problems with relatively high accuracy, significantly reducing both the testing workload and time required for the mine ventilation resistance measurements.

Details DOI

AAAI Conference 2025 Conference Paper

Few-Shot Audio-Visual Class-Incremental Learning with Temporal Prompting and Regularization

  • Yawen Cui
  • Li Liu
  • Zitong Yu
  • Guanjie Huang
  • Xiaopeng Hong

Audio-Visual Learning (AVL) aims at the audio-visual perception with both audio and vision modalities. AVL also suffers from data insufficiency in many applications as with other unimodal tasks. Concurrently, AVL often needs to continuously learn over time rather than all knowledge simultaneously. Considering the above two perspectives, our work mainly focuses on benchmarking the unexplored Few-Shot Audio-Visual Class-Incremental Learning (FS-AVCIL), i.e., continually perceiving novel categories described by a limited number of labeled examples with audio and visual modalities. Firstly, we provide the detailed task configuration together with a thorough analysis of the challenges in FS-AVCIL: (1) how to efficiently learn and fuse multimodal information with limited labeled examples; and (2) how to alleviate catastrophic forgetting cross-modal semantic correlations with limited data. Then, we propose an efficient framework based on Vision Transformer to solve FS-AVCIL. This framework contains two parts: temporal-residual prompting for audio-visual synergy adapter and temporal prompt regularization. Specifically, temporal-residual prompting is incorporated into the audio-visual adapter to efficiently finetune the pre-trained foundation model with limited data and capture audio-visual correlation by learning temporal-relevant prompts. Besides, we regularize temporal-relevant prompts to memorize previous knowledge by fully using the temporal knowledge from various perspectives. This framework is validated in audio-visual classification tasks under the FS-AVCIL scenario, and extensive experiments demonstrate its superior performance.

PDF Details DOI

AAAI Conference 2025 Conference Paper

Fusing Pruned and Backdoored Models: Optimal Transport-based Data-free Backdoor Mitigation

  • Weilin Lin
  • Li Liu
  • Jianze Li
  • Hui Xiong

Backdoor attacks present a serious security threat to deep neuron networks (DNNs). Although numerous effective defense techniques have been proposed in recent years, they inevitably rely on the availability of either clean or poisoned data. In contrast, data-free defense techniques have evolved slowly and still lag significantly in performance. To address this issue, different from the traditional approach of pruning followed by fine-tuning, we propose a novel data-free defense method named Optimal Transport-based Backdoor Repairing (OTBR) in this work. This method, based on our findings on neuron weight changes (NWCs) of random unlearning, uses optimal transport (OT)-based model fusion to combine the advantages of both pruned and backdoored models. Specifically, we first demonstrate our findings that the NWCs of random unlearning are positively correlated with those of poison unlearning. Based on this observation, we propose a random-unlearning NWC pruning technique to eliminate the backdoor effect and obtain a backdoor-free pruned model. Then, motivated by the OT-based model fusion, we propose the pruned-to-backdoored OT-based fusion technique, which fuses pruned and backdoored models to combine the advantages of both, resulting in a model that demonstrates high clean accuracy and a low attack success rate. To our knowledge, this is the first work to apply OT and model fusion techniques to backdoor defense. Extensive experiments show that our method successfully defends against all seven backdoor attacks across three benchmark datasets, outperforming both state-of-the-art (SOTA) data-free and data-dependent methods.

PDF Details DOI

AAAI Conference 2025 Conference Paper

HiPoser: 3D Human Pose Estimation with Hierarchical Shared Learning at Parts-Level Using Inertial Measurement Units

  • Guorui Liao
  • Chunyuan Zheng
  • Li Cheng
  • Haoyu Xie
  • Shanshan Huang
  • Jun Liao
  • Haoxuan Li
  • Li Liu

This paper considers the challenging problem of 3D Human Pose Estimation (HPE) from a sparse set of Inertial Measurement Units (IMUs). Existing efforts typically reconstruct a pose sequence by either directly tackling whole-body motions or focusing on distinctive spatio-temporal features of local body parts. Unfortunately, these methods ignore existing interdependent motor synergies amongst body parts, which may lead to pose estimation with ambiguous local parts. This observation motivates us to propose a hierarchical learning-based approach, HiPoser, which utilizes a hierarchical shared structure using Mamba blocks as the backbone to focus on the following estimation tasks, involving: 1) torso pose, 2) lower limbs pose, 3) upper limbs pose, and finally 4) global translation. These tasks selectively incorporate body motion states and are to be carried out sequentially in reconstructing part-based poses, which are amalgamated to estimate the final full-body pose with the global translation that satisfies inter-part consistencies. Our hierarchical structure allows HiPoser the flexibility in prioritizing different aspects of pose estimation, to emphasize more on detail or stability. Empirical evaluations over three benchmark datasets demonstrate the superiority of HiPoser over existing state-of-the-art models, suggesting that analyzing the synergistic movement of body parts is indeed important for advancing IMU-based 3D HPE.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

Luminance-Aware Statistical Quantization: Unsupervised Hierarchical Learning for Illumination Enhancement

  • Derong Kong
  • Zhixiong Yang
  • Shengxi Li
  • Shuaifeng Zhi
  • Li Liu
  • Zhen Liu
  • Jingyuan Xia

Low-light image enhancement (LLIE) faces persistent challenges in balancing reconstruction fidelity with cross-scenario generalization. While existing methods predominantly focus on deterministic pixel-level mappings between paired low/normal-light images, they often neglect the continuous physical process of luminance transitions in real-world environments, leading to performance drop when normal-light references are unavailable. Inspired by empirical analysis of natural luminance dynamics revealing power-law distributed intensity transitions, this paper introduces Luminance-Aware Statistical Quantification (LASQ), a novel framework that reformulates LLIE as a statistical sampling process over hierarchical luminance distributions. Our LASQ re-conceptualizes luminance transition as a power-law distribution in intensity coordinate space that can be approximated by stratified power functions, therefore, replacing deterministic mappings with probabilistic sampling over continuous luminance layers. A diffusion forward process is designed to autonomously discover optimal transition paths between luminance layers, achieving unsupervised distribution emulation without normal-light references. In this way, it considerably improves the performance in practical situations, enabling more adaptable and versatile light restoration. This framework is also readily applicable to cases with normal-light references, where it achieves superior performance on domain-specific datasets alongside better generalization-ability across non-reference datasets. The code is available at: https: //github. com/XYLGroup/LASQ.

PDF Details

EAAI Journal 2025 Journal Article

Open-pit mine occlusion object detection for unmanned transport vehicles

  • Chao Zheng
  • Guoxing Bai
  • Yu Meng
  • Lu Wang
  • Xianyao Jiang
  • Li Liu

Accurate object recognition in open-pit mine environments is crucial for the safety of autonomous transport vehicles. Existing autonomous driving perception mostly focuses on urban structured road traffic, and it is hard to adapt to the challenging open-pit mine environment. Lacking of datasets further limits the development of the specific work. In this paper, we propose an object detection dataset for open-pit mine autonomous driving applications. This dataset encompasses data from several mines and includes different periods such as day, dusk, and night. It provides detailed annotations for diverse objects in the open-pit mines and incorporates additional attributes for evaluating occlusion detection. In addition, to address the multi-scale changes of objects in open-pit mines and the occlusion problems caused by dust, we propose a novel occlusion mine object general distribution detection method, utilizing soft labels and vehicle attribute location to reduce the positioning ambiguity in difficult backgrounds and achieve specific object detection in harsh open-pit mine environments. Our work explores the benchmark for open-pit mine object recognition involving occlusion. Comparison with mainstream techniques on the benchmark demonstrates that our approach outperforms existing state-of-the-art methods and can achieve 82. 2%, 81. 7%, and 76. 7% average precision in easy, moderate, and hard modes, respectively.

Details DOI

AAAI Conference 2025 Conference Paper

Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion

  • Yan Rong
  • Li Liu

Face-based Voice Conversion (FVC) is a novel task that leverages facial images to generate the target speaker's voice style. Previous work has two shortcomings: (1) suffering from obtaining facial embeddings that are well-aligned with the speaker's voice identity information, and (2) inadequacy in decoupling content and speaker identity information from the audio input. To address these issues, we present a novel FVC method, Identity-Disentanglement Face-based Voice Conversion (ID-FaceVC), which overcomes the above two limitations. More precisely, we propose an Identity-Aware Query-based Contrastive Learning (IAQ-CL) module to extract speaker-specific facial features, and a Mutual Information-based Dual Decoupling (MIDD) module to purify content features from audio, ensuring clear and high-quality voice conversion. Besides, unlike prior works, our method can accept either audio or text inputs, offering controllable speech generation with adjustable emotional tone and speed. Extensive experiments demonstrate that ID-FaceVC achieves state-of-the-art performance across various metrics, with qualitative and user study results confirming its effectiveness in naturalness, similarity, and diversity.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

Traversal Verification for Speculative Tree Decoding

  • Yepeng Weng
  • Qiao Hu
  • Xujie Chen
  • Li Liu
  • Dianwen Mei
  • Huishi Qiu
  • Jiang Tian
  • Zhongchao Shi

Speculative decoding is a promising approach for accelerating large language models. The primary idea is to use a lightweight draft model to speculate the output of the target model for multiple subsequent timesteps, and then verify them in parallel to determine whether the drafted tokens should be accepted or rejected. To enhance acceptance rates, existing frameworks typically construct token trees containing multiple candidates in each timestep. However, their reliance on token-level verification mechanisms introduces two critical limitations: First, the probability distribution of a sequence differs from that of individual tokens, leading to suboptimal acceptance length. Second, current verification schemes begin from the root node and proceed layer by layer in a top-down manner. Once a parent node is rejected, all its child nodes should be discarded, resulting in inefficient utilization of speculative candidates. This paper introduces Traversal Verification, a novel speculative decoding algorithm that fundamentally rethinks the verification paradigm through leaf-to-root traversal. Our approach considers the acceptance of the entire token sequence from the current node to the root, and preserves potentially valid subsequences that would be prematurely discarded by existing methods. We theoretically prove that the probability distribution obtained through Traversal Verification is identical to that of the target model, guaranteeing lossless inference while achieving substantial acceleration gains. Experimental results on various models and multiple tasks demonstrate that our method consistently improves acceptance length and throughput over token-level verification.

PDF Details

EAAI Journal 2024 Journal Article

A survey of causal discovery based on functional causal model

  • Lei Wang
  • Shanshan Huang
  • Shu Wang
  • Jun Liao
  • Tingpeng Li
  • Li Liu

Causal discovery finds widespread applications, ranging from estimating treatment effectiveness in medicine, analyzing policy impacts in economics, to constructing predictive models in machine learning—all of which rely on the study and discovery of causal relationships. In recent years, as causal learning has progressed, causal discovery has been classified into different categories depending on assumptions and learning strategies. In this paper, we undertake an exploration of causal discovery methods based on the functional causal model (FCM). We commence by introducing essential terminology associated with causal discovery and laying out the foundational assumptions underpinning FCM-based methods. Following this, we conduct a comprehensive exploration of classical FCM algorithms that have gained prominence in recent years. Furthermore, we scrutinize the performance of these FCM methods across a selection of benchmark datasets. Finally, we deliberate on unresolved issues within this category of methodologies and outline potential avenues for future research.

Details DOI

IJCAI Conference 2024 Conference Paper

Bridge to Non-Barrier Communication: Gloss-Prompted Fine-Grained Cued Speech Gesture Generation with Diffusion Model

  • Wentao Lei
  • Li Liu
  • Jun Wang

Cued Speech (CS) is an advanced visual phonetic encoding system that integrates lip reading with hand codings, enabling people with hearing impairments to communicate efficiently. CS video generation aims to produce specific lip and gesture movements of CS from audio or text inputs. The main challenge is that given limited CS data, we strive to simultaneously generate fine-grained hand and finger movements, as well as lip movements, meanwhile the two kinds of movements need to be asynchronously aligned. Existing CS generation methods are fragile and prone to poor performance due to template-based statistical models and careful hand-crafted pre-processing to fit the models. Therefore, we propose a novel Gloss-prompted Diffusion-based CS Gesture generation framework (called GlossDiff). Specifically, to integrate additional linguistic rules knowledge into the model. we first introduce a bridging instruction called Gloss, which is an automatically generated descriptive text to establish a direct and more delicate semantic connection between spoken language and CS gestures. Moreover, we first suggest rhythm is an important paralinguistic feature for CS to improve the communication efficacy. Therefore, we propose a novel Audio-driven Rhythmic Module (ARM) to learn rhythm that matches audio speech. Moreover, in this work, we design, record, and publish the first Chinese CS dataset with four CS cuers. Extensive experiments demonstrate that our method quantitatively and qualitatively outperforms current state-of-the-art (SOTA) methods. We will release the code and data at glossdiff. github. io/.

PDF Details DOI

EAAI Journal 2024 Journal Article

Passable area segmentation for open-pit mine road from vehicle perspective

  • Chao Zheng
  • Li Liu
  • Yu Meng
  • Meijun Wang
  • Xianyao Jiang

Recognition of passable areas of mine roads based on vehicle perspective is crucial for autonomous vehicles to drive in unmanned open-pit mine scenes. In the past few years, deep learning-based passable area recognition methods have been proven feasible. However, these works have mainly focused on structured urban road environments. Few works designed for mine road region detection due to dataset scarcity. In this article, we propose the mine road scapes segmentation dataset, which is collected from open-pit mine road environments. The dataset is collected from different mine areas, covers different daytime and dusk periods, and annotates the road and non-road areas in detail. Besides, a novel transformer-based road segmentation network is proposed, which is designed for the mine road characteristics of texture and distribution. The segmentation network is composed of an improved transformer feature extraction backbone, which can simultaneously aggregate global context and local detail information, and realize pixel-level recognition. We solve the problem of mine road segmentation benchmark under vehicle perspective. Moreover, by comparing the results of current mainstream methods on the mine road dataset, our method obtains state-of-the-art performance.

Details DOI

EAAI Journal 2024 Journal Article

Recognizing wearable upper-limb rehabilitation gestures by a hybrid multi-feature neural network

  • Shu Wang
  • Jiawei Liu
  • Shen Chen
  • Shanshan Wang
  • Yuxin Peng
  • Changbo Liao
  • Li Liu

Stroke remains a leading cause of disability, presenting significant challenges to individuals and society. The post-stroke rehabilitation process demands prolonged professional training and evaluation. To tackle the issue of limited resources hindering patients’ access to frequent assessments and to facilitate personalized rehabilitation treatment, wearable technology has emerged as a promising solution. However, current wearable-based approaches often rely solely on raw sensor data fed directly into deep neural networks, which may not effectively capture intricate temporal relationships unless they incorporate the knowledge typically employed in clinical analysis. In this study, we introduce a hybrid multi-feature neural network that combines manually designed features, commonly utilized in clinical analysis, with latent features generated by deep networks. By explicitly considering the motion context and spatio-temporal relations among multiple body parts in the upper limb, our model can accurately detect their real-time motions. Empirical evaluations on our proprietary dataset reveal that the accuracy for subject-dependent and subject-independent experiments on 8 coarse-grained actions is 0. 9849 and 0. 9871, respectively, while for 24 fine-grained actions, they are 0. 9724 and 0. 9829, respectively. These results indicate that our model exhibits superior performance compared to other methods, contributing to the advancement of stroke rehabilitation and personalized therapy utilizing wearable systems.

Details DOI

NeurIPS Conference 2024 Conference Paper

Right this way: Can VLMs Guide Us to See More to Answer Questions?

  • Li Liu
  • Diji Yang
  • Sijia Zhong
  • Kalyana S. Tholeti
  • Lei Ding
  • Yi Zhang
  • Leilani H. Gilpin

In question-answering scenarios, humans can assess whether the available information is sufficient and seek additional information if necessary, rather than providing a forced answer. In contrast, Vision Language Models (VLMs) typically generate direct, one-shot responses without evaluating the sufficiency of the information. To investigate this gap, we identify a critical and challenging task in the Visual Question Answering (VQA) scenario: can VLMs indicate how to adjust an image when the visual information is insufficient to answer a question? This capability is especially valuable for assisting visually impaired individuals who often need guidance to capture images correctly. To evaluate this capability of current VLMs, we introduce a human-labeled dataset as a benchmark for this task. Additionally, we present an automated framework that generates synthetic training data by simulating ``where to know'' scenarios. Our empirical results show significant performance improvements in mainstream VLMs when fine-tuned with this synthetic data. This study demonstrates the potential to narrow the gap between information assessment and acquisition in VLMs, bringing their performance closer to humans.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

SARDet-100K: Towards Open-Source Benchmark and ToolKit for Large-Scale SAR Object Detection

  • Yuxuan Li
  • Xiang Li
  • Weijie Li
  • Qibin Hou
  • Li Liu
  • Ming-Ming Cheng
  • Jian Yang

Synthetic Aperture Radar (SAR) object detection has gained significant attention recently due to its irreplaceable all-weather imaging capabilities. However, this research field suffers from both limited public datasets (mostly comprising <2K images with only mono-category objects) and inaccessible source code. To tackle these challenges, we establish a new benchmark dataset and an open-source method for large-scale SAR object detection. Our dataset, SARDet-100K, is a result of intense surveying, collecting, and standardizing 10 existing SAR detection datasets, providing a large-scale and diverse dataset for research purposes. To the best of our knowledge, SARDet-100K is the first COCO-level large-scale multi-class SAR object detection dataset ever created. With this high-quality dataset, we conducted comprehensive experiments and uncovered a crucial challenge in SAR object detection: the substantial disparities between the pretraining on RGB datasets and finetuning on SAR datasets in terms of both data domain and model structure. To bridge these gaps, we propose a novel Multi-Stage with Filter Augmentation (MSFA) pretraining framework that tackles the problems from the perspective of data input, domain transition, and model migration. The proposed MSFA method significantly enhances the performance of SAR object detection models while demonstrating exceptional generalizability and flexibility across diverse models. This work aims to pave the way for further advancements in SAR object detection. The dataset and code is available at \url{https: //github. com/zcablii/SARDet_100K}.

PDF Details DOI

AAAI Conference 2024 Conference Paper

SENCR: A Span Enhanced Two-Stage Network with Counterfactual Rethinking for Chinese NER

  • Hang Zheng
  • Qingsong Li
  • Shen Chen
  • Yuxuan Liang
  • Li Liu

Recently, lots of works that incorporate external lexicon information into character-level Chinese named entity recognition(NER) to overcome the lackness of natural delimiters of words, have achieved many advanced performance. However, obtaining and maintaining high-quality lexicons is costly, especially in special domains. In addition, the entity boundary bias caused by high mention coverage in some boundary characters poses a significant challenge to the generalization of NER models but receives little attention in the existing literature. To address these issues, we propose SENCR, a Span Enhanced Two-stage Network with Counterfactual Rethinking for Chinese NER, that contains a boundary detector for boundary supervision, a convolution-based type classifier for better span representation and a counterfactual rethinking(CR) strategy for debiased boundary detection in inference. The proposed boundary detector and type classifier are jointly trained with the same contextual encoder and then the trained boundary detector is debiased by our proposed CR strategy without modifying any model parameters in the inference stage. Extensive experiments on four Chinese NER datasets show the effectiveness of our proposed approach.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Unveiling and Mitigating Backdoor Vulnerabilities based on Unlearning Weight Changes and Backdoor Activeness

  • Weilin Lin
  • Li Liu
  • Shaokui Wei
  • Jianze Li
  • Hui Xiong

The security threat of backdoor attacks is a central concern for deep neural networks (DNNs). Recently, without poisoned data, unlearning models with clean data and then learning a pruning mask have contributed to backdoor defense. Additionally, vanilla fine-tuning with those clean data can help recover the lost clean accuracy. However, the behavior of clean unlearning is still under-explored, and vanilla fine-tuning unintentionally induces back the backdoor effect. In this work, we first investigate model unlearning from the perspective of weight changes and gradient norms, and find two interesting observations in the backdoored model: 1) the weight changes between poison and clean unlearning are positively correlated, making it possible for us to identify the backdoored-related neurons without using poisoned data; 2) the neurons of the backdoored model are more active ( i. e. , larger gradient norm) than those in the clean model, suggesting the need to suppress the gradient norm during fine-tuning. Then, we propose an effective two-stage defense method. In the first stage, an efficient Neuron Weight Change (NWC)-based Backdoor Reinitialization is proposed based on observation 1). In the second stage, based on observation 2), we design an Activeness-Aware Fine-Tuning to replace the vanilla fine-tuning. Extensive experiments, involving eight backdoor attacks on three benchmark datasets, demonstrate the superior performance of our proposed method compared to recent state-of-the-art backdoor defense approaches. The code is available at https: //github. com/linweiii/TSBD. git.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

WebUOT-1M: Advancing Deep Underwater Object Tracking with A Million-Scale Benchmark

  • Chunhui Zhang
  • Li Liu
  • Guanjie Huang
  • Hao Wen
  • Xi Zhou
  • Yanfeng Wang

Underwater Object Tracking (UOT) is essential for identifying and tracking submerged objects in underwater videos, but existing datasets are limited in scale, diversity of target categories and scenarios covered, impeding the development of advanced tracking algorithms. To bridge this gap, we take the first step and introduce WebUOT-1M, \ie, the largest public UOT benchmark to date, sourced from complex and realistic underwater environments. It comprises 1. 1 million frames across 1, 500 video clips filtered from 408 target categories, largely surpassing previous UOT datasets, \eg, UVOT400. Through meticulous manual annotation and verification, we provide high-quality bounding boxes for underwater targets. Additionally, WebUOT-1M includes language prompts for video sequences, expanding its application areas, \eg, underwater vision-language tracking. Given that most existing trackers are designed for open-air conditions and perform poorly in underwater environments due to domain gaps, we propose a novel framework that uses omni-knowledge distillation to train a student Transformer model effectively. To the best of our knowledge, this framework is the first to effectively transfer open-air domain knowledge to the UOT model through knowledge distillation, as demonstrated by results on both existing UOT datasets and the newly proposed WebUOT-1M. We have thoroughly tested WebUOT-1M with 30 deep trackers, showcasing its potential as a benchmark for future UOT research. The complete dataset, along with codes and tracking results, are publicly accessible at \href{https: //github. com/983632847/Awesome-Multimodal-Object-Tracking}{\color{magenta}{here}}.

PDF Details DOI

EAAI Journal 2023 Journal Article

A review of wearable sensors based fall-related recognition systems

  • Jiawei Liu
  • Xiaohu Li
  • Shanshan Huang
  • Rui Chao
  • Zhidong Cao
  • Shu Wang
  • Aiguo Wang
  • Li Liu

Falls are an important factor in significantly deteriorating quality of life of older adults, consequently leading to both physical and psychological harm. A wearable-based fall-related recognition system (WFRS) indeed facilitates the prediction, detection, and classification of fall events in helping fallers. Previous studies have provided a relatively comprehensive introduction to WFRSs from the perspective of sensor types and recognition algorithms. However, while these studies provide a clear technical direction, how to choose the appropriate technology for each phase of the experiment is a stumbling block for newly interested researchers. Accordingly, a comprehensive review article covering the mainstream technologies of WFRSs is imperative and meaningful. This review analyzes 48 state-of-the-art researches in WFRSs from three databases (i. e. , IEEE Explorer, ScienceDirect, and MDPI) and introduces the pipeline techniques that consist of data acquisition, preprocessing, feature extraction, model training, and evaluation. Specifically, we first analyze the pros and cons of the use of different number of sensors for data collection. We then introduce the widely used preprocessing techniques including filtering and data augmentation. Afterwards, we detail the extraction of various features and illustrate methods for the selection, training, and evaluation of fall recognition models. We finally discuss factors affecting the overall performance of a model and offer suggestions for future research.

Details DOI

EAAI Journal 2023 Journal Article

Counterfactual-based minority oversampling for imbalanced classification

  • Shu Wang
  • Hao Luo
  • Shanshan Huang
  • Qingsong Li
  • Li Liu
  • Guoxin Su
  • Ming Liu

A key challenge of oversampling in imbalanced classification is that the generation of new minority samples often neglects the usage of majority classes, resulting in most new minority sampling spreading the whole minority space. In view of this, we present a new oversampling framework based on the counterfactual theory. Our framework introduces a counterfactual objective by leveraging the rich inherent information of majority classes and explicitly perturbing majority samples to generate new samples in the territory of minority space. It can be analytically shown that the new minority samples satisfy the minimum inversion. Therefore, most of them are located near the decision boundary. The empirical evaluation of the six benchmark datasets shows that our approach clearly outperforms the state-of-the-art methods.

Details DOI

YNIMG Journal 2023 Journal Article

Exploring the neural mechanisms underlying achalasia: A study of functional connectivity and regional brain activity

  • Nina Zhang
  • Binyu Teng
  • Xinyi Lu
  • Liangliang Shi
  • Li Liu
  • Fan Zhou
  • Ni Jiang
  • Xin Zhang

BACKGROUND AND AIMS: The pathophysiology of achalasia, which involves central nuclei abnormalities, remains unknown. We investigated the resting-state functional MRI (rs-fMRI) features of patients with achalasia. METHODS: We applied resting-state functional MRI (rs-fMRI) to investigate the brain features in patients with achalasia (n = 27), compared to healthy controls (n = 29). Focusing on three regions of interest (ROIs): the dorsal motor nucleus of the vagus (DMV), the nucleus ambiguus (NA), and the nucleus of the solitary tract (NTS), we analyzed variations in resting-state functional connectivity (rs-FC), fractional amplitude of low-frequency fluctuations (fALFF), and regional homogeneity (ReHo). RESULTS: Achalasia patients demonstrated stronger functional connectivity between the NA and the right precentral gyrus, left postcentral gyrus, and left insula. No significant changes were found in the DMV or NTS. The fMRI analysis showed higher rs-FC values for NA-DMV and NA-NTS connections in achalasia patients. Achalasia patients exhibited decreased fALFF values in the NA, DMV, and NTS regions, as well as increased ReHo values in the NA and DMV regions. A positive correlation was observed between fALFF values in all six ROIs and the width of the barium meal. The NTS fALFF value and NA ReHo value displayed a positive correlation with integrated relaxation pressure (IRP), while the ReHo value in the right precentral gyrus showed an inverse correlation with the height of the barium meal. CONCLUSIONS: Abnormal rs-FC and regional brain activity was found in patients with achalasia. Our study provides new insights into the pathophysiology of achalasia and highlights the potential of rs-fMRI in improving the diagnosis and treatment of this condition.

Details DOI

ICLR Conference 2023 Conference Paper

Learning Symbolic Models for Graph-structured Physical Mechanism

  • Hongzhi Shi
  • Jingtao Ding
  • Yufan Cao 0002
  • Quanming Yao
  • Li Liu
  • Yong Li 0008

Graph-structured physical mechanisms are ubiquitous in real-world scenarios, thus revealing underneath formulas is of great importance for scientific discovery. However, classical symbolic regression methods fail on this task since they can only handle input-output pairs that are not graph-structured. In this paper, we propose a new approach that generalizes symbolic regression to graph-structured physical mechanisms. The essence of our method is to model the formula skeleton with a message-passing flow, which helps transform the discovery of the skeleton into the search for the message-passing flow. Such a transformation guarantees that we are able to search a message-passing flow, which is efficient and Pareto-optimal in terms of both accuracy and simplicity. Subsequently, the underneath formulas can be identified by interpreting component functions of the searched message-passing flow, reusing classical symbolic regression methods. We conduct extensive experiments on datasets from different physical domains, including mechanics, electricity, and thermology, and on real-world datasets of pedestrian dynamics without ground-truth formulas. The experimental results not only verify the rationale of our design but also demonstrate that the proposed method can automatically learn precise and interpretable formulas for graph-structured physical mechanisms.

Details

AAAI Conference 2023 Conference Paper

Preserving Structural Consistency in Arbitrary Artist and Artwork Style Transfer

  • Jingyu Wu
  • Lefan Hou
  • Zejian Li
  • Jun Liao
  • Li Liu
  • Lingyun Sun

Deep generative models are effective in style transfer. Previous methods learn one or several specific artist-style from a collection of artworks. These methods not only homogenize the artist-style of different artworks of the same artist but also lack generalization for the unseen artists. To solve these challenges, we propose a double-style transferring module (DSTM). It extracts different artist-style and artwork-style from different artworks (even untrained) and preserves the intrinsic diversity between different artworks of the same artist. DSTM swaps the two styles in the adversarial training and encourages realistic image generation given arbitrary style combinations. However, learning style from single artwork can often cause over-adaption to it, resulting in the introduction of structural features of style image. We further propose an edge enhancing module (EEM) which derives edge information from multi-scale and multi-level features to enhance structural consistency. We broadly evaluate our method across six large-scale benchmark datasets. Empirical results show that our method achieves arbitrary artist-style and artwork-style extraction from a single artwork, and effectively avoids introducing the style image’s structural features. Our method improves the state-of-the-art deception rate from 58.9% to 67.2% and the average FID from 48.74 to 42.83.

PDF Details DOI

IJCAI Conference 2022 Conference Paper

Anti-Forgery: Towards a Stealthy and Robust DeepFake Disruption Attack via Adversarial Perceptual-aware Perturbations

  • Run Wang
  • Ziheng Huang
  • Zhikai Chen
  • Li Liu
  • Jing Chen
  • Lina Wang

DeepFake is becoming a real risk to society and brings potential threats to both individual privacy and political security due to the DeepFaked multimedia are realistic and convincing. However, the popular DeepFake passive detection is an ex-post forensics countermeasure and failed in blocking the disinformation spreading in advance. To address this limitation, researchers study the proactive defense techniques by adding adversarial noises into the source data to disrupt the DeepFake manipulation. However, the existing studies on proactive DeepFake defense via injecting adversarial noises are not robust, which could be easily bypassed by employing simple image reconstruction revealed in a recent study MagDR. In this paper, we investigate the vulnerability of the existing forgery techniques and propose a novel anti-forgery technique that helps users protect the shared facial images from attackers who are capable of applying the popular forgery techniques. Our proposed method generates perceptual-aware perturbations in an incessant manner which is vastly different from the prior studies by adding adversarial noises that is sparse. Experimental results reveal that our perceptual-aware perturbations are robust to diverse image transformations, especially the competitive evasion technique, MagDR via image reconstruction. Our findings potentially open up a new research direction towards thorough understanding and investigation of perceptual-aware adversarial attack for protecting facial images against DeepFakes in a proactive and robust manner. Code is available at https: //github. com/AbstractTeen/AntiForgery.

PDF Details DOI

EAAI Journal 2022 Journal Article

Deep learning for image colorization: Current and future prospects

  • Shanshan Huang
  • Xin Jin
  • Qian Jiang
  • Li Liu

Image colorization, as an essential problem in computer vision (CV), has attracted an increasing amount of researchers attention in recent years, especially deep learning-based image colorization techniques(DLIC). Generally, most recent image colorization methods can be regarded as knowledge-based systems because they are usually trained by big datasets. Unlike the existing reviews, this paper adopts a unique deep learning-based perspective to review the latest progress in image colorization techniques systematically and comprehensively. In this paper, a comprehensive review of recent DLIC approaches from algorithm classification to existing challenges is provided to facilitate researchers’ in-depth understanding of DLIC. In particular, we review DLIC algorithms from various perspectives, including color space, network structure, loss function, level of automation, and application fields. Furthermore, other important issues are discussed, such as publicly available benchmark datasets and performance evaluation metrics. Finally, we discuss several open issues of image colorization and outline future research directions. This survey can serve as a reference for researchers in image colorization and related fields.

Details DOI

ICRA Conference 2022 Conference Paper

DKNAS: A Practical Deep Keypoint Extraction Framework Based on Neural Architecture Search

  • Li Liu
  • Xing Cai
  • Ge Li 0002
  • Thomas H. Li

Keypoint extraction including both keypoint detection and description is a fundamental step in a wide range of geometric multimedia applications. In recent years, many learning-based approaches for keypoint extraction emerge and achieve promising results. However, they usually design network architectures empirically and lack of considerations about the comprehensive performance, which leads to limited applications. In this paper, we propose a practical framework based on Neural Architecture Search (NAS) technology, DKNAS, which can search architectures automatically and maintain efficiency and effectiveness, simultaneously. To the best of our knowledge, the proposed framework is the first NAS framework for keypoint extraction. The evaluation on HPatches dataset shows that our method achieves state-of-the-art results in the metrics of repeatability, localization error, homography accuracy and matching scores. Besides, our model is applied to a traditional Simultaneous Localization and Mapping (SLAM) system, ORB-SLAM2, to replace the handcrafted keypoints. Experimental results demonstrate that the system adopting our model outperforms ORB-SLAM2 and some other deep keypoints enhanced systems.

Details

AAAI Conference 2022 Conference Paper

Explainable Survival Analysis with Convolution-Involved Vision Transformer

  • Yifan Shen
  • Li Liu
  • Zhihao Tang
  • Zongyi Chen
  • Guixiang Ma
  • Jiyan Dong
  • Xi Zhang
  • Lin Yang

Image-based survival prediction models can facilitate doctors in diagnosing and treating cancer patients. With the advance of digital pathology technologies, the big whole slide images (WSIs) provide increased resolution and more details for diagnosis. However, the gigabytesize or even terabyte-size WSIs would make most models computationally infeasible. To this end, instead of using the complete WSIs, most of the existing models only use a pre-selected subset of key patches or patch clusters as input, which might discard some important morphology information. In this work, we propose a novel survival analysis model to fully utilize the complete WSI information. We show that the use of a Vision Transformer (ViT) backbone, together with convolution operations involved in it, is an effective approach to improve the prediction performance. Additionally, we present a post-hoc explainable method to identify the most salient patches and distinct morphology features, making the model more faithful and the results easier to comprehend by human users. Evaluations on two large cancer datasets show that our proposed model is more effective and has better interpretability for survival prediction. We would make the code publicly available upon acceptance.

PDF Details

NeurIPS Conference 2022 Conference Paper

Pre-activation Distributions Expose Backdoor Neurons

  • Runkai Zheng
  • Rongjun Tang
  • Jianze Li
  • Li Liu

Convolutional neural networks (CNN) can be manipulated to perform specific behaviors when encountering a particular trigger pattern without affecting the performance on normal samples, which is referred to as backdoor attack. The backdoor attack is usually achieved by injecting a small proportion of poisoned samples into the training set, through which the victim trains a model embedded with the designated backdoor. In this work, we demonstrate that backdoor neurons are exposed by their pre-activation distributions, where populations from benign data and poisoned data show significantly different moments. This property is shown to be attack-invariant and allows us to efficiently locate backdoor neurons. On this basis, we make several proper assumptions on the neuron activation distributions, and propose two backdoor neuron detection strategies based on (1) the differential entropy of the neurons, and (2) the Kullback-Leibler divergence between the benign sample distribution and a poisoned statistics based hypothetical distribution. Experimental results show that our proposed defense strategies are both efficient and effective against various backdoor attacks.

PDF Details

YNICL Journal 2022 Journal Article

Specific structuro-metabolic pattern of thalamic subnuclei in fatal familial insomnia: A PET/MRI imaging study

  • Kexin Xie
  • Yaojing Chen
  • Min Chu
  • Yue Cui
  • Zhongyun Chen
  • Jing Zhang
  • Li Liu
  • Donglai Jing

BACKGROUND: Dysfunction of the thalamus has been proposed as a core mechanism of fatal familial insomnia. However, detailed metabolic and structural alterations in thalamic subnuclei are not well documented. We aimed to address the multimodal structuro-metabolic pattern at the level of the thalamic nuclei in fatal familial insomnia patients, and investigated the clinical presentation of primary thalamic alterations. MATERIALS AND METHODS: Five fatal familial insomnia patients and 10 healthy controls were enrolled in this study. All participants underwent neuropsychological assessments, polysomnography, electroencephalogram, and cerebrospinal fluid tests. MRI and fluorodeoxyglucose PET were acquired on a hybrid PET/MRI system. Structural and metabolic changes were compared using voxel-based morphometry analyses and standardized uptake value ratio analyses, focusing on thalamic subnuclei region of interest analyses. Correlation analysis was conducted between gray matter volume and metabolic decrease ratios, and clinical features. RESULTS: The whole-brain analysis showed that gray matter volume decline was confined to the bilateral thalamus and right middle temporal pole in fatal familial insomnia patients, whereas hypometabolism was observed in the bilateral thalamus, basal ganglia, and widespread cortices, mainly in the forebrain. In the regions of interest analysis, gray matter volume and metabolism decreases were prominent in bilateral medial dorsal nuclei, anterior nuclei, and the pulvinar, which is consistent with neuropathological and clinical findings. A positive correlation was found between gray matter volume and metabolic decrease ratios. CONCLUSIONS: Our study revealed specific structuro-metabolic pattern of fatal familial insomnia that demonstrated the essential roles of medial dorsal nuclei, anterior nuclei, and pulvinar, which may be a potential biomarker in diagnosis. Also, primary thalamic subnuclei alterations may be correlated with insomnia, neuropsychiatric, and autonomic symptoms sparing primary cortical involvement.

Details DOI

YNICL Journal 2022 Journal Article

Thalamic-insomnia phenotype in E200K Creutzfeldt–Jakob disease: A PET/MRI study

  • Hong Ye
  • Min Chu
  • Zhongyun Chen
  • Kexin Xie
  • Li Liu
  • Haitian Nan
  • Yue Cui
  • Jing Zhang

BACKGROUND: Insomnia and thalamic involvement were frequently reported in patients with genetic Creutzfeldt-Jakob disease (gCJD) with E200K mutations, suggesting E200K might have discrepancy with typical sporadic CJD (sCJD). The study aimed to explore the clinical and neuroimage characteristics of genetic E200K CJD patients by comprehensive neuroimage analysis. METHODS: Six patients with gCJD carried E200K mutation on Prion Protein (PRNP) gene, 13 patients with sporadic CJD, and 22 age- and sex-matched normal controls were enrolled in the study. All participants completed a hybrid positron emission tomography/magnetic resonance imaging (PET/MRI) examination. Signal intensity on diffusion-weighted imaging (DWI) and metabolism on PET were visually rating analyzed, statistical parameter mapping analysis was performed on PET and 3D-T1 images. Clinical and imaging characteristics were compared between the E200K, sCJD, and control groups. RESULTS: There was no group difference in age or gender among the E200K, sCJD, and control groups. Insomnia was a primary complaint in patients with E200K gCJD (4/2 versus 1/12, p = 0.007). Hyperintensity on DWI and hypometabolism on PET of the thalamus were observed during visual rating analysis of images in patients with E200K gCJD. Gray matter atrophy (uncorrected p < 0.001) and hypometabolism (uncorrected p < 0.001) of the thalamus were more pronounced in patients with E200K gCJD. CONCLUSION: The clinical and imaging characteristics of patients with gCJD with PRNP E200K mutations manifested as a thalamic-insomnia phenotype. PET is a sensitive approach to help identify the functional changes in the thalamus in prion disease.

Details DOI

AAAI Conference 2021 Conference Paper

How to Train Your Agent to Read and Write

  • Li Liu
  • Mengge He
  • Guanghui Xu
  • Mingkui Tan
  • Qi Wu

Reading and writing research papers is one of the most privileged abilities that a qualified researcher should master. However, it is difficult for new researchers (e. g. , students) to fully grasp this ability. It would be fascinating if we could train an intelligent agent to help people read and summarize papers, and perhaps even discover and exploit the potential knowledge clues to write novel papers. Although there have been existing works focusing on summarizing (i. e. , reading) the knowledge in a given text or generating (i. e. , writing) a text based on the given knowledge, the ability of simultaneously reading and writing is still under development. Typically, this requires an agent to fully understand the knowledge from the given text materials and generate correct and fluent novel paragraphs, which is very challenging in practice. In this paper, we propose a Deep ReAder-Writer (DRAW) network, which consists of a Reader that can extract knowledge graphs (KGs) from input paragraphs and discover potential knowledge, a graph-to-text Writer that generates a novel paragraph, and a Reviewer that reviews the generated paragraph from three different aspects. Extensive experiments show that our DRAW network outperforms considered baselines and several state-of-the-art methods on AGENDA and M-AGENDA datasets. Our code and supplementary are released at https: //github. com/menggehe/DRAW.

PDF Details

IJCAI Conference 2020 Conference Paper

Set and Rebase: Determining the Semantic Graph Connectivity for Unsupervised Cross-Modal Hashing

  • Weiwei Wang
  • Yuming Shen
  • Haofeng Zhang
  • Yazhou Yao
  • Li Liu

The label-free nature of unsupervised cross-modal hashing hinders models from exploiting the exact semantic data similarity. Existing research typically simulates the semantics by a heuristic geometric prior in the original feature space. However, this introduces heavy bias into the model as the original features are not fully representing the underlying multi-view data relations. To address the problem above, in this paper, we propose a novel unsupervised hashing method called Semantic-Rebased Cross-modal Hashing (SRCH). A novel ‘Set-and-Rebase’ process is defined to initialize and update the cross-modal similarity graph of training data. In particular, we set the graph according to the intra-modal feature geometric basis and then alternately rebase it to update the edges within according to the hashing results. We develop an alternating optimization routine to rebase the graph and train the hashing auto-encoders with closed-form solutions so that the overall framework is efficiently trained. Our experimental results on benchmarked datasets demonstrate the superiority of our model against state-of-the-art algorithms.

PDF Details DOI

IJCAI Conference 2019 Conference Paper

Dynamically Visual Disambiguation of Keyword-based Image Search

  • Yazhou Yao
  • Zeren Sun
  • Fumin Shen
  • Li Liu
  • Limin Wang
  • Fan Zhu
  • Lizhong Ding
  • Gangshan Wu

Due to the high cost of manual annotation, learning directly from the web has attracted broad attention. One issue that limits their performance is the problem of visual polysemy. To address this issue, we present an adaptive multi-model framework that resolves polysemy by visual disambiguation. Compared to existing methods, the primary advantage of our approach lies in that our approach can adapt to the dynamic changes in the search results. Our proposed framework consists of two major steps: we first discover and dynamically select the text queries according to the image search results, then we employ the proposed saliency-guided deep multi-instance learning network to remove outliers and learn classification models for visual disambiguation. Extensive experiments demonstrate the superiority of our proposed approach.

PDF Details

IJCAI Conference 2019 Conference Paper

Toward Efficient Navigation of Massive-Scale Geo-Textual Streams

  • Chengcheng Yang
  • Lisi Chen
  • Shuo Shang
  • Fan Zhu
  • Li Liu
  • Ling Shao

With the popularization of portable devices, numerous applications continuously produce huge streams of geo-tagged textual data, thus posing challenges to index geo-textual streaming data efficiently, which is an important task in both data management and AI applications, e. g. , real-time data streams mining and targeted advertising. This, however, is not possible with the state-of-the-art indexing methods as they focus on search optimizations of static datasets, and have high index maintenance cost. In this paper, we present NQ-tree, which combines new structure designs and self-tuning methods to navigate between update and search efficiency. Our contributions include: (1) the design of multiple stores each with a different emphasis on write-friendness and read-friendness; (2) utilizing data compression techniques to reduce the I/O cost; (3) exploiting both spatial and keyword information to improve the pruning efficiency; (4) proposing an analytical cost model, and using an online self-tuning method to achieve efficient accesses to different workloads. Experiments on two real-world datasets show that NQ-tree outperforms two well designed baselines by up to 10×.

PDF Details

NeurIPS Conference 2019 Conference Paper

Two Generator Game: Learning to Sample via Linear Goodness-of-Fit Test

  • Lizhong Ding
  • Mengyang Yu
  • Li Liu
  • Fan Zhu
  • Yong Liu
  • Yu Li
  • Ling Shao

Learning the probability distribution of high-dimensional data is a challenging problem. To solve this problem, we formulate a deep energy adversarial network (DEAN), which casts the energy model learned from real data into an optimization of a goodness-of-fit (GOF) test statistic. DEAN can be interpreted as a GOF game between two generative networks, where one explicit generative network learns an energy-based distribution that fits the real data, and the other implicit generative network is trained by minimizing a GOF test statistic between the energy-based distribution and the generated data, such that the underlying distribution of the generated data is close to the energy-based distribution. We design a two-level alternative optimization procedure to train the explicit and implicit generative networks, such that the hyper-parameters can also be automatically learned. Experimental results show that DEAN achieves high quality generations compared to the state-of-the-art approaches.

PDF Details

IJCAI Conference 2018 Conference Paper

Localized Incomplete Multiple Kernel k-means

  • Xinzhong Zhu
  • Xinwang Liu
  • Miaomiao Li
  • En Zhu
  • Li Liu
  • Zhiping Cai
  • Jianping Yin
  • Wen Gao

The recently proposed multiple kernel k-means with incomplete kernels (MKKM-IK) optimally integrates a group of pre-specified incomplete kernel matrices to improve clustering performance. Though it demonstrates promising performance in various applications, we observe that it does not \emph{sufficiently consider the local structure among data and indiscriminately forces all pairwise sample similarity to equally align with their ideal similarity values}. This could make the incomplete kernels less effectively imputed, and in turn adversely affect the clustering performance. In this paper, we propose a novel localized incomplete multiple kernel k-means (LI-MKKM) algorithm to address this issue. Different from existing MKKM-IK, LI-MKKM only requires the similarity of a sample to its k-nearest neighbors to align with their ideal similarity values. This helps the clustering algorithm to focus on closer sample pairs that shall stay together and avoids involving unreliable similarity evaluation for farther sample pairs. We carefully design a three-step iterative algorithm to solve the resultant optimization problem and theoretically prove its convergence. Comprehensive experiments on eight benchmark datasets demonstrate that our algorithm significantly outperforms the state-of-the-art comparable algorithms proposed in the recent literature, verifying the advantage of considering local structure.

PDF Details

TIST Journal 2018 Journal Article

Multiview Discrete Hashing for Scalable Multimedia Search

  • Xiaobo Shen
  • Fumin Shen
  • Li Liu
  • Yun-Hao Yuan
  • Weiwei Liu
  • Quan-Sen Sun

Hashing techniques have recently gained increasing research interest in multimedia studies. Most existing hashing methods only employ single features for hash code learning. Multiview data with each view corresponding to a type of feature generally provides more comprehensive information. How to efficiently integrate multiple views for learning compact hash codes still remains challenging. In this article, we propose a novel unsupervised hashing method, dubbed multiview discrete hashing (MvDH), by effectively exploring multiview data. Specifically, MvDH performs matrix factorization to generate the hash codes as the latent representations shared by multiple views, during which spectral clustering is performed simultaneously. The joint learning of hash codes and cluster labels enables that MvDH can generate more discriminative hash codes, which are optimal for classification. An efficient alternating algorithm is developed to solve the proposed optimization problem with guaranteed convergence and low computational complexity. The binary codes are optimized via the discrete cyclic coordinate descent (DCC) method to reduce the quantization errors. Extensive experimental results on three large-scale benchmark datasets demonstrate the superiority of the proposed method over several state-of-the-art methods in terms of both accuracy and scalability.

Details DOI

YNIMG Journal 2018 Journal Article

Structural correlates of literacy difficulties in the second language: Evidence from Mandarin-speaking children learning English

  • Hehui Li
  • James R. Booth
  • Nathalie N. Bélanger
  • Xiaoxia Feng
  • Mengyu Tian
  • Weiyi Xie
  • Manli Zhang
  • Yue Gao

Several neuroimaging studies have explored the neural basis of literacy difficulties in the second language (L2). However, it remains unclear whether the associated neural alterations are related to literacy abilities in the first language (L1). Using magnetic resonance imaging, we explore this issue with two experiments in Mandarin-speaking children learning English as second language. In the first experiment, we investigated children with literacy difficulties in L2 and L1 (poor in both, PB) and children with literacy difficulties only in L2 (poor in English, PE). We compared the brain structure in these two groups to a control literacy (CL) group. The results showed that the CL group had significantly less gray matter volume in the left supramarginal gyrus compared to the PB group and moderately less gray matter volume compared to the PE group. In addition, the PB group had significant greater gray matter volume in the left medial fusiform gyrus compared to the PE group and had marginally greater gray matter volume compared to the CL group. In the second experiment, we explored the relationship between the two atypical regions and literacy abilities in the two languages in an independent sample consisting of children with typical literacy. Correlation analyses revealed that the left supramarginal gyrus was significantly associated with literacy performance only in the second language, English, whereas the left medial fusiform gyrus did not correlate with the performances in either L1 or L2. Taken together, these findings suggest that literacy difficulties in an alphabetic L2 are associated with a structural abnormality in the left supramarginal gyrus, a region implicated in phonological processing, which is independent of literacy abilities in the native language.

Details DOI

AAAI Conference 2018 Conference Paper

Towards Affordable Semantic Searching: Zero-Shot Retrieval via Dominant Attributes

  • Yang Long
  • Li Liu
  • Yuming Shen
  • Ling Shao

Instance-level retrieval has become an essential paradigm to index and retrieves images from large-scale databases. Conventional instance search requires at least an example of the query image to retrieve images that contain the same object instance. Existing semantic retrieval can only search semantically-related images, such as those sharing the same category or a set of tags, not the exact instances. Meanwhile, the unrealistic assumption is that all categories or tags are known beforehand. Training models for these semantic concepts highly rely on instance-level attributes or human captions which are expensive to acquire. Given the above challenges, this paper studies the Zero-shot Retrieval problem that aims for instance-level image search using only a few dominant attributes. The contributions are: 1) we utilise automatic word embedding to infer class-level attributes to circumvent expensive human labelling; 2) the inferred class-attributes can be extended into discriminative instance attributes through our proposed Latent Instance Attributes Discovery (LIAD) algorithm; 3) our method is not restricted to complete attribute signatures, query of dominant attributes can also be dealt with. On two benchmarks, CUB and SUN, extensive experiments demonstrate that our method can achieve promising performance for the problem. Moreover, our approach can also benefit conventional ZSL tasks.

PDF Details

IJCAI Conference 2018 Conference Paper

Unsupervised Deep Hashing via Binary Latent Factor Models for Large-scale Cross-modal Retrieval

  • Gengshen Wu
  • Zijia Lin
  • Jungong Han
  • Li Liu
  • Guiguang Ding
  • Baochang Zhang
  • Jialie Shen

Despite its great success, matrix factorization based cross-modality hashing suffers from two problems: 1) there is no engagement between feature learning and binarization; and 2) most existing methods impose the relaxation strategy by discarding the discrete constraints when learning the hash function, which usually yields suboptimal solutions. In this paper, we propose a novel multimodal hashing framework, referred as Unsupervised Deep Cross-Modal Hashing (UDCMH), for multimodal data search in a self-taught manner via integrating deep learning and matrix factorization with binary latent factor models. On one hand, our unsupervised deep learning framework enables the feature learning to be jointly optimized with the binarization. On the other hand, the hashing system based on the binary latent factor models can generate unified binary codes by solving a discrete-constrained objective function directly with no need for a relaxation step. Moreover, novel Laplacian constraints are incorporated into the objective function, which allow to preserve not only the nearest neighbors that are commonly considered in the literature but also the farthest neighbors of data, even if the semantic labels are not available. Extensive experiments on multiple datasets highlight the superiority of the proposed framework over several state-of-the-art baselines.

PDF Details

IJCAI Conference 2017 Conference Paper

Discriminative Dictionary Learning With Ranking Metric Embedded for Person Re-Identification

  • De Cheng
  • Xiaojun Chang
  • Li Liu
  • Alexander G. Hauptmann
  • Yihong Gong
  • Nanning Zheng

The goal of person re-identification (Re-Id) is to match pedestrians captured from multiple non-overlapping cameras. In this paper, we propose a novel dictionary learning based method with the ranking metric embedded, for person Re-Id. A new and essential ranking graph Laplacian term is introduced, which minimizes the intra-personal compactness and maximizes the inter-personal dispersion in the objective. Different from the traditional dictionary learning based approaches and their extensions, which just use the same or not information, our proposed method can explore the ranking relationship among the person images, which is essential for such retrieval related tasks. Simultaneously, one distance measurement has been explicitly learned in the model to further improve the performance. Since we have reformulated these ranking constraints into the graph Laplacian form, the proposed method is easy-to-implement but effective. We conduct extensive experiments on three widely used person Re-Id benchmark datasets, and achieve state-of-the-art performances.

PDF Details

IJCAI Conference 2017 Conference Paper

Unsupervised Deep Video Hashing with Balanced Rotation

  • Gengshen Wu
  • Li Liu
  • Yuchen Guo
  • Guiguang Ding
  • Jungong Han
  • Jialie Shen
  • Ling Shao

Recently, hashing video contents for fast retrieval has received increasing attention due to the enormous growth of online videos. As the extension of image hashing techniques, traditional video hashing methods mainly focus on seeking the appropriate video features but pay little attention to how the video-specific features can be leveraged to achieve optimal binarization. In this paper, an end-to-end hashing framework, namely Unsupervised Deep Video Hashing (UDVH), is proposed, where feature extraction, balanced code learning and hash function learning are integrated and optimized in a self-taught manner. Particularly, distinguished from previous work, our framework enjoys two novelties: 1) an unsupervised hashing method that integrates the feature clustering and feature binarization, enabling the neighborhood structure to be preserved in the binary space; 2) a smart rotation applied to the video-specific features that are widely spread in the low-dimensional space such that the variance of dimensions can be balanced, thus generating more effective hash codes. Extensive experiments have been performed on two real-world datasets and the results demonstrate its superiority, compared to the state-of-the-art video hashing methods. To bootstrap further developments, the source code will be made publically available.

PDF Details

IJCAI Conference 2016 Conference Paper

Aligning Users across Social Networks Using Network Embedding

  • Li Liu
  • William K. Cheung
  • Xin Li
  • Lejian Liao

In this paper, we adopt the representation learning approach to align users across multiple social networks where the social structures of the users are exploited. In particular, we propose to learn a network embedding with the follower-ship/followee-ship of each user explicitly modeled as input/output context vector representations so as to preserve the proximity of users with "similar" followers/followees in the embedded space. For the alignment, we add both known and potential anchor users across the networks to facilitate the transfer of context information across networks. We solve both the network embedding problem and the user alignment problem simultaneously under a unified optimization framework. The stochastic gradient descent and negative sampling algorithms are used to address scalability issues. Extensive experiments on real social network datasets demonstrate the effectiveness and efficiency of the proposed approach compared with several state-of-the-art methods.

PDF Details

AAAI Conference 2016 Conference Paper

Fusing Social Networks with Deep Learning for Volunteerism Tendency Prediction

  • Yongpo Jia
  • Xuemeng Song
  • Jingbo Zhou
  • Li Liu
  • Liqiang Nie
  • David Rosenblum

Social networks contain a wealth of useful information. In this paper, we study a challenging task for integrating users’ information from multiple heterogeneous social networks to gain a comprehensive understanding of users’ interests and behaviors. Although much effort has been dedicated to study this problem, most existing approaches adopt linear or shallow models to fuse information from multiple sources. Such approaches cannot properly capture the complex nature of and relationships among different social networks. Adopting deep learning approaches to learning a joint representation can better capture the complexity, but this neglects measuring the level of confidence in each source and the consistency among different sources. In this paper, we present a framework for multiple social network learning, whose core is a novel model that fuses social networks using deep learning with source confidence and consistency regularization. To evaluate the model, we apply it to predict individuals’ tendency to volunteerism. With extensive experimental evaluations, we demonstrate the effectiveness of our model, which outperforms several state-of-the-art approaches in terms of precision, recall and F1-score.

PDF Details

YNIMG Journal 2016 Journal Article

How does language distance between L1 and L2 affect the L2 brain network? An fMRI study of Korean–Chinese–English trilinguals

  • Say Young Kim
  • Ting Qi
  • Xiaoxia Feng
  • Guosheng Ding
  • Li Liu
  • Fan Cao

The present study tested the hypothesis that language distance between first language (L1) and second language (L2) influences the assimilation and accommodation pattern in Korean–Chinese–English trilinguals. The distance between English and Korean is smaller than that between Chinese and Korean in terms of orthographic transparency, because both English and Korean are alphabetic, whereas Chinese is logographic. During fMRI, Korean trilingual participants performed a visual rhyming judgment task in three languages (Korean: KK, Chinese: KC, English: KE). Two L1 control groups were native Chinese and English speakers performing the task in their native languages (CC and EE, respectively). The general pattern of brain activation of KC was more similar to that of CC than KK, suggesting accommodation. Higher accuracy in KC was associated with decreased activation in regions of the KK network, suggesting reduced assimilation. In contrast, the brain activation of KE was more similar to that of KK than EE, suggesting assimilation. Higher accuracy in KE was associated with decreased activation in regions of the EE network, suggesting reduced accommodation. Finally, an ROI analysis on the left middle frontal gyrus revealed greater activation for KC than for KE, suggesting its selective involvement in the L2 with more arbitrary mapping between orthography and phonology (i. e. , Chinese). Taken together, the brain network involved in L2 reading is similar to the L1 network when L2 and L1 are similar in orthographic transparency, while significant accommodation is expected when L2 is more opaque than L1.

Details DOI

YNIMG Journal 2016 Journal Article

More bilateral, more anterior: Alterations of brain organization in the large-scale structural network in Chinese dyslexia

  • Ting Qi
  • Bin Gu
  • Guosheng Ding
  • Gaolang Gong
  • Chunming Lu
  • Danling Peng
  • Jeff G. Malins
  • Li Liu

Abnormalities in large-scale brain networks have been recently reported in dyslexia; however, it remains unclear whether these abnormalities are congenital (due to dyslexia per se) or arise later in development. Here, structural magnetic resonance imaging data of 17 Chinese reading disabled (RD) and 17 age-matched typically developing (TD) children were used to construct cortical thickness (sensitive to postnatal development) and surface area (sensitive to prenatal development) networks. In the thickness network, compared to TD, RD showed reduced nodal network properties (e. g. , degree and betweenness) in the left hemisphere along with enhanced nodal properties mainly in the right hemisphere. As for the surface area network, compared to TD, RD demonstrated lower nodal properties in the posterior brain regions and higher nodal properties in the anterior brain regions. Furthermore, hubs in both the thickness and surface area networks in RD were more distributed in frontal areas and less distributed in parietal areas, whereas TD showed the opposite pattern. Altogether, these findings indicate that the aberrant structural connectivity in the dyslexic individuals was not only due to a late developmental effect reflected in the altered thickness network, but may also be a congenital effect during prenatal development, reflected in the altered surface network.

Details DOI

AAAI Conference 2016 Conference Paper

Recognizing Complex Activities by a Probabilistic Interval-Based Model

  • Li Liu
  • Li Cheng
  • Ye Liu
  • Yongpo Jia
  • David Rosenblum

A key challenge in complex activity recognition is the fact that a complex activity can often be performed in several different ways, with each consisting of its own configuration of atomic actions and their temporal dependencies. This leads us to define an atomic activity-based probabilistic framework that employs Allen’s interval relations to represent local temporal dependencies. The framework introduces a latent variable from the Chinese Restaurant Process to explicitly characterize these unique internal configurations of a particular complex activity as a variable number of tables. It can be analytically shown that the resulting interval network satisfies the transitivity property, and as a result, all local temporal dependencies can be retained and are globally consistent. Empirical evaluations on benchmark datasets suggest our approach significantly outperforms the state-of-the-art methods.

PDF Details

YNIMG Journal 2014 Journal Article

Theta–gamma coupling reflects the interaction of bottom-up and top-down processes in speech perception in children

  • Juan Wang
  • Danqi Gao
  • Duan Li
  • Amy S. Desroches
  • Li Liu
  • Xiaoli Li

This study investigates how the interaction of different brain oscillations (particularly theta–gamma coupling) modulates the bottom-up and top-down processes during speech perception. We employed a speech perception paradigm that manipulated the congruency between a visually presented picture and an auditory stimulus and asked participants to judge whether they matched or mismatched. A group of children (mean age 10years, 5months) participated in this study and their electroencephalographic (EEG) data were recorded while performing the experimental task. It was found that in comparison with mismatch condition, match condition facilitated speech perception by eliciting greater theta–gamma coupling in the frontal area and smaller theta–gamma coupling in the left temporal area. These findings suggested that a top-down facilitation effect from congruent visual pictures engaged different mechanisms in low-level sensory (temporal) regions and high-level linguistic and decision (frontal) regions. Interestingly, hemispheric asymmetry is with higher theta–gamma coupling in the match condition in the right hemisphere and higher theta–gamma coupling in the mismatch condition in the left hemisphere. This indicates that a fast global processing strategy and a slow detailed processing strategy were differentially adopted in the match and mismatch conditions. This study provides new insight into the mechanisms of speech perception from the interaction of different oscillatory activities and provides neural evidence for theories of speech perception allowing for top-down feedback connections. Furthermore, it sheds light on children's speech perception development by showing a similar pattern of integration of bottom-up and top-down information during speech perception as previous studies have revealed in adults.

Details DOI

IJCAI Conference 2013 Conference Paper

Learning Discriminative Representations from RGB-D Video Data

  • Li Liu
  • Ling Shao

Recently, the low-cost Microsoft Kinect sensor, which can capture real-time high-resolution RGB and depth visual information, has attracted increasing attentions for a wide range of applications in computer vision. Existing techniques extract hand-tuned features from the RGB and the depth data separately and heuristically fuse them, which would not fully exploit the complementarity of both data sources. In this paper, we introduce an adaptive learning methodology to automatically extract (holistic) spatio-temporal features, simultaneously fusing the RGB and depth information, from RGB- D video data for visual recognition tasks. We address this as an optimization problem using our proposed restricted graph-based genetic programming (RGGP) approach, in which a group of primitive 3D operators are first randomly assembled as graph-based combinations and then evolved generation by generation by evaluating on a set of RGB- D video samples. Finally the best-performed combination is selected as the (near-)optimal representation for a pre-defined task. The proposed method is systematically evaluated on a new hand gesture dataset, SKIG, that we collected ourselves and the public MSRDailyActivity3D dataset, respectively. Extensive experimental results show that our approach leads to significant advantages compared with state-of-the-art hand-crafted and machine-learned features.

PDF Details DOI

EAAI Journal 2012 Journal Article

Coupling of logistic regression analysis and local search methods for characterization of water distribution system contaminant source

  • Li Liu
  • Emily M. Zechman
  • G. Mahinthakumar
  • S. Ranji Ranjithan

Accidental or intentional drinking water contamination has long been and remains a major threat to water security throughout the world. An inverse problem can be constructed, given sensor measurements in a water distribution system (WDS), to identify the contaminant source characteristics by integrating a WDS simulation model with an optimization method. However, this approach requires numerous compute-intensive simulation runs to evaluate potential solutions; thus, determining the best source characteristic within a reasonable computational time is challenging. In this paper, we describe the development of a WDS contamination characterization algorithm by coupling a statistical model with a heuristic search method. The statistical model is used to identify potential source locations of contamination and a local search aims at further refining contaminant source characteristics. Application of the proposed approach to two illustrative example water distribution networks demonstrates its capability of adaptively discovering contaminant source characteristics as well as evaluating the degree of non-uniqueness of solutions. The results also showed that the local search as an optimizer has better performance than a standard evolutionary algorithm (EA).

Details DOI

EAAI Journal 2011 Journal Article

Real-time particle swarm optimization based current harmonic cancellation

  • Wenxin Liu
  • Il-Yop Chung
  • Li Liu
  • Siyu Leng
  • David A. Cartes

As a powerful optimization algorithm, particle swarm optimization (PSO) has been widely applied to power system researches. However, most existing applications of PSO can only be implemented offline. The difficulties of online implementation mainly come from the unavoidable lengthy simulation time to evaluate a candidate solution. Recently, PSO was implemented online that can identify parameters in a motor control systems. In this paper, the real-time PSO (RT-PSO) based identification technique is applied to cancel current harmonics in power systems. By transforming the identification problem to optimization problem, RT-PSO can simultaneously identify four parameters associated with fundamental current from measurement. In this way, there is no need to identify the fundamental frequency separately or construct fundamental signal from identified harmonic information. The identification algorithm can be applied to three-phases independently, even for unbalanced system or single-phase system. The identified fundamental signal is then used as the reference for current harmonics cancellation. The RT-PSO based harmonic cancellation is realized with an active filter and used to compensate harmonic current created by a nonlinear load. Simulation results demonstrate that the RT-PSO algorithm can provide accurate identification of the fundamental current which in turn will result in good harmonic cancellation performance. As a capable online optimization technique, RT-PSO can be extensively applied to many optimization and control problems.

Details DOI

EAAI Journal 2010 Journal Article

An adaptive optimization technique for dynamic environments

  • Li Liu
  • S. Ranji Ranjithan

The use of evolutionary algorithms (EAs) is beneficial for addressing optimization problems in dynamic environments. The objective function for such problems changes continually; thus, the optimal solutions likewise change. Such dynamic changes pose challenges to EAs due to the poor adaptability of EAs once they have converged. However, appropriate preservation of a sufficient level of individual diversity may help to increase the adaptive search capability of EAs. This paper proposes an EA-based Adaptive Dynamic OPtimization Technique (ADOPT) for solving time-dependent optimization problems. The purpose of this approach is to identify the current optimal solution as well as a set of alternatives that is not only widespread in the decision space, but also performs well with respect to the objective function. The resultant solutions may then serve as a basis solution for the subsequent search while change is occurring. Thus, such an algorithm avoids the clustering of individuals in the same region as well as adapts to changing environments by exploiting diverse promising regions in the solution space. Application of the algorithm to a test problem and a groundwater contaminant source identification problem demonstrates the effectiveness of ADOPT to adaptively identify solutions in dynamic environments.

Details DOI

TCS Journal 2008 Journal Article

Energy conservation in wireless sensor networks and connectivity of graphs

  • Hao Li
  • Huifang Miao
  • Li Liu
  • Lian Li
  • Heping Zhang

In wireless sensor networks (WSNs), the energy source is usually a battery cell, which is impossible to recharge while WSNs are working. Therefore, one of the main issues in wireless sensor networks is how to prolong the network lifetime of WSNs with certain energy sources as well as how to maintain coverage and connectivity. In this paper, we consider wireless sensor networks satisfying the case that each node either monitors one target or is just for connection. Assume that the wireless sensor network has l targets, and that each is monitored by k sensor nodes. If k = 2 and the graph G corresponding to the wireless sensor network is ( l + max { 1, l − 4 } ) -connected, or k ≥ 3 and G is ( l ( k − 1 ) + 1 ) -connected, then we can find k (the maximum number) disjoint sets, each of which completely covers all the targets and remains connected to one of the central processing nodes. The disjoint sets are activated successively, and only the sensor nodes from the active set are responsible for monitoring the targets and connectivity; all other nodes are in a sleep mode. In addition, we also give the related algorithms to find the k disjoint sets.

Details DOI

EAAI Journal 2008 Journal Article

Particle swarm optimization-based parameter identification applied to permanent magnet synchronous motors

  • Li Liu
  • Wenxin Liu
  • David A. Cartes

The high-performance application of high-power permanent magnet synchronous motors (PMSM) is increasing. PMSM models with accurate parameters are significant for precise control system designs. Acquisition of these parameters during motor operation is a challenging task due to the inherent nonlinearity of motor dynamics. This paper proposes an intelligent model parameter identification method using particle swarm optimization (PSO). PSO, an intelligent computational method based on stochastic search, is shown to be a versatile and efficient tool for this complicated engineering problem. Through both simulation and experiment, this paper verifies the effectiveness of the proposed method in identification of PMSM model parameters. Specifically, stator resistance and load torque disturbance are identified in this PMSM application. Though PMSM is presented, the method is generally applicable to other types of electrical motors, as well as other dynamic systems with nonlinear model structure.

Details DOI

YNIMG Journal 2006 Journal Article

Dissociation in the neural basis underlying Chinese tone and vowel production

  • Li Liu
  • Danling Peng
  • Guosheng Ding
  • Zhen Jin
  • Lei Zhang
  • Ke Li
  • Chuansheng Chen

Neuropsychologists have debated over whether the processing of segmental and suprasegmental units involves different neural mechanisms. Focusing on the production of Chinese lexical tones (suprasegmental units) and vowels (segmental units), this study used the adaptation paradigm to investigate a possible neural dissociation for tone and vowel production. Ten native Chinese speakers were asked to name Chinese characters and pinyin (Romanized phonetic system for Chinese language) that varied in terms of tones and vowels. fMRI results showed significant differences in the right inferior frontal gyrus between tone and vowel production (more activation for tones than for vowels). Brain asymmetry analysis further showed that tone production was less left-lateralized than vowel production, although both showed left-hemisphere dominance.

Details DOI

ICRA Conference 2002 Conference Paper

Control System Design of THBIP-I Humanoid Robot

  • Mingguo Zhao
  • Li Liu
  • Jingsong Wang
  • Ken Chen
  • Jiandong Zhao
  • Kai Xu

Describes the progress of the control system design and implementation of the THBIP-I humanoid robot. The robot has 32 degrees of freedom and each joint is driven by a brushless DC electronic motor. Screw/nuts transmission mechanism is adapted in some joints of lower limbs to achieve compact and good dynamic performance. The control system of the robot has four subsystems: remote brain work station, mobile controller, distributed control units and sensor processing unit. At the present state, the lower limbs and upper limbs have been built and tested with off line gait planning. The distributed control units use PID schemes to servo the pre-generated joint trajectories. Under this architecture, the robot can perform stable walking with 30 centimeters step at 20 second per step.

Details
v2026.09.13