Author name cluster

Xu Liu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

36 papers

2 author rows

AAAI Conference 2026 Conference Paper

A Closer Look at Knowledge Distillation in Spiking Neural Network Training

Xu Liu
Na Xia
Jinxing Zhou
Jingyuan Xu
Dan Guo

Spiking Neural Networks (SNNs) become popular due to excellent energy efficiency, yet facing challenges for effective model training. Recent works improve this by introducing knowledge distillation (KD) techniques, with the pre-trained artificial neural networks (ANNs) used as teachers and the target SNNs as students. This is commonly accomplished through a straightforward element-wise alignment of intermediate features and prediction logits from ANNs and SNNs, often neglecting the intrinsic differences between their architectures. Specifically, ANN's outputs exhibit a continuous distribution, whereas SNN's outputs are characterized by sparsity and discreteness. To mitigate this issue, we introduce two innovative KD strategies. Firstly, we propose the Saliency-scaled Activation Map Distillation (SAMD), which aligns the spike activation map of the student SNN with the class-aware activation map of the teacher ANN. Rather than performing KD directly on the raw features of ANN and SNN, our SAMD directs the student to learn from saliency activation maps that exhibit greater semantic and distribution consistency. Additionally, we propose a Noise-smoothed Logits Distillation (NLD), which utilizes Gaussian noise to smooth the sparse logits of student SNN, facilitating the alignment with continuous logits from teacher ANN. Extensive experiments on multiple datasets demonstrate the effectiveness of our methods.

PDF Details DOI

EAAI Journal 2026 Journal Article

Anti-vibration wavefront interferometry by physics-informed U-Net

Shuai Wang
Chengyu Gu
Jinming Gao
Shouli Sun
Yinxu Bian
Cuifang Kuang
Xu Liu

High-precision wavefront interferometry is extremely sensitive to environmental vibrations, which limits the industrial deployment of conventional phase-shifting interferometers. In this paper, an anti-vibration wavefront interferometry (AVWI) framework using only one or two interferograms is proposed. AVWI reduces the reliance on vibration-isolated environments and high-precision phase shifters by explicitly decoupling vibration-induced errors from the test wavefront. This is enabled by a vibration-detection model based on a physics-informed U-Net (AVWI-PIUN). The network learns vibration information from interferograms and enforces cross-frame consistency among phase-shifted interferograms. A physics-informed loss, denoted as L P h y, is further introduced using partial-derivative relationships between phase-shifted interferograms, which significantly reduces the required training data and accelerates convergence. Experiments under vibrating conditions demonstrate that AVWI achieves superior accuracy, repeatability, and efficiency compared with a temporal phase-shifting method. Specifically, AVWI achieves a Q peak-to-valley (PVq) figure error of 13. 750 nm (nm), a root mean square (RMS) figure error of 1. 784 nm, and an RMS repeatability of 0. 411 nm, corresponding to reductions of 50. 1%, 76. 0%, and 92. 2%, respectively. In addition, AVWI tolerates phase-shifting errors up to ±9π/20. These results indicate that AVWI provides a practical route toward nanometer-level surface figure metrology in vibration-prone industrial and in-situ machining environments.

Details DOI

AAAI Conference 2026 Conference Paper

Evolving Semantic Propagation for Aerial Semantic 3D Gaussian Splatting

Zihan Gao
Lingling Li
Xu Liu
Fang Liu
Licheng Jiao
Puhua Chen
Wenping Ma
Shuyuan Yang

Semantic understanding of large-scale aerial scenes represents a critical challenge in 3D computer vision, hindered by the prohibitive cost of dense annotation. This paper introduces EvoPropGS, a novel approach for the semantic segmentation of 3D Gaussian Splatting models that requires only minimal supervision. Our core insight is to leverage the inherent structural repetitions within aerial environments to propagate semantic information from a sparse set of annotations across the entire 3D scene. Our approach constructs a prompt library by pairing SAM-generated mask candidates with DINOv2 feature embeddings from annotated views. For unannotated regions, we generate pseudo-labels by matching region proposals with these featured prompts via cosine similarity. We then formulate optimal prompt selection as a discrete optimization problem solved via evolutionary search, guided by our novel fitness function that evaluates both 3D consistency and 2D semantic coherence. Extensive experiments demonstrate that EvoPropGS achieves accurate segmentation with only 2 percent annotated pixels.

PDF Details DOI

AAAI Conference 2026 Conference Paper

HTTrack: Learning to Perceive Targets via Historical Trajectories in Satellite Video Tracking

Jiahao Wang
Fang Liu
Licheng Jiao
Hao Wang
Shuo Li
Xinyi Wang
Lingling Li
Puhua Chen

In recent years, the rapid progress of deep learning has driven notable advancements in satellite video tracking, a critical task for applications such as environmental monitoring, disaster management, and defense. Despite these strides, existing approaches remain constrained by their inability to handle dynamic challenges, such as target appearance variations, complex motion patterns, and occlusions. Traditional methods often suffer from static template matching or overly complex update mechanisms, compromising their robustness and practicality in real-world scenarios. To address these limitations, we propose a paradigm shift in satellite video tracking by integrating historical trajectory knowledge with visual features. This fusion enhances the tracker's perceptual understanding of targets over time, enabling more adaptive and resilient tracking. By aligning spatial, temporal, and cross-modal information, our approach effectively bridges the gap between fragmented observations and coherent tracking performance, even under challenging conditions like small target detection and cluttered backgrounds. Extensive experiments conducted on multiple satellite video tracking benchmarks demonstrate the superiority of our method, with HTTrack achieving success rates of 51.5% on SV248S, 52.9% on SatSOT, and 32.6% on VISO, significantly outperforming state-of-the-art trackers and marking a step forward in achieving robust, accurate, and scalable satellite video tracking.

PDF Details DOI

AAAI Conference 2026 Conference Paper

IndoorUAV: Benchmarking Vision-Language UAV Navigation in Continuous Indoor Environments

Xu Liu
Yu Liu
Hanshuo Qiu
Yang Qirong
Zhouhui Lian

Vision-Language Navigation (VLN) enables agents to navigate in complex environments by following natural language instructions grounded in visual observations. Although most existing work has focused on ground-based robots or outdoor Unmanned Aerial Vehicles (UAVs), indoor UAV-based VLN remains underexplored, despite its relevance to real-world applications such as inspection, delivery, and search-and-rescue in confined spaces. To bridge this gap, we introduce IndoorUAV, a novel benchmark and method specifically tailored for VLN with indoor UAVs. We begin by curating over 1,000 diverse and structurally rich 3D indoor scenes from the Habitat simulator. Within these environments, we simulate realistic UAV flight dynamics to collect diverse 3D navigation trajectories manually, further enriched through data augmentation techniques. Furthermore, we design an automated annotation pipeline to generate natural language instructions of varying granularity for each trajectory. This process yields over 16,000 high-quality trajectories, comprising the IndoorUAV-VLN subset, which focuses on long-horizon VLN. To support short-horizon planning, we segment long trajectories into sub-trajectories by selecting semantically salient keyframes and regenerating concise instructions, forming the IndoorUAV-VLA subset. Finally, we introduce IndoorUAV-Agent, a novel navigation model designed for our benchmark, leveraging task decomposition and multimodal reasoning. We hope IndoorUAV serves as a valuable resource to advance research on vision-language embodied AI in the indoor aerial navigation domain.

PDF Details DOI

JBHI Journal 2026 Journal Article

Integrating ECG and PCG Signals through a Dual-Modal ViT for Coronary Artery Disease Detection

Xu Liu
Ling You
Chengcong Lv
Mingyuan Chen
Lianhuan Wei
Yineng Zheng
Xingming Guo

Cardiovascular disease (CVD) remains the leading cause of mortality worldwide, with coronary artery disease (CAD) being the most prevalent form. To improve screening efficiency, there is a critical need for accurate, non-invasive, and cost-effective CAD detection methods. This study presents Co-Attention Dual-Modal ViT (CAD-ViT), a novel classification framework based on the Vision Transformer that integrates both electrocardiogram (ECG) and phonocardiogram (PCG) signals. Unlike prior approaches that process ECG and PCG features independently or fuse them through simple concatenation, the proposed model introduces two key modules: a Co-Attention mechanism that enables bidirectional cross-modal interaction to effectively capture complementary features between ECG and PCG signals, and a Dynamic Weighted Fusion (DWF) module that adaptively adjusts the contribution of each modality for robust feature fusion. CAD-ViT is evaluated on a private clinical dataset comprising 132 CAD and 101 non-CAD subjects, achieving an accuracy of 97. 08%, precision of 97. 18%, specificity of 98. 52%, F1-score of 97. 04, and recall of 96. 94%. Additional validation on two public datasets confirms the model’s robustness and generalization capability. These results demonstrate the effectiveness of the proposed approach and its potential for practical deployment in CAD screening using multimodal biosignals.

Details DOI

AAAI Conference 2026 Conference Paper

Let’s Think with Images Efficiently! An Interleaved-Modal Chain-of-Thought Reasoning Framework with Dynamic and Precise Visual Thoughts

Xu Liu
Yongheng Zhang
Qiguang Chen
Yao Li
Sheng Wang
Libo Qin

Recently, Interleaved-modal Chain-of-Thought (ICoT) reasoning has achieved remarkable success by leveraging both multimodal inputs and outputs, attracting increasing attention. While achieving promising performance, current ICoT methods still suffer from two major limitations: (1) Static Visual Thought Positioning, which statically inserts visual information at fixed steps, resulting in inefficient and inflexible reasoning; and (2) Broken Visual Thought Representation, which involves discontinuous and semantically incoherent visual tokens. To address these limitations, we introduce Interleaved-modal Chain-of-Thought reasoning with Dynamic and Precise Visual Thoughts (DaP-ICoT), which incorporates two key components: (1) Dynamic Visual Thought Integration adaptively introduces visual inputs based on reasoning needs, reducing redundancy and improving efficiency. (2) Precise Visual Thought Guidance ensures visual semantically coherent and contextually aligned representations. Experiments across multiple benchmarks and models demonstrate that DaP-ICoT achieves state-of-the-art performance. In addition, DaP-ICoT significantly reduces the number of inserted images, leading to a 72.6% decrease in token consumption, enabling more efficient ICoT reasoning.

PDF Details DOI

EAAI Journal 2026 Journal Article

Neural network-based in-loop filter based on dual-branch collaborative architecture

Cheolkon Jung
Xu Liu
Shaochong Wu
Jiang Han

In this paper, we propose a neural network-based in-loop filter (NNLF) for versatile video coding (VVC) intra coding based on a dual-branch collaborative architecture. The proposed NNLF leverages spatial-frequency feature fusion to effectively learn multi-type input features and adopts a dual-branch collaborative architecture to process global quantization features from the quantization parameter (QP) map branch and local artifacts from the prediction/partition frame branch separately. In the former branch, the multi-scale refinement block, driven by wavelet transform convolution (WTConv), suppresses inter-block brightness differences by operating on frequency-decomposed subbands. In the latter branch, the progressive residual enhancement network restores texture discontinuities in the predicted regions through a dual-attention mechanism. Moreover, the adaptive fusion block alleviates input feature conflicts based on two pixel-wise attention maps by learning the global features from the QP map and the local features from the prediction and partition frames, enabling an effective fusion of complementary information. Experimental results show that the proposed NNLF achieves average Bjøntegaard Delta (BD)-rate gains of 8. 43% (Y), 22. 16% (U), and 23. 00% (V) in peak signal-to-noise ratio (PSNR) over the VVC test model (VTM)-11. 0_neural network-based video coding (NNVC)-3. 0 anchor under All Intra (AI) configuration. The code is available at: https: //github. com/Xidian-Media-Lab/Dual-Branch-NNLF.

Details DOI

AAAI Conference 2026 Conference Paper

SOAR: Semi-Supervised Open-Vocabulary Aerial Object Detection via Dual-Aware Enhanced Prior Denoising

Xu Liu
Yihong Huang
Dan Zhang
Lingling Li
Long Sun
Licheng Jiao

Open-Vocabulary Object Detection (OVOD) shows promise in remote sensing (RS), but due to its unique value, there are challenges such as the predominance of background regions, sparse labels, limited semantic information, and difficulties in semi-supervised training. To tackle these challenges, we propose the Semi-Supervised Open-Vocabulary Aerial Object Detection with Dual-Perception Prior Denoising (SOAR), which explicitly models the background embeddings of each scene to indirectly construct foreground priors, thereby capitalizing on the abundant background information present in RS imagery. We further introduce a query enhancement module that integrates language and foreground prior information to enhance the effectiveness of query selection and feature augmentation. During the decoding stage of semi-supervised training, we perform denoising and reconstruction of the foreground priors to generate pseudo-labels that support the training process. Additionally, we address the sparsity of label information through expansion and aggregation techniques, further improving model performance. Experimental evaluations reveal that, in the open-vocabulary object detection task on the DIOR dataset, our method achieves a mean Average Precision (mAP) of 68.5% and Harmonic Mean (HM) of 55.9%, outperforming the previous state-of-the-art model’s mAP of 61.6% and HM of 53.6%. Our approach offers a novel solution to the open-vocabulary challenge in aerial object detection.

PDF Details DOI

AAMAS Conference 2025 Conference Paper

Adaptive Offline Data Replay in Offline-to-Online Reinforcement Learning

Xu Liu
Tong Yu
Shuai Li

Offline-to-online reinforcement learning combines the advantages of offline data utilization with online exploration to enhance sample efficiency and performance. A primary challenge lies in managing the distribution shift between offline and online data, which significantly impacts training effectiveness. Existing methods often employ fixed mixing ratios for data replay, but these require taskspecific tuning and may fail to generalize across different environments. To address this, we introduce a metric that evaluates policy quality relative to offline and online data, and propose a banditbased strategy to adjust the mixing ratio adaptively, optimizing policy quality during training. Experiments across diverse environments demonstrate that our approach outperforms static methods, offering robust adaptability and minimizing manual tuning.

PDF

EAAI Journal 2025 Journal Article

BV-NORM: A neural operator learning framework for parametric boundary value problems on complex geometric domains in engineering

Zhiliang Deng
Qinglu Meng
Yingguang Li
Xu Liu
Gengxiang Chen
Lu Chen
Changqing Liu
Xiaozhong Hao

Boundary Value Problems (BVPs) are extensively employed in engineering for process modelling and optimisation. These problems frequently involve complex geometries and require the massive solution of BVPs under different boundary conditions (BCs). Neural operators (NOs), capable of learning mappings between infinite-dimensional functions, present a potential solution for solving parametric BVPs. However, existing NOs are typically designed for scenarios where the input and output functions share the same domain, thus not applicable to BVPs in which the BC and solution functions are defined over different complex domains. Therefore, this study presents a novel deep learning framework called Boundary Value Neural Operator on Riemannian Manifolds (BV-NORM) for solving parametric BVPs involving complex geometric domains. BV-NORM introduces two sub-networks, Geometry-NET (Geo-NET) and Boundary condition-NET (BC-NET), to encode geometric information and boundary conditions. Consequently, the geometric information of the output domain can then be incorporated into the learning process. Furthermore, the Laplace kernel integration module of NORM is employed to construct the sub-networks, thereby enhancing the capacity to analyse complex geometric domains. The performance of the proposed method is evaluated in four benchmark cases, including toy Partial differential equation (PDE) cases and engineering applications, through comparisons with existing baseline neural operators. The experimental results validate that BV-NORM effectively addresses BVPs across various engineering scenarios.

Details DOI

EAAI Journal 2025 Journal Article

Fast prediction and compensation of curing deformation behaviours of composite parts with complex geometry based on neural operator on Riemannian manifolds

Lu Chen
Yingguang Li
Jingyan Su
Weiwei Xu
Lin Hu
Gengxiang Chen
Xu Liu
Xiaozhong Hao

Controlling the curing deformation of composite parts is becoming increasingly challenging with the ever-increasing performance requirements of aerospace equipment. Mould surface compensation, which adjusts the mould surface to minimise the discrepancy between the cured part geometry and the nominal part geometry, has become a primary deformation control way in engineering. The existing mirror compensation methods focus on the deformation dominated by spring-in and are challenging for complex deformation modes. Surrogate model-based shape optimisation provides a feasible idea, but establishing a surrogate model to predict curing deformation fields on complex part geometries remains a challenge. Therefore, this study explores a novel neural operator-driven framework for fast curing deformation prediction and compensation. A clustering-based deformation field segmentation method is proposed to manipulate the mould surface morphing using limited design variables. The neural operator on Riemannian manifolds is introduced for the first time to establish the surrogate model between the mould surface and the curing deformation fields on complex part geometries. To control the global error distribution of the composite part, two error metrics are designed to optimise the mould surface by genetic algorithm. The verification results show that the proposed framework exhibits significant potential in predicting and compensating for the curing deformation field of composite parts with complex geometry.

Details DOI

NeurIPS Conference 2025 Conference Paper

FlowNet: Modeling Dynamic Spatio-Temporal Systems via Flow Propagation

Yutong Feng
Xu Liu
Yutong Xia
Yuxuan Liang

Accurately modeling complex dynamic spatio-temporal systems requires capturing flow-mediated interdependencies and context-sensitive interaction dynamics. Existing methods, predominantly graph-based or attention-driven, rely on similarity-driven connectivity assumptions, neglecting asymmetric flow exchanges that govern system evolution. We propose Spatio-Temporal Flow, a physics-inspired paradigm that explicitly models dynamic node couplings through quantifiable flow transfers governed by conservation principles. Building on this, we design FlowNet, a novel architecture leveraging flow tokens as information carriers to simulate source-to-destination transfers via Flow Allocation Modules, ensuring state redistribution aligns with physical laws. FlowNet dynamically adjusts the interaction radius through an Adaptive Spatial Masking module, suppressing irrelevant noise while enabling context-aware propagation. A cascaded architecture enhances scalability and nonlinear representation capacity. Experiments demonstrate that FlowNet significantly outperforms existing SOTA approaches on seven metrics in the modeling of three real-world systems, validating its efficiency and physical interpretability. We establish a principled methodology for modeling complex systems through spatio-temporal flow interactions.

PDF Details

YNIMG Journal 2025 Journal Article

Investigating the role of spatial perception in literal and metaphorical language comprehension

Xu Liu
Jianshe Wu

The human brain seamlessly integrates information across cognitive domains, flexibly adapting to changing environmental demands. Yet, the neural mechanisms underlying this dynamic integration-especially how perceptual input shapes language comprehension-remain insufficiently understood. We investigated how vertical spatial perception is integrated into the processing of both literal and metaphorical spatial words. Time-frequency analyses revealed language-space congruency effects for both word types but with distinct oscillatory patterns: a theta-alpha-low beta ERS for literal spatial words, and a high beta-gamma ERD for metaphorical spatial words. However, these effects were not replicated in the ERP data. The ERP results demonstrated that, for literal spatial words, downward items elicited a more positive LPC, indicating an upward processing advantage, conversely, for metaphorical spatial (emotion-laden) words, downward (negative) items elicited more positive P200, N400, and LPC, indicating a downward processing advantage. Collectively, these findings show that spatial perception influences word processing through cross-modal interaction, supporting embodied cognition while refuting purely amodal accounts. Moreover, the discrepancies between time-frequency and ERP results, as well as the distinct oscillatory patterns and location processing advantages for literal versus metaphorical spatial words, challenge strong embodied view and provide novel evidence that spatial perception modulates language comprehension via context-dependent neural dynamics.

Details DOI

IJCAI Conference 2025 Conference Paper

Language-Guided Hybrid Representation Learning for Visual Grounding on Remote Sensing Images

Biao Liu
Xu Liu
Lingling Li
Licheng Jiao
Fang Liu
Xinyu Sun
Youlin Huang

Visual grounding (VG) refers to detecting the specific objects in images based on linguistic expressions, and it has profound significance in the advanced interpretation of natural images. In remote sensing image interpretation, visual grounding is limited by characteristics such as the complex scenes and diverse object sizes. To solve this problem, we propose a novel remote sensing visual grounding (RSVG) framework, named language-guided hybrid representation learning Transformer (LGFormer). Specifically, we designed a multimodal dual-encoder Transformer structure called the adaptive multimodal feature fusion module. This structure innovatively integrates text and visual features as hybrid queries, enabling early-stage decoding queries to perceive the target position accurately. Then, the different modal information from the dual encoders is aggregated by hybrid queries to obtain the final object embedding for coordinate regression. Besides, a multi-scale cross-modal feature enhancement module (MSCM) is designed to enhance the self-representation of the extracted text and visual features and align them semantically. As for the hybrid queries, we use linguistic guidance to select visual features as the visual part and sentence-level features as the textual part. Finally, the LGFormer model we designed achieved the best results compared to existing models on the DIOR-RSVG and OPT-RSVG datasets.

PDF Details DOI

ICML Conference 2025 Conference Paper

Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation

Xiao Huang
Xu Liu
Enze Zhang
Tong Yu 0001
Shuai Li 0010

Offline-to-online Reinforcement Learning (O2O RL) aims to perform online fine-tuning on an offline pre-trained policy to minimize costly online interactions. Existing work used offline datasets to generate data that conform to the online data distribution for data augmentation. However, generated data still exhibits a gap with the online data, limiting overall performance. To address this, we propose a new data augmentation approach, Classifier-Free Diffusion Generation (CFDG). Without introducing additional classifier training overhead, CFDG leverages classifier-free guidance diffusion to significantly enhance the generation quality of offline and online data with different distributions. Additionally, it employs a reweighting method to enable more generated data to align with the online data, enhancing performance while maintaining the agent’s stability. Experimental results show that CFDG outperforms replaying the two data types or using a standard diffusion model to generate new data. Our method is versatile and can be integrated with existing offline-to-online RL algorithms. By implementing CFDG to popular methods IQL, PEX and APL, we achieve a notable 15% average improvement in empirical performance on the D4RL benchmark such as MuJoCo and AntMaze.

Details

ICLR Conference 2025 Conference Paper

Online-to-Offline RL for Agent Alignment

Xu Liu
Haobo Fu
Stefano V. Albrecht
Qiang Fu
Shuai Li

Reinforcement learning (RL) has shown remarkable success in training agents to achieve high-performing policies, particularly in domains like Game AI where simulation environments enable efficient interactions. However, despite their success in maximizing these returns, such online-trained policies often fail to align with human preferences concerning actions, styles, and values. The challenge lies in efficiently adapting these online-trained policies to align with human preferences, given the scarcity and high cost of collecting human behavior data. In this work, we formalize the problem as *online-to-offline* RL and propose ALIGNment of Game AI to Preferences (ALIGN-GAP), an innovative approach for the alignment of well-trained game agents to human preferences. Our method features a carefully designed reward model that encodes human preferences from limited offline data and incorporates curriculum-based preference learning to align RL agents with targeted human preferences. Experiments across diverse environments and preference types demonstrate the performance of ALIGN-GAP, achieving effective alignment with human preferences.

Details

EAAI Journal 2025 Journal Article

Rapid prediction of thermal stress on satellites via domain decomposition-based Hybrid Fourier Neural Operator

Kangrui Zhou
Wei Peng
Xiaoya Zhang
Xu Liu
Wen Yao

Rapid thermal stress analysis is crucial for the thermal design of satellites. To overcome the disadvantages of traditional algorithms in terms of efficiency, deep learning methods have been used to tackle these problems. However, using uniform grid-based techniques is challenging when faced with complex geometric shapes. To address this, we introduce the domain decomposition-based Hybrid Fourier Neural Operator (HFNO), a comprehensive framework for learning a multi-scale and end-to-end operator on two-dimensional point clouds. We then propose two decomposition metrics: a stress gradient-based metric for scenarios with prior knowledge of training data, and a mesh density-based metric for scenarios without prior knowledge. Leveraging K-Dimension tree-based domain decomposition optimized via Monte Carlo tree search, we decompose the computational domain into several disjoint rectangular subdomains. In the proposed hybrid framework, a Geometry-aware Fourier Neural Operator (Geo-FNO) is used to deal with subdomains with high-frequency information, while a Non-Uniform Fourier Neural Operator (NU-FNO) is used to deal with subdomains with low-frequency information. This framework effectively combines the advantages of two Fourier Neural Operator variants, overcoming the issue of large prediction errors on the subdomains with high-frequency information and ensuring stable prediction performance across different positions. Furthermore, we introduce a boundary loss term during the training process to enhance continuity across subdomain boundaries. The numerical results demonstrate that our method achieves a superior balance between efficiency and precision, surpassing that of a single algorithm.

Details DOI

NeurIPS Conference 2025 Conference Paper

Vocabulary-Guided Gait Recognition

Panjian Huang
Saihui Hou
Chunshui Cao
Xu Liu
Yongzhen Huang

What is a gait? Appearance-based gait networks consider a gait as the human shape and motion information from images. Model-based gait networks treat a gait as the human inherent structure from points. However, the considerations remain vague for humans to comprehend truly. In this work, we introduce a novel paradigm Vocabulary-Guided Gait Recognition, dubbed Gait-World, which attempts to explore gait concepts through human vocabularies with Vision-Language Models (VLMs). Despite VLMs have achieved the remarkable progress in various vision tasks, the cognitive capability regarding gait modalities remains limited. The success element in Gait-World is the proper vocabulary prompt where this paradigm carefully selects gait cycle actions as Vocabulary Base, bridging the gait and vocabulary feature spaces and further promoting human understanding for the gait. How to extract gait features? Although previous gait networks have made significant progress, learning solely from gait modalities on limited gait databases makes it difficult to learn robust gait features for practicality. Therefore, we propose the first Gait-World model, dubbed $\alpha$-Gait, which guides the gait network learning with universal vocabulary knowledge from VLMs. However, due to the heterogeneity of the modalities, directly integrating vocabulary and gait features is highly challenging as they reside in different embedding spaces. To address the issues, $\alpha$-Gait designs Vocabulary Relation Mapper and Gait Fine-grained Detector to map and establish vocabulary relations in the gait space for detecting corresponding gait features. Extensive experiments on CASIA-B, CCPG, SUSTech1K, Gait3D and GREW reveal the potential value and research directions of vocabulary information from VLMs in the gait field.

PDF Details

JBHI Journal 2024 Journal Article

Embracing Large Natural Data: Enhancing Medical Image Analysis via Cross-Domain Fine-Tuning

Qiankun Li
Xiaolong Huang
Bo Fang
Huabao Chen
Siyuan Ding
Xu Liu

With the rapid advancements of Big Data and computer vision, many large-scale natural visual datasets are proposed, such as ImageNet-21K, LAION-400M, and LAION-2B. These large-scale datasets significantly improve the robustness and accuracy of models in the natural vision domain. However, the field of medical images continues to face limitations due to relatively small-scale datasets. In this article, we propose a novel method to enhance medical image analysis across domains by leveraging pre-trained models on large natural datasets. Specifically, a Cross-Domain Transfer Module (CDTM) is proposed to transfer natural vision domain features to the medical image domain, facilitating efficient fine-tuning of models pre-trained on large datasets. In addition, we design a Staged Fine-Tuning (SFT) strategy in conjunction with CDTM to further improve the model performance. Experimental results demonstrate that our method achieves state-of-the-art performance on multiple medical image datasets through efficient fine-tuning of models pre-trained on large natural datasets.

Details DOI

YNIMG Journal 2024 Journal Article

Establishing the relationship between subjective perception and neural responses: Insights from correlation analysis and representational similarity analysis

Xu Liu
Shiyu Wei
Xiangyue Zhao
Yanzhi Bi
Li Hu

Exploring the relationship between sensory perception and brain responses holds important theoretical and clinical implications. However, commonly used methodologies like correlation analysis performed either intra- or inter- individually often yield inconsistent results across studies, limiting their generalizability. Representational similarity analysis (RSA), a method that assesses the perception-response relationship by calculating the correlation between behavioral and neural patterns, may offer a fresh perspective to reveal novel findings. Here, we delivered a series of graded sensory stimuli of four modalities (i. e. , nociceptive somatosensory, non-nociceptive somatosensory, visual, and auditory) to/near the left or right hand of 107 healthy subjects and collected their single-trial perceptual ratings and electroencephalographic (EEG) responses. We examined the relationship between sensory perception and brain responses using within- and between-subject correlation analysis and RSA, and assessed their stability across different numbers of subjects and trials. We found that within-subject and between-subject correlations yielded distinct results: within-subject correlation revealed strong and reliable correlations between perceptual ratings and most brain responses, while between-subject correlation showed weak correlations that were vulnerable to the change of subject number. In addition to verifying the correlation results, RSA revealed some novel findings, i. e. , correlations between behavioral and neural patterns were observed in some additional neural responses, such as “γ-ERS” in the visual modality. RSA results were sensitive to the trial number, but not to the subject number, suggesting that consistent results could be obtained for studies with relatively small sample sizes. In conclusion, our study provides a novel perspective on establishing the relationship between behavior and brain activity, emphasizing that RSA holds promise as a method for exploring this pattern relationship in future research.

Details DOI

JMLR Journal 2024 Journal Article

More Efficient Estimation of Multivariate Additive Models Based on Tensor Decomposition and Penalization

Xu Liu
Heng Lian
Jian Huang

We consider parsimonious modeling of high-dimensional multivariate additive models using regression splines, with or without sparsity assumptions. The approach is based on treating the coefficients in the spline expansions as a third-order tensor. Note the data does not have tensor predictors or tensor responses, which distinguishes our study from the existing ones. A Tucker decomposition is used to reduce the number of parameters in the tensor. We also combined the Tucker decomposition with penalization to enable variable selection. The proposed method can avoid the statistical inefficiency caused by estimating a large number of nonparametric functions. We provide sufficient conditions under which the proposed tensor-based estimators achieve the optimal rate of convergence for the nonparametric regression components. We conduct simulation studies to demonstrate the effectiveness of the proposed novel approach in fitting high-dimensional multivariate additive models and illustrate its application on a breast cancer copy number variation and gene expression data set. [abs] [ pdf ][ bib ] &copy JMLR 2024. ( edit, beta )

PDF Details

IJCAI Conference 2024 Conference Paper

Purpose Enhanced Reasoning through Iterative Prompting: Uncover Latent Robustness of ChatGPT on Code Comprehension

Yi Wang
Qidong Zhao
Dongkuan Xu
Xu Liu

Code comments are crucial for gaining in-depth insights to facilitate code comprehension. The key to obtaining these insights lies in precisely summarizing the main purpose of the code. Recent approaches on code comment generation lie in prompting large language models (LLMs) such as ChatGPT, instead of training/fine-tuning specific models. Although ChatGPT demonstrates an impressive performance in code comprehension, it still suffers from robustness challenges in consistently producing high-quality code comments. This is because ChatGPT prioritizes the semantics of code tokens, which makes it vulnerable to commonly encountered benign perturbations such as variable name replacements. This study proposes a modular prompting paradigm Perthept to effectively mitigate the negative effects caused by such minor perturbations. Perthept iteratively enhances the reasoning depth to reach the main purpose of the code. Perthept demonstrates robustness under the scenario where there is stochasticity or unreliability in ChatGPT's responses. We give a comprehensive evaluation across four public datasets to show the consistent robustness improvement with our proposed methodology over other models.

PDF Details DOI

AAAI Conference 2024 Conference Paper

QAGait: Revisit Gait Recognition from a Quality Perspective

Zengbin Wang
Saihui Hou
Man Zhang
Xu Liu
Chunshui Cao
Yongzhen Huang
Peipei Li
Shibiao Xu

Gait recognition is a promising biometric method that aims to identify pedestrians from their unique walking patterns. Silhouette modality, renowned for its easy acquisition, simple structure, sparse representation, and convenient modeling, has been widely employed in controlled in-the-lab research. However, as gait recognition rapidly advances from in-the-lab to in-the-wild scenarios, various conditions raise significant challenges for silhouette modality, including 1) unidentifiable low-quality silhouettes (abnormal segmentation, severe occlusion, or even non-human shape), and 2) identifiable but challenging silhouettes (background noise, non-standard posture, slight occlusion). To address these challenges, we revisit gait recognition pipeline and approach gait recognition from a quality perspective, namely QAGait. Specifically, we propose a series of cost-effective quality assessment strategies, including Maxmial Connect Area and Template Match to eliminate background noises and unidentifiable silhouettes, Alignment strategy to handle non-standard postures. We also propose two quality-aware loss functions to integrate silhouette quality into optimization within the embedding space. Extensive experiments demonstrate our QAGait can guarantee both gait reliability and performance enhancement. Furthermore, our quality assessment strategies can seamlessly integrate with existing gait datasets, showcasing our superiority. Code is available at https://github.com/wzb-bupt/QAGait.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecasting

Qingxiang Liu
Xu Liu
Chenghao Liu
Qingsong Wen
Yuxuan Liang

Unlike natural language processing and computer vision, the development of Foundation Models (FMs) for time series forecasting is blocked due to data scarcity. While recent efforts are focused on building such FMs by unlocking the potential of language models (LMs) for time series analysis, dedicated parameters for various downstream forecasting tasks need training, which hinders the common knowledge sharing across domains. Moreover, data owners may hesitate to share the access to local data due to privacy concerns and copyright protection, which makes it impossible to simply construct a FM on cross-domain training instances. To address these issues, we propose Time-FFM, a Federated Foundation Model for Time series forecasting by leveraging pretrained LMs. Specifically, we begin by transforming time series into the modality of text tokens. To bootstrap LMs for time series reasoning, we propose a prompt adaption module to determine domain-customized prompts dynamically instead of artificially. Given the data heterogeneity across domains, we design a personalized federated training strategy by learning global encoders and local prediction heads. Our comprehensive experiments indicate that Time-FFM outperforms state-of-the-arts and promises effective few-shot and zero-shot forecaster. The code is available at https: //github. com/CityMind-Lab/NeurIPS24-Time-FFM/tree/main.

PDF Details DOI

AAAI Conference 2024 Conference Paper

ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization

Hao Wang
Fang Liu
Licheng Jiao
Jiahao Wang
Zehua Hao
Shuo Li
Lingling Li
Puhua Chen

Pre-trained vision-language(V-L) models such as CLIP have demonstrated impressive Zero-Shot performance in many downstream tasks. Since adopting contrastive video-text pairs methods like CLIP to video tasks is limited by its high cost and scale, recent approaches focus on efficiently transferring the image-based CLIP to the video domain. A major finding is that fine-tuning the pre-trained model to achieve strong fully supervised performance leads to low zero shot, few shot, and base to novel generalization. Instead, freezing the backbone network to maintain generalization ability weakens fully supervised performance. Otherwise, no single prompt tuning branch consistently performs optimally. In this work, we proposed a multimodal prompt learning scheme that balances supervised and generalized performance. Our prompting approach contains three sections: 1) Independent prompt on both the vision and text branches to learn the language and visual contexts. 2) Inter-modal prompt mapping to ensure mutual synergy. 3) Reducing the discrepancy between the hand-crafted prompt (a video of a person doing [CLS]) and the learnable prompt, to alleviate the forgetting about essential video scenarios. Extensive validation of fully supervised, zero-shot, few-shot, base-to-novel generalization settings for video recognition indicates that the proposed approach achieves competitive performance with less commute cost.

PDF Details DOI

EAAI Journal 2023 Journal Article

A computation offloading algorithm based on multi-objective evolutionary optimization in mobile edge computing

Zheng-yi Chai
Xu Liu
Ya-Lun Li

For computation offloading problem (COP) in mobile edge computing (MEC), the energy consumption of terminal equipments(TEs) and the delay of mobile equipment applications are two optimization goals. In real life, terminal equipment is dynamic, and their number, mobility, and continuous changes in wireless channels will affect the balance between the mentioned energy consumption and delay. Different from available works, we model the COP in MEC as a dynamic multi-objective problem (DMOP) in this paper, and propose an improved dynamic multi-objective evolutionary optimization based on decomposition (DMOEA/D) to solve it, namely DMOEA/D-COPMEC. In the proposed algorithm, the environmental change is detected by a fixed detector, and whether the current change is similar to the historical change is determined. If so, the difference prediction is used to re-locate the population individual in the new MEC environment, otherwise, the memory-based strategy is used to response environmental change. In MOEA/D, an adaptive weight adjustment strategy based on chain segmentation (CS) is adopted to generate a set of uniformly distributed weight vectors. The simulation results show that the proposed algorithm can better balance the application delay and the terminal energy consumption if there is environment change. The solution set is closer to reality and better than the related algorithms.

Details DOI

AAAI Conference 2023 Conference Paper

Coupling Artificial Neurons in BERT and Biological Neurons in the Human Brain

Xu Liu
Mengyue Zhou
Gaosheng Shi
Yu Du
Lin Zhao
Zihao Wu
David Liu
Tianming Liu

Linking computational natural language processing (NLP) models and neural responses to language in the human brain on the one hand facilitates the effort towards disentangling the neural representations underpinning language perception, on the other hand provides neurolinguistics evidence to evaluate and improve NLP models. Mappings of an NLP model’s representations of and the brain activities evoked by linguistic input are typically deployed to reveal this symbiosis. However, two critical problems limit its advancement: 1) The model’s representations (artificial neurons, ANs) rely on layer-level embeddings and thus lack fine-granularity; 2) The brain activities (biological neurons, BNs) are limited to neural recordings of isolated cortical unit (i.e., voxel/region) and thus lack integrations and interactions among brain functions. To address those problems, in this study, we 1) define ANs with fine-granularity in transformer-based NLP models (BERT in this study) and measure their temporal activations to input text sequences; 2) define BNs as functional brain networks (FBNs) extracted from functional magnetic resonance imaging (fMRI) data to capture functional interactions in the brain; 3) couple ANs and BNs by maximizing the synchronization of their temporal activations. Our experimental results demonstrate 1) The activations of ANs and BNs are significantly synchronized; 2) the ANs carry meaningful linguistic/semantic information and anchor to their BN signatures; 3) the anchored BNs are interpretable in a neurolinguistic context. Overall, our study introduces a novel, general, and effective framework to link transformer-based NLP models and neural activities in response to language and may provide novel insights for future studies such as brain-inspired evaluation and development of NLP models.

PDF Details DOI

AAMAS Conference 2023 Conference Paper

CraftEnv: A Flexible Collective Robotic Construction Environment for Multi-Agent Reinforcement Learning

Rui Zhao
Xu Liu
Yizheng Zhang
Minghao Li
Cheng Zhou
Shuai Li
Lei Han

CraftEnv is a flexible Collective Robotic Construction (CRC) environment for Multi-Agent Reinforcement Learning (MARL) research. CraftEnv can be used to study how artificial intelligent agents may learn to cooperate and solve complex real world tasks, such as collective construction and intelligent warehousing. The environment contains a set of collective construction tasks, which require a group of robotic vehicles to cooperate and learn to build different constructions efficiently. There are different elements in the CraftEnv, such as smartcars, blocks, and slopes. The smartcars can use the blocks and slopes to build different structures. The CraftEnv is highly flexible and simple to use, which enables creative and quick task-designs. The environment is written in python and can be rendered using PyBullet. The simulation is built based on real world robotic systems, designed with real-world constraints in mind. The learned policy can be transferred to the real world robotic system. CraftEnv is tailored for effective use by the research community and pushing forward collective intelligence and swarm technology.

PDF

NeurIPS Conference 2023 Conference Paper

Deciphering Spatio-Temporal Graph Forecasting: A Causal Lens and Treatment

Yutong Xia
Yuxuan Liang
Haomin Wen
Xu Liu
Kun Wang
Zhengyang Zhou
Roger Zimmermann

Spatio-Temporal Graph (STG) forecasting is a fundamental task in many real-world applications. Spatio-Temporal Graph Neural Networks have emerged as the most popular method for STG forecasting, but they often struggle with temporal out-of-distribution (OoD) issues and dynamic spatial causation. In this paper, we propose a novel framework called CaST to tackle these two challenges via causal treatments. Concretely, leveraging a causal lens, we first build a structural causal model to decipher the data generation process of STGs. To handle the temporal OoD issue, we employ the back-door adjustment by a novel disentanglement block to separate the temporal environments from input data. Moreover, we utilize the front-door adjustment and adopt edge-level convolution to model the ripple effect of causation. Experiments results on three real-world datasets demonstrate the effectiveness of CaST, which consistently outperforms existing methods with good interpretability. Our source code is available at https: //github. com/yutong-xia/CaST.

PDF Details

NeurIPS Conference 2023 Conference Paper

LargeST: A Benchmark Dataset for Large-Scale Traffic Forecasting

Xu Liu
Yutong Xia
Yuxuan Liang
Junfeng Hu
Yiwei Wang
Lei Bai
Chao Huang
Zhenguang Liu

Road traffic forecasting plays a critical role in smart city initiatives and has experienced significant advancements thanks to the power of deep learning in capturing non-linear patterns of traffic data. However, the promising results achieved on current public datasets may not be applicable to practical scenarios due to limitations within these datasets. First, the limited sizes of them may not reflect the real-world scale of traffic networks. Second, the temporal coverage of these datasets is typically short, posing hurdles in studying long-term patterns and acquiring sufficient samples for training deep models. Third, these datasets often lack adequate metadata for sensors, which compromises the reliability and interpretability of the data. To mitigate these limitations, we introduce the LargeST benchmark dataset. It encompasses a total number of 8, 600 sensors in California with a 5-year time coverage and includes comprehensive metadata. Using LargeST, we perform in-depth data analysis to extract data insights, benchmark well-known baselines in terms of their performance and efficiency, and identify challenges as well as opportunities for future research. We release the datasets and baseline implementations at: https: //github. com/liuxu77/LargeST.

PDF Details

AAAI Conference 2022 Conference Paper

Random Mapping Method for Large-Scale Terrain Modeling

Xu Liu
Decai Li
Yuqing He

The vast amount of data captured by robots in large-scale environments brings the computing and storage bottlenecks to the typical methods of modeling the spaces the robots travel in. In order to efficiently construct a compact terrain model from uncertain, incomplete point cloud data of large-scale environments, in this paper, we first propose a novel feature mapping method, named random mapping, based on the fast random construction of base functions, which can efficiently project the messy points in the low-dimensional space into the high-dimensional space where the points are approximately linearly distributed. Then, in this mapped space, we propose to learn a continuous linear regression model to represent the terrain. We show that this method can model the environments in much less computation time, memory consumption, and access time, with high accuracy. Furthermore, the models possess the generalization capabilities comparable to the performances on the training set, and its inference accuracy gradually increases as the random mapping dimension increases. To better solve the large-scale environmental modeling problem, we adopt the idea of parallel computing to train the models. This strategy greatly reduces the wall-clock time of calculation without losing much accuracy. Experiments show the effectiveness of the random mapping method and the effects of some important parameters on its performance. Moreover, we evaluate the proposed terrain modeling method based on the random mapping method and compare its performances with popular typical methods and state-of-art methods.

PDF Details

EAAI Journal 2022 Journal Article

Temperature field inversion of heat-source systems via physics-informed neural networks

Xu Liu
Wei Peng
Zhiqiang Gong
Weien Zhou
Wen Yao

Temperature field inversion of heat-source systems (TFI-HSS) with limited observations is essential to monitor the system health. Although some methods such as interpolation have been proposed to solve TFI-HSS, those existing methods ignore correlations between data constraints and physics constraints, causing the low precision. In this work, we develop a physics-informed neural network-based temperature field inversion (PINN-TFI) method to solve the TFI-HSS task and a coefficient matrix condition number based position selection of observations (CMCN-PSO) method to select optimal positions of noisy observations. For the TFI-HSS task, the PINN-TFI method encodes constrain terms into the loss function and thus the task is transformed into an optimization problem of minimizing the loss function. In addition, we have found that noise significantly affect reconstruction performances of the PINN-TFI method. To alleviate the effect of noises in observations, we propose the CMCN-PSO method to find optimal positions, where the condition number of observations is used to evaluate positions. The results demonstrate that the PINN-TFI method can significantly improve prediction precisions and the CMCN-PSO method can find good positions to improve the robustness of the PINN-TFI method.

Details DOI

TIST Journal 2022 Journal Article

The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems

Sixu Hu
Yuan Li
Xu Liu
Qinbin Li
Zhaomin Wu
Bingsheng He

This article presents and characterizes an Open Application Repository for Federated Learning (OARF), a benchmark suite for federated machine learning systems. Previously available benchmarks for federated learning (FL) have focused mainly on synthetic datasets and use a limited number of applications. OARF mimics more realistic application scenarios with publicly available datasets as different data silos in image, text, and structured data. Our characterization shows that the benchmark suite is diverse in data size, distribution, feature distribution, and learning task complexity. The extensive evaluations with reference implementations show the future research opportunities for important aspects of FL systems. We have developed reference implementations, and evaluated the important aspects of FL, including model accuracy, communication cost, throughput, and convergence time. Through these evaluations, we discovered some interesting findings such as FL can effectively increase end-to-end throughput. The code of OARF is publicly available on GitHub. 1

Details DOI

IROS Conference 2021 Conference Paper

Exploring Imitation Learning for Autonomous Driving with Feedback Synthesizer and Differentiable Rasterization

Jinyun Zhou
Rui Wang
Xu Liu
Yifei Jiang
Shu Jiang
Jiaming Tao
Jinghao Miao
Shiyu Song

We present a learning-based planner that aims to robustly drive a vehicle by mimicking human drivers’ driving behavior. We leverage a mid-to-mid approach that allows us to manipulate the input to our imitation learning network freely. With that in mind, we propose a novel feedback synthesizer for data augmentation. It allows our agent to gain more driving experience in various previously unseen environments that are likely to encounter, thus improving overall performance. This is in contrast to prior works that rely purely on random synthesizers. Furthermore, rather than completely commit to imitating, we introduce task losses that penalize undesirable behaviors, such as collision, off-road, and so on. Unlike prior works, this is done by introducing a differentiable vehicle rasterizer that directly converts the waypoints output by the network into images. This effectively avoids the usage of heavyweight ConvLSTM networks, therefore, yields a faster model inference time. About the network architecture, we exploit an attention mechanism that allows the network to reason critical objects in the scene and produce better interpretable attention heatmaps. To further enhance the safety and robustness of the network, we add an optional optimization-based post-processing planner improving the driving comfort. We comprehensively validate our method’s effectiveness in different scenarios that are specifically created for evaluating self-driving vehicles. Results demonstrate that our learning-based planner achieves high intelligence and can handle complex situations. Detailed ablation and visualization analysis are included to further demonstrate each of our proposed modules’ effectiveness in our method.

Details

NeurIPS Conference 2020 Conference Paper

Group Contextual Encoding for 3D Point Clouds

Xu Liu
Chengtao Li
Jian Wang
Jingbo Wang
Boxin Shi
Xiaodong He

Global context is crucial for 3D point cloud scene understanding tasks. In this work, we extended the contextual encoding layer that was originally designed for 2D tasks to 3D Point Cloud scenarios. The encoding layer learns a set of code words in the feature space of the 3D point cloud to characterize the global semantic context, and then based on these code words, the method learns a global contextual descriptor to reweight the featuremaps accordingly. Moreover, compared to 2D scenarios, data sparsity becomes a major issue in 3D point cloud scenarios, and the performance of contextual encoding quickly saturates when the number of code words increases. To mitigate this problem, we further proposed a group contextual encoding method, which divides the channel into groups and then performs encoding on group-divided feature vectors. This method facilitates learning of global context in grouped subspace for 3D point clouds. We evaluate the effectiveness and generalizability of our method on three widely-studied 3D point cloud tasks. Experimental results have shown that the proposed method outperformed the VoteNet remarkably with 3 mAP on the benchmark of SUN-RGBD, with the metrics of mAP@ 0. 25, and a much greater margin of 6. 57 mAP on ScanNet with the metrics of mAP@ 0. 5. Compared to the baseline of PointNet++, the proposed method leads to an accuracy of 86 %, outperforming the baseline by 1. 5 %. Our proposed method have outperformed the non-grouping baseline methods across the board and establishes new state-of-the-art on these benchmarks.

PDF Details