Jun Liu

EAAI Journal 2026 Journal Article

A multi-modal multi-task learning network for intelligent parameter measurement in gas–liquid two-phase flow

• Hanqing Chen
• Zhiqiang Zhao
• Bang Zhou
• Ruiqi Wang
• Mengyu Li
• Wei Li
• Jun Liu
• Weidong Cao

Accurate identification of flow patterns and reliable measurement of phase fraction are fundamental for monitoring and control in gas–liquid two-phase flow systems. Conventional sensing and modeling approaches, however, are often constrained by limited spatial resolution and adaptability to dynamic operating conditions. A multi-modal multi-task learning network (MMLNet) is proposed, which integrates spatially distributed conductance time-series signals acquired from a custom-designed sensor with synchronized high-speed flow images. The network adopts a dual-branch architecture, where modality-specific backbones are constructed using multi-scale depthwise separable convolutions, followed by attention-driven cross-modal interaction and a per-token sample gate for adaptive fusion. Under a unified multi-task objective, MMLNet jointly optimizes flow pattern classification and gas volume fraction (GVF) regression, thereby exploiting the inherent correlation between the two tasks to improve accuracy and generalization. Experimental results show that MMLNet achieves 99. 88% accuracy in flow pattern classification, with a mean absolute error (MAE) of 0. 63%, and a mean absolute percentage error (MAPE) of 2. 23% for GVF prediction, outperforming state-of-the-art baselines. These results highlight the potential of MMLNet as a scalable soft-sensing solution for multiphase flow monitoring.

AAAI Conference 2026 Conference Paper

AdaptCLIP: Adapting CLIP for Universal Visual Anomaly Detection

• Bin-Bin Gao
• Yue Zhou
• Jiangtao Yan
• Yuezhi Cai
• Weixi Zhang
• Meng Wang
• Jun Liu
• Yong Liu

Universal visual anomaly detection aims to identify anomalies from novel or unseen vision domains without additional fine-tuning, which is critical in open scenarios. Recent studies have demonstrated that pre-trained vision-language models like CLIP exhibit strong generalization with just zero or a few normal images. However, existing methods struggle to design prompt templates, handle complex token interactions, or require fine-tuning on target domains, resulting in limited flexibility. In this work, we present a simple yet effective AdaptCLIP based on two key insights. First, adaptive visual and textual representations should be learned alternately rather than jointly. Second, comparative learning between query and normal image prompt should incorporate both contextual and aligned residual features, rather than relying solely on residual features. AdaptCLIP treats CLIP models as a foundational service, adding only three simple adapters, visual adapter, textual adapter, and prompt-query adapter, at its input or output ends. AdaptCLIP supports zero-/few-shot generalization across domains and provides a training-free approach on target domains once trained on a base dataset. AdaptCLIP achieves state-of-the-art performance on 12 anomaly detection benchmarks from industrial and medical domains, significantly outperforming existing competitive methods.

EAAI Journal 2026 Journal Article

An interpretable power system transient stability assessment method with expert guiding neural regression tree

• Hanxuan Wang
• Na Lu
• Zixuan Wang
• Jiacheng Liu
• Jun Liu

Deep neural networks have achieved great success in power system transient stability assessment (TSA); however, their black-box nature severely limits industrial application. Although numerous studies have explored the interpretability of network solutions, several challenges remain unsolved: (1) the discrepancy between widely accepted power system knowledge and the generated interpretive rules being large, (2) achieving an optimal balance between accuracy and interpretability remains difficult. To address these issues, an interpretable TSA model with Expert guiding Neural Regression Tree (ENRT) is proposed. In ENRT, a specialized nonlinear regression tree is introduced to approximate the deep learning-based TSA model, with its decision paths simulating the black-box reasoning process of the neural network. The nonlinear terms in the tree model are extracted from the power flow calculation model with expert knowledge, ensuring that the generated interpretive rules better align with human cognition. By regularizing the neural network with the average decision depth of the nonlinear regression tree, we establish a connection between the neural network and the tree model at the training level, achieving a better trade-off between accuracy and interpretability. Our experiments across multiple TSA tasks demonstrate that ENRT effectively balances accuracy and interpretability, achieving a 97. 1% fidelity between the neural network and the tree model, along with an accuracy of 94. 3%. Furthermore, the generated interpretive rules capture nonlinear terms with clear physical significance, aligning more closely with power system knowledge.

EAAI Journal 2026 Journal Article

Enhancing multivariate weather forecasting via temporal attention and spatiotemporal fusion

• Jun Liu
• Zhibo Kong
• Xiaoying Wang
• Li Wu
• Guojing Zhang

Accurate multivariate weather forecasting underpins agriculture, transportation, and hazard preparedness, yet numerical weather prediction (NWP) is computationally intensive and purely data-driven artificial intelligence (AI) models often struggle with scale-coupled spatiotemporal dependencies. We introduce the Cross-Spatiotemporal Fusion Model (C-STFM), a lightweight deep-learning framework based on a U-shaped encoder–decoder convolutional neural network (U-Net) and built from stacked fusion blocks that combine Sequential Temporal Attention Fusion (STAF) and a Cross-Spatiotemporal Fusion Unit (CSFU). STAF performs ordered, segment-wise temporal fusion with carry-over, while CSFU enables mutually modulated temporal and spatial attention to realize explicit cross-variable interaction. Using the Fifth-Generation reanalysis from the European Centre for Medium-Range Weather Forecasts (ERA5) over Qinghai Province, we evaluate C-STFM against 17 representative convolutional, recurrent, video-prediction, and transformer baselines under a unified split and preprocessing pipeline. With a 12-hour input window, we report both 12-hour-ahead and 24-hour-ahead forecasts. Across six variables and four metrics — root mean square error (RMSE), mean squared error (MSE), mean absolute error (MAE), and anomaly correlation coefficient (ACC) — C-STFM achieves the lowest aggregate MSE at both horizons and shows statistically significant improvements over the best single baseline on all metrics (paired permutation tests with 95% bootstrap confidence intervals, false discovery rate (FDR) controlled; p < 1 × 1 0 − 4 ). Ablation studies isolate the contributions of STAF and CSFU (on/off), the number of stacked fusion blocks (depth), and STAF temporal segmentation, revealing that both modules are necessary and that moderate depth with four temporal segments provides the best accuracy–efficiency trade-off. Efficiency measurements (parameter counts, wall-clock training time, and inference throughput) indicate that gains are not obtained simply by increasing model capacity. Overall, C-STFM improves the stability of lead-time skill and the fidelity of multivariate spatial patterns, offering a practical path toward fast and accurate short-term weather prediction. Source code is available at https: //github. com/liusir1632/C-STFM/tree/master

AAAI Conference 2026 Conference Paper

Exploring Category-level Articulated Object Pose Tracking on SE(3) Manifolds

• Xianhui Meng
• Yukang Huo
• Li Zhang
• Liu Liu
• Haonan Jiang
• Yan Zhong
• Pingrui Zhang
• Cewu Lu

Articulated objects are prevalent in daily life and robotic manipulation tasks. However, compared to rigid objects, pose tracking for articulated objects remains an underexplored problem due to their inherent kinematic constraints. To address these challenges, this work proposes a novel point-pair-based pose tracking framework, termed PPF-Tracker. The proposed framework first performs quasi-canonicalization of point clouds in the SE(3) Lie group space, and then models articulated objects using Point Pair Features (PPF) to predict pose voting parameters by leveraging the invariance properties of SE(3). Finally, semantic information of joint axes is incorporated to impose unified kinematic constraints across all parts of the articulated object. PPF-Tracker is systematically evaluated on both synthetic datasets and real-world scenarios, demonstrating strong generalization across diverse and challenging environments. Experimental results highlight the effectiveness and robustness of PPF-Tracker in multi-frame pose tracking of articulated objects. We believe this work can foster advances in robotics, embodied intelligence, and augmented reality.

AAAI Conference 2026 Conference Paper

From Detection to Diagnosis: Advancing Hallucination Analysis with Automated Data Synthesis

• Yanyi Liu
• Qingwen Yang
• Tiezheng Guo
• Feiyu Qu
• Jun Liu
• Yingyou Wen

Hallucinations in Large Language Models (LLMs), defined as the generation of content inconsistent with facts or context, represent a core obstacle to their reliable deployment in critical domains. Current research primarily focuses on binary "detection" approaches that, while capable of identifying hallucinations, fail to provide interpretable and actionable feedback for model improvement, thus limiting practical utility. To address this limitation, a new research paradigm is proposed, shifting from "detection" to "diagnosis". The Hallucination Diagnosis Task is introduced, a task which requires models to not only detect hallucinations, but also perform error localization, causal explanation, and content correction. We develop the Hallucination Diagnosis Generator (HDG), an automated pipeline that systematically generates high-quality training samples with rich diagnostic metadata from raw corpora through multi-dimensional augmentation strategies including controlled fact fabrication and reasoning chain perturbation. Using HDG-generated data, we train HDM-4B-RL, a 4-billion-parameter hallucination diagnosis model, employing Group Relative Policy Optimization (GRPO) with a comprehensive reward function incorporating structural, accuracy, and localization signals. Experimental results demonstrate that our model surpasses previous state-of-the-art detection models on the HaluEval benchmark while achieving comparable performance to advanced general-purpose models. In comprehensive diagnosis tasks, HDM-4B-RL matches the capabilities of larger general models while maintaining a smaller size. This work validates the feasibility and value of hallucination diagnosis, providing an effective methodology for building more trustworthy and reliable generative AI systems.

EAAI Journal 2026 Journal Article

Leveraging community context and frequency-adaptive aggregation for robust fraud detection

• Zheng Zhang
• Jun Wan
• Jun Liu
• Mingyang Zhou
• Kezhong Lu
• Claudio J. Tessone
• Guoliang Chen
• Hao Liao

As the main threat to the healthy development of major internet platforms, fraud is increasingly carried out in organized, group-based forms. Such collusive fraud activities are easier to obtain illegal benefits at a lower exposure risk. Recently, graph neural network-based fraud detection methods have attracted increasing attention due to their ability to solve camouflage problems in fraud scenarios. However fraudsters’ evolving camouflage strategies pose great challenges to the design of graph neural network (GNN)-based detection models. Furthermore, most existing GNN-based approaches focus on the representation learning of node-level and structural-level features, and often ignores the contextual high-order information of the fraud group where the fraud node is located. To address these limitations, this paper proposes a community context-driven and frequency-adaptive graph neural network (CCFA-GNN) for detecting collaborative camouflage review fraudsters. Specifically, a collusive reviewer graph is constructed to capture the deep collaborative relationship among fraudsters. Then we incorporate the high-order representation of collusive fraud into graph embedding learning for community context based on the maximization of the co-occurrence probability of fraudsters. Finally, a frequency-adaptive feature aggregation module is adopted to simultaneously leverage the high-frequency and low-frequency information of features to enhance the node embedding representation. Extensive experiments on real-world fraud datasets have been conducted to verify the effectiveness, robustness, and interpretability of the proposed model, rendering it highly suitable for fraud detection applications in e-commerce and financial transaction scenarios.

AAAI Conference 2026 Conference Paper

MAPS: Multi-Agent Personality Shaping for Collaborative Reasoning

• Jian Zhang
• Zhiyuan Wang
• Zhangqi Wang
• Fangzhi Xu
• Qika Lin
• Lingling Zhang
• Rui Mao
• Erik Cambria

Collaborative reasoning with multiple agents offers the potential for more robust and diverse problem-solving. However, existing approaches often suffer from homogeneous agent behaviors and lack of reflective and rethinking capabilities. We propose Multi-Agent Personality Shaping ((MAPS), a novel framework that enhances reasoning through agent diversity and internal critique. Inspired by the Big Five personality theory, MAPS assigns distinct personality traits to individual agents, shaping their reasoning styles and promoting heterogeneous collaboration. To enable deeper and more adaptive reasoning, MAPS introduces a Critic agent that reflects on intermediate outputs, revisits flawed steps, and guides iterative refinement. This integration of personality-driven agent design and structured collaboration improves both reasoning depth and flexibility. Empirical evaluations across three benchmarks demonstrate the strong performance of MAPS, with further analysis confirming its generalizability across different large language models and validating the benefits of multi-agent collaboration.

AAAI Conference 2026 Conference Paper

MARS: Multi-Agent Adaptive Reasoning with Socratic Guidance for Automated Prompt Optimization

• Jian Zhang
• Zhangqi Wang
• Haiping Zhu
• Kangda Cheng
• Kai He
• Bo Li
• Qika Lin
• Jun Liu

Large language models (LLMs) typically operate in a question-answering paradigm, where the quality of the input prompt critically affects the response. Automated Prompt Optimization (APO) aims to overcome the cognitive biases of manually crafted prompts and explore a broader prompt design space. However, existing APO methods often suffer from rigid template structures and inefficient exploration in the prompt space. To this end, we propose a Multi-Agent Adaptive Reasoning with Socratic guidance framework (MARS) for APO. MARS consists of five complementary agents and formulates the optimization process as a Partially Observable Markov Decision Process (POMDP), enabling adaptive prompt refinement through explicit state modeling and interactive feedback. Specifically, a Planner agent generates flexible optimization trajectories, a Teacher-Critic-Student triad engages in Socratic-style dialogue to iteratively optimize the prompt based on pseudo-gradient signals in the text space, and a Target agent executes the prompt in downstream tasks to provide performance feedback. MARS integrates reasoning, feedback, and state transition into a unified hidden-state evolution process, improving both the effectiveness and interpretability of optimization. Extensive experiments on multiple datasets demonstrate that MARS outperforms existing APO methods in terms of optimization performance, search efficiency, and interpretability.

EAAI Journal 2026 Journal Article

Multi-objective mix optimization of geopolymer recycled aggregate concrete using prior knowledge embedded neural network and evolutionary algorithm

• Yajiang Guo
• Danying Gao
• Wei Ma
• Jun Liu

Geopolymer recycled aggregate concrete (GPRC) is a promising construction material that reduces carbon emissions and resource consumption. However, its application is limited by the lack of an effective mix design methodology. This study developed a prior Knowledge Embedded Neural Network (KENN) by incorporating governing equations into the neural network, resulting in a high-precision compressive strength prediction model that effectively captures the complex nonlinear interactions among GPRC mix components. The trained compressive strength prediction model was then embedded as an objective function within the Non-dominated Sorting Genetic Algorithm II (NSGA-II) for multi-objective mix optimization, from which a mix design method was constructed that maximizes 28-day compressive strength while simultaneously minimizing costs, CO2 emission, and energy consumption. The results demonstrate that the KENN-based prediction model achieved a high coefficient of determination of 0. 9126 on the testing set. Feature importance and Shapley Additive Explanations (SHAP) value analyses identified the recycled concrete powder replacement ratio, sand ratio, slag content, fine aggregate-to-precursor materials ratio, and recycled fine aggregate replacement ratio as the primary factors influencing the compressive strength of GPRC. Additionally, a well-defined Pareto front of the optimized GPRC mixtures was successfully constructed. Experimental validation of mixtures target strength grades of C30, C50, and C70 demonstrated that the compressive strength prediction errors were maintained within 5. 0 %, verifying the precision of the proposed mix design. This study offers an innovative and practical solution for the efficient and sustainable design of GPRC, with potential applicability to the optimization of other eco-friendly concrete mixtures.

EAAI Journal 2026 Journal Article

Multilayer inverse dynamic deduction algorithm of standard contradiction separation rule based on parallel mechanism

• Guoyan Zeng
• Guanfeng Wu
• Shuwei Chen
• Peiyao Liu
• Jun Liu
• Yang Xu
• Jian Zhong

The standard contradiction separation (S-CS) rule is a new inference rule recently proposed in the field of automated reasoning, which is characterized by dynamism, robustness, and collaborative deduction of multiple clauses. According to the above characteristics, to further utilize the inference ability of the S-CS rule, we propose an inverse and parallel algorithms to extend and enhance S-CS rule. Specifically, a multi-layer inverse and parallel deduction algorithm (in short MIP) is built. This algorithm transforms the first-order logic clause set into multiple clause sets, which are then recursively and iteratively deduced in parallel such that whenever a clause set is unsatisfiable, the original clause set is unsatisfiable. The main advantages of this algorithm are inverse deduction, parallel deduction, and depth (multi-layer) deduction. In order to improve the performance of automated theorem prover, we embed this algorithm into the current top automated theorem provers Vampire and E to form the new provers MIP_V and MIP_E. Then we test MIP_V with the problems from the international competition (CASC) for automated theorem provers, and test MIP_E and MIP_V with the hardest problem of rating = 1 from the benchmark library TPTP. The experimental results show that MIP_V (MIP_E) has a better performance than Vampire (E), and MIP_V and MIP_E can solve 66 problems with rating = 1.

FLAP Journal 2026 Journal Article

On the Structure of Dual-line Standard Contradictions and their General Forms in First-order Logic

• Xingxing He
• Jia Xu
• Yingfang Li
• Jun Liu

Contradiction separation (CS) and its first-order version S-CS are multi- clause inference schemes for clausal refutation. They isolate a standard con- tradiction core within a clause set and derive a propagated clause from the remaining literals. This paper develops structural characterizations of non-unit standard contradictions that make such cores explicit and easier to identify. In propositional logic, we introduce a canonical dual-line family and prove that every instance is a standard contradiction. We study admissible literal exten- sions, define ladder structures as maximal dual-line extensions, and present a regular triple-line family with constructive generation schemes. We also analyze how dual-line cores compose via clause connections and give sufficient condi- tions under which the composed clause set remains a standard contradiction. In first-order logic, we exhibit clause families that are not standard contradictions syntactically but become standard contradictions after suitable instantiation and controlled clause reuse. ∗ The corresponding author.

I&C Journal 2026 Journal Article

Spiking Neural P Systems Incorporating Winner-take-all Mechanism

• Tingting Bao
• Bifan Wei
• Bo Li
• Lingling Zhang
• Hong Peng
• Xiaoqing Zhang
• Jun Liu

Spiking neural P systems (SNP systems), a class of parallel distributed computational models inspired by biological neurons, have become an important research direction in biocomputing in recent years due to their biological interpretability and low-power computing. There are many studies on the expressive power of SNP system variants, but their computational efficiency is not high in terms of resource overhead. Inspired by the biological winner-take-all (WTA) computation mechanism, this study proposes spiking neural P systems incorporating WTA mechanism (WTASNP systems). Competing neuron nodes are introduced into the WTASNP systems to realize the competitive selection and inhibition control mechanisms of the WTA computation. After competition, only the winner neuron is allowed to emit spikes, while the loser neurons are inhibited. It is proved that the WTASNP systems have Turing universality. Furthermore, it reduces the computational resource requirements for solving the NP-complete SAT problem with SNP systems from O(n 2) or O(2 n ) complexity levels down to linearly solvable O(n). The WTASNP systems not only effectively preserve the preamble spike information, but also inhibits the loser neuron spike issuance by the WTA computational mechanism of competing neurons, reduces redundant computation to avoid neuron over-excitation, and improves the computational efficiency and expressive power.

AAMAS Conference 2026 Conference Paper

Structured Agent Distillation for Large Language Model Agents

• Jun Liu
• Zhenglun Kong
• Peiyan Dong
• Changdi Yang
• Tianqin Li
• Yanyue Xie
• Yifan Gong
• Xuan Shen

Largelanguagemodels(LLMs)exhibitstrongcapabilitiesasdecisionmaking agents by interleaving reasoning and actions, as seen in ReAct-style frameworks. Yet, their practical deployment is constrained by high inference costs and large model sizes. We propose Structured Agent Distillation, the first framework to distill a ReAct-based LLM agent into a smaller model while preserving both reasoning fidelity and action consistency. Our method introduces a structured, span-level distillation strategy that explicitly segments trajectories into reasoning and action spans, enabling fine-grained alignment beyond standard token-level imitation. Unlike other advanced distillation methods, Our method segments trajectories into [REASON] and [ACT] spans, applying segmentspecific losses to align each component with the teacher’s behavior. This structure-aware supervision enables compact agents to better Corresponding Authors: Pu Zhao, Hao Tang. This work is licensed under a Creative Commons Attribution International 4. 0 License. Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026), C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.), May 25 – 29, 2026, Paphos, Cyprus. © 2026 International Foundation for Autonomous Agents and Multiagent Systems (www. ifaamas. org). https: //doi. org/10. 65109/OLHJ8062 replicate the teacher’s decision process. Experiments on ALFWorld, HotPotQA-ReAct, and WebShop show that our approach consistently outperforms token-level and imitation learning baselines, achieving significant compression with minimal performance drop. Scaling and ablation results further highlight the importance of span-level alignment for efficient and deployable agents. We will release code upon acceptance.

EAAI Journal 2025 Journal Article

A continuous verification mechanism for ensuring client data forgetfulness in Federated Unlearning

• Fudu Xing
• Jun Liu
• Shanshan Chen
• Tianlong Yu
• Yang Yang

In Federated Learning (FL), it is sometimes necessary to unlearn client data through Federated Unlearning (FU) methods, which help protect user privacy and recover from data poisoning. One critical task is to check the consistency of FU methods and verify if certain clients’ data has been effectively unlearned. However, none of the current FU verification methods can be performed on clients who opt out of the FL process, failing to meet the universal demand in FL, such as the user’s “right to be forgotten” (RTBF). Specifically, after clients leave the FL cooperation, they can no longer verify whether the FL model unlearns their data as the FL continues training for several rounds. To address this, we introduce a continuous verification mechanism for FL clients called Backdoor Attack-based Forgetting Verification (BAFV). The BAFV method embeds a persistent mark for clients who propose to leave, allowing them to verify FU long after leaving the FL cooperation. Extensive experiments across diverse FU environments and datasets demonstrate that our method maintains the model’s accuracy and provides clients with a continuous verification mechanism to ensure their data is unlearned. Our combinatorial marking strategy and gradient-amplified persistence mechanism represent significant advancements beyond existing verification schemes. Our code of BAFV is publicly available at: https: //github. com/FuduXing/newBAFV. git.

JBHI Journal 2025 Journal Article

Belief-Rule-Based System With Self-Organizing and Multi-Temporal Modeling for Sensor-Based Human Activity Recognition

• Long-Hao Yang
• Fei-Fei Ye
• Chris Nugent
• Jun Liu
• Ying-Ming Wang

Smart environment is an efficient and cost-effective way to afford intelligent supports for the elderly people. Human activity recognition is a crucial aspect of the research field of smart environments, and it has attracted widespread attention lately. The goal of this study is to develop an effective sensor-based human activity recognition model based on the belief-rule-based system (BRBS), which is one of representative rule-based expert systems. Specially, a new belief rule base (BRB) modeling approach is proposed by taking into account the self- organizing rule generation method and the multi-temporal rule representation scheme, in order to address the problem of combination explosion that existed in the traditional BRB modelling procedure and the time correlation found in continuous sensor data in chronological order. The new BRB modeling approach is so called self-organizing and multi-temporal BRB (SOMT-BRB) modeling procedure. A case study is further deducted to validate the effectiveness of the SOMT-BRB modeling procedure. By comparing with some conventional BRBSs and classical activity recognition models, the results show a significant improvement of the BRBS in terms of the number of belief rules, modelling efficiency, and activity recognition accuracy.

NeurIPS Conference 2025 Conference Paper

Boosting Skeleton-based Zero-Shot Action Recognition with Training-Free Test-Time Adaptation

• Jingmin Zhu
• Anqi Zhu
• Hossein Rahmani
• Jun Liu
• Mohammed Bennamoun
• Qiuhong Ke

We introduce Skeleton-Cache, the first training-free test-time adaptation framework for skeleton-based zero-shot action recognition (SZAR), aimed at improving model generalization to unseen actions during inference. Skeleton-Cache reformulates inference as a lightweight retrieval process over a non-parametric cache that stores structured skeleton representations, combining both global and fine-grained local descriptors. To guide the fusion of descriptor-wise predictions, we leverage the semantic reasoning capabilities of large language models (LLMs) to assign class-specific importance weights. By integrating these structured descriptors with LLM-guided semantic priors, Skeleton-Cache dynamically adapts to unseen actions without any additional training or access to training data. Extensive experiments on NTU RGB+D 60/120 and PKU-MMD II demonstrate that Skeleton-Cache consistently boosts the performance of various SZAR backbones under both zero-shot and generalized zero-shot settings. The code is publicly available at https: //github. com/Alchemist0754/Skeleton-Cache.

NeurIPS Conference 2025 Conference Paper

Causal-R: A Causal-Reasoning Geometry Problem Solver for Optimized Solution Exploration

• Wenjun Wu
• Lingling Zhang
• Bo Zhao
• Muye Huang
• Qianying Wang
• Jun Liu

The task of geometry problem solving has been a long-standing focus in the automated mathematics community and draws growing attention due to its complexity for both symbolic and neural models. Although prior studies have explored various effective approaches for enhancing problem solving performances, two fundamental challenges remain unaddressed, which are essential to the application in practical scenarios. First, the multi-step reasoning gap between the initial geometric conditions and ultimate problem goal leads to a great search space for solution exploration. Second, obtaining multiple interpretable and shorter solutions remains an open problem. In this work, we introduce the Causal-Reasoning Geometry Problem Solver to overcome these challenges. Specifically, the Causal Graph Reasoning theory is proposed to perform symbolic reasoning before problem solving. Several causal graphs are constructed according to predefined rule base, where each graph is composed of primitive nodes, causal edges and prerequisite edges. By applying causal graph deduction from initial conditions, the reachability status of nodes are iteratively conveyed by causal edges until reaching the target nodes, representing feasible causal deduction paths. In this way, the search space of solutions is compressed from the beginning, the end and intermediate reasoning paths, while ensuring the interpretability and variety of solutions. To achieve this, we further propose Forward Matrix Deduction which transforms the causal graphs into matrices and vectors, and applies matrix operations to update the status value of reachable nodes in iterations. Finally, multiple solutions can be generated by tracing back from the target nodes after validation. Experiments demonstrate the effectiveness of our method to obtain multiple shorter and interpretable solutions. Code is available after acceptance.

NeurIPS Conference 2025 Conference Paper

ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding

• Muye Huang
• Lingling Zhang
• Jie Ma
• Han Lai
• Fangzhi Xu
• Yifei Li
• Wenjun Wu
• Yaqiang Wu

Charts are high-density visualization carriers for complex data, serving as a crucial medium for information extraction and analysis. Automated chart understanding poses significant challenges to existing multimodal large language models (MLLMs) due to the need for precise and complex visual reasoning. Current step-by-step reasoning models primarily focus on text-based logical reasoning for chart understanding. However, they struggle to refine or correct their reasoning when errors stem from flawed visual understanding, as they lack the ability to leverage multimodal interaction for deeper comprehension. Inspired by human cognitive behavior, we propose ChartSketcher, a multimodal feedback-driven step-by-step reasoning method designed to address these limitations. ChartSketcher is a chart understanding model that employs Sketch-CoT, enabling MLLMs to annotate intermediate reasoning steps directly onto charts using a programmatic sketching library, iteratively feeding these visual annotations back into the reasoning process. This mechanism enables the model to visually ground its reasoning and refine its understanding over multiple steps. We employ a two-stage training strategy: a cold start phase to learn sketch-based reasoning patterns, followed by off-policy reinforcement learning to enhance reflection and generalization. Experiments demonstrate that ChartSketcher achieves promising performance on chart understanding benchmarks and general vision tasks, providing an interactive and interpretable approach to chart comprehension.

NeurIPS Conference 2025 Conference Paper

CoFFT: Chain of Foresight-Focus Thought for Visual Language Models

• Xinyu Zhang
• Yuxuan Dong
• Lingling Zhang
• Chengyou Jia
• Zhuohang Dang
• Basura Fernando
• Jun Liu
• Mike Zheng Shou

Despite significant advances in Vision Language Models (VLMs), they remain constrained by the complexity and redundancy of visual input. When images contain large amounts of irrelevant information, VLMs are susceptible to interference, thus generating excessive task-irrelevant reasoning processes or even hallucinations. This limitation stems from their inability to discover and process the required regions during reasoning precisely. To address this limitation, we present the Chain of Foresight-Focus Thought (CoFFT), a novel training-free approach that enhances VLMs' visual reasoning by emulating human visual cognition. Each Foresight-Focus Thought consists of three stages: (1) Diverse Sample Generation: generates diverse reasoning samples to explore potential reasoning paths, where each sample contains several reasoning steps; (2) Dual Foresight Decoding: rigorously evaluates these samples based on both visual focus and reasoning progression, adding the first step of optimal sample to the reasoning process; (3) Visual Focus Adjustment: precisely adjust visual focus toward regions most beneficial for future reasoning, before returning to stage (1) to generate subsequent reasoning samples until reaching the final answer. These stages function iteratively, creating an interdependent cycle where reasoning guides visual focus and visual focus informs subsequent reasoning. Empirical results across multiple benchmarks using Qwen2. 5-VL, InternVL-2. 5, and Llava-Next demonstrate consistent performance improvements of 3. 1-5. 8\% with controllable increasing computational overhead.

IJCAI Conference 2025 Conference Paper

Cross-modal Collaborative Representation Learning for Text-to-Image Person Retrieval

• Shuanglin Yan
• Jun Liu
• Neng Dong
• Liyan Zhang
• Jinhui Tang

Text-to-image person retrieval (TIPR) aims to find images of the same identity that match a given text description. Current TIPR methods mainly focus on mining the association between images and texts, ignoring their potential complementarity. Besides, existing matching losses treat all positive pairs from the same identity equally, leading to noisy correspondences. In this paper, we propose CoRL: a cross-modal Collaborative Representation Learning framework designed to improve TIPR by effectively leveraging the complementarity between modalities. The text typically contains identity details with less noise, which helps distinguish visually similar pedestrians. This inspires us to integrate it into the corresponding image to emphasize identity-related and modality-shared visual information. However, corresponding text for each image is not always available, especially during inference. Accordingly, we introduce a Virtual-text Embedding Synthesizer that generates high-quality virtual-text features for cross-modal collaboration, eliminating the need for actual texts. We then design a Cross-Modal Collaboration learning process, incorporating a Cross-modal Relation Consistency loss to promote interaction and fusion between image and virtual-text features for mutual enhancement. Additionally, an Identity-bounded Matching loss is proposed to handle different types of image-text pairs distinctly, leading to more accurate cross-modal correspondences. Extensive experiments on multiple benchmarks demonstrate the superiority of CoRL over existing TIPR methods.

EAAI Journal 2025 Journal Article

Data-driven joint multiobjective prediction and optimization for tunnel-induced adjacent bridge pier displacement: A case study in China

• Hongyu Chen
• Jun Liu
• Qiping Geoffrey Shen
• Tiejun Li
• Yang Liu

To reduce the impact of tunnel construction on adjacent bridge pile foundations and ensure safety during construction, a hybrid intelligent framework combining Bayesian optimization (BO), categorical boosting (CatBoost), and the nondominated sorting genetic algorithm-III (NSGA-III) is proposed in this paper. The nonlinear mapping function relationship between the nine input parameters and the bridge pier vertical and horizontal displacements is established via BO-CatBoost. The key optimization parameters are for interpretability analysis and determined via Shapley additive explanations (SHAP) method. NSGA-III is established with the goal of minimizing pier displacement. The applicability and validity of the proposed method are tested in a case study of the Wuhan Metro. The key findings of this study include the following. (1) The accuracy of the prediction model obtained by the BO-CatBoost algorithm on the basis of the training and simulation of the measured engineering data is high. On the bridge pier horizontal and vertical displacement test sets, the R2 values are 0. 823 and 0. 826, the RMSE values are 0. 452 and 0. 539, and the MAEs are 0. 293 and 0. 360, respectively. (2) The optimization effect of the two objectives is significant, and the average percentage of improvement stands at 35. 54%. When five shield construction parameters are adjusted simultaneously, the optimization effect of the two objectives is the best, and the average improvement percentage is 54. 76%. (3) The optimization effect of the developed BO-CatBoost-NSGA-III intelligent algorithm is greater than that of single-objective optimization. Therefore, the intelligent optimization framework proposed in this paper can provide guidance for the optimal control of pier displacement in shield underpass construction engineering.

AAAI Conference 2025 Conference Paper

Debate on Graph: A Flexible and Reliable Reasoning Framework for Large Language Models

• Jie Ma
• Zhitao Gao
• Qi Chai
• Wangchun Sun
• Pinghui Wang
• Hongbin Pei
• Jing Tao
• Lingyun Song

Large Language Models (LLMs) may suffer from hallucinations in real-world applications due to the lack of relevant knowledge. In contrast, knowledge graphs encompass extensive, multi-relational structures that store a vast array of symbolic facts. Consequently, integrating LLMs with knowledge graphs has been extensively explored, with Knowledge Graph Question Answering (KGQA) serving as a critical touchstone for the integration. This task requires LLMs to answer natural language questions by retrieving relevant triples from knowledge graphs. However, existing methods face two significant challenges: *excessively long reasoning paths distracting from the answer generation*, and *false-positive relations hindering the path refinement*. In this paper, we propose an iterative interactive KGQA framework that leverages the interactive learning capabilities of LLMs to perform reasoning and Debating over Graphs (DoG). Specifically, DoG employs a subgraph-focusing mechanism, allowing LLMs to perform answer trying after each reasoning step, thereby mitigating the impact of lengthy reasoning paths. On the other hand, DoG utilizes a multi-role debate team to gradually simplify complex questions, reducing the influence of false-positive relations. This debate mechanism ensures the reliability of the reasoning process. Experimental results on five public datasets demonstrate the effectiveness and superiority of our architecture. Notably, DoG outperforms the state-of-the-art method ToG by 23.7% and 9.1% in accuracy on WebQuestions and GrailQA, respectively. Furthermore, the integration experiments with various LLMs on the mentioned datasets highlight the flexibility of DoG.

AAAI Conference 2025 Conference Paper

Deconfound Semantic Shift and Incompleteness in Incremental Few-shot Semantic Segmentation

• Yirui Wu
• Yuhang Xia
• Hao Li
• Lixin Yuan
• Junyang Chen
• Jun Liu
• Tong Lu
• Shaohua Wan

Incremental few-shot semantic segmentation (IFSS) expands segmentation capacity of the trained model to segment new-class images with few samples. However, semantic meanings may shift from background to object class or vice versa during incremental learning. Moreover, new-class samples often lack representative attribute features when the new class greatly differs from the pre-learned old class. In this paper, we propose a causal framework to discuss the cause of semantic shift and incompleteness in IFSS, and we deconfound the revealed causal effects from two aspects. First, we propose a Causal Intervention Module (CIM) to resist semantic shift. CIM progressively and adaptively updates prototypes of old class, and removes the confounder in an intervention manner. Second, a Prototype Refinement Module (PRM) is proposed to complete the missing semantics. In PRM, knowledge gained from the episode learning scheme assists in fusing features of new-class and old-class prototypes. Experiments on both PASCAL-VOC 2012 and ADE20k benchmarks demonstrate the outstanding performance of our method.

NeurIPS Conference 2025 Conference Paper

Deliberation on Priors: Trustworthy Reasoning of Large Language Models on Knowledge Graphs

• Jie Ma
• NING QU
• Zhitao Gao
• Xing Rui
• Jun Liu
• Hongbin Pei
• Jiang Xie
• Lingyun Song

Knowledge graph-based retrieval-augmented generation seeks to mitigate hallucinations in Large Language Models (LLMs) caused by insufficient or outdated knowledge. However, existing methods often fail to fully exploit the prior knowledge embedded in knowledge graphs (KGs), particularly their structural information and explicit or implicit constraints. The former can enhance the faithfulness of LLMs' reasoning, while the latter can improve the reliability of response generations. Motivated by these, we propose a trustworthy reasoning framework, termed Deliberation over Priors (\texttt{DP}), which sufficiently utilizes the priors contained in KGs. Specifically, \texttt{DP} adopts a progressive knowledge distillation strategy that integrates structural priors into LLMs through a combination of supervised fine-tuning and Kahneman-Tversky Optimization, thereby improving the faithfulness of relation path generation. Furthermore, our framework employs a reasoning-introspection strategy, which guides LLMs to perform refined reasoning verification based on extracted constraint priors, ensuring the reliability of response generation. Extensive experiments on three benchmark datasets demonstrate that \texttt{DP} achieves new state-of-the-art performance, especially a H@1 improvement of 13% on the ComplexWebQuestions dataset, and generates highly trustworthy responses. We also conduct various analyses to verify its flexibility and practicality. Code is available at https: //github. com/mira-ai-lab/Deliberation-on-Priors.

IJCAI Conference 2025 Conference Paper

Diffuse&amp; Refine: Intrinsic Knowledge Generation and Aggregation for Incremental Object Detection

• Jianzhou Wang
• Yirui Wu
• Lixin Yuan
• Wenxiao Zhang
• Jun Liu
• Junyang Chen
• Huan Wang
• Wenhai Wang

Incremental Object Detection(IOD) targets at progressively extending capability of object detectors to recognize new classes. However, representation confusion between old and new classes leads to catastrophic forgetting. To alleviate this problem, we propose DiffKA, with intrinsic knowledge generated and aggregated by forward and backward diffusion, gradually establishing rigid class boundary. With incremental streaming data, forward diffusion spreads information to generate potential inter-class associations among new- and old-class prototypes within a hierarchical tree, named as Intrinsic Correlation Tree(ICTree), to store intrinsic knowledge. Afterwards, backward diffusion refines and aggregates the generated knowledge in ICTree, explicitly establishing rigid class boundary to mitigate representation confusion. To keep semantic consistency with extreme IOD settings, we reorganize semantic relevance of old- and new-class prototypes in paradigms to adaptively and effectively update DiffKA. Experiments on MS COCO dataset show DiffKA achieves state-of-the-art performance on IOD tasks with significant advantages.

TIST Journal 2025 Journal Article

Edge Manipulations for the Maximum Vertex-Weighted Bipartite b -matching

• Gennaro Auricchio
• Jun Liu
• Qun Ma
• Jie Zhang

In this article, we explore the Mechanism Design aspects of the Maximum Vertex-Weighted \(b\) -matching (MVbM) problem on bipartite graphs \((A\cup T,E)\). The set \(A\) comprises agents, while \(T\) represents tasks. The set \(E\), which connects \(A\) and \(T\), is the private information of either agents or tasks. In this framework, we investigate three mechanisms— \(\mathbb{M}_{BFS}\), \(\mathbb{M}_{DFS}\), and \(\mathbb{M}_{G}\). We examine scenarios in which either agents or tasks are strategic and report their adjacent edges to one of the three mechanisms. In both cases, we assume that the strategic entities are bounded by their statements: They can hide edges, but they cannot report edges that do not exist. First, we consider the case in which agents can manipulate. In this framework, \(\mathbb{M}_{BFS}\) and \(\mathbb{M}_{DFS}\) are optimal but not truthful. By characterizing the Nash Equilibria induced by \(\mathbb{M}_{BFS}\) and \(\mathbb{M}_{DFS}\), we reveal that both mechanisms have a Price of Anarchy ( \(PoA\) ) and Price of Stability ( \(PoS\) ) of \(2\). These efficiency guarantees are tight; no deterministic mechanism can achieve a lower \(PoA\) or \(PoS\). In contrast, the third mechanism, \(\mathbb{M}_{G}\), is not optimal, but truthful and its approximation ratio is \(2\). We demonstrate that this ratio is optimal; no deterministic and truthful mechanism can outperform it. We then shift our focus to scenarios where tasks can exhibit strategic behavior. In this case, \(\mathbb{M}_{BFS}\), \(\mathbb{M}_{DFS}\), and \(\mathbb{M}_{G}\) all maintain truthfulness, making \(\mathbb{M}_{BFS}\) and \(\mathbb{M}_{DFS}\) truthful and optimal mechanisms. In conclusion, we investigate the manipulability of \(\mathbb{M}_{BFS}\) and \(\mathbb{M}_{DFS}\) through experiments on randomly generated graphs. We observe that (i) \(\mathbb{M}_{BFS}\) is less prone to be manipulated by the first agent than \(\mathbb{M}_{DFS}\), and (ii) \(\mathbb{M}_{BFS}\) is more manipulable on instances in which the total capacity of the agents is equal to the number of tasks. 1

AAAI Conference 2025 Conference Paper

EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding

• Muye Huang
• Han Lai
• Xinyu Zhang
• Wenjun Wu
• Jie Ma
• Lingling Zhang
• Jun Liu

Chart understanding enables automated data analysis for humans, which requires models to achieve highly accurate visual comprehension. While existing Visual Language Models (VLMs) have shown progress in chart understanding, the lack of high-quality training data and comprehensive evaluation benchmarks hinders VLM chart comprehension. In this paper, we introduce EvoChart, a novel self-training method for generating synthetic chart data to enhance VLMs' capabilities in real-world chart comprehension. We also propose EvoChart-QA, a noval benchmark for measuring models' chart comprehension abilities in real-world scenarios. Specifically, EvoChart is a unique self-training data synthesis approach that simultaneously produces high-quality training corpus and a high-performance chart understanding model. EvoChart-QA consists of 650 distinct real-world charts collected from 140 different websites and 1,250 expert-curated questions that focus on chart understanding. Experimental results on various open-source and proprietary VLMs tested on EvoChart-QA demonstrate that even the best proprietary model, GPT-4o, achieves only 49.8% accuracy. Moreover, the EvoChart method significantly boosts the performance of open-source VLMs on real-world chart understanding tasks, achieving 54.2% accuracy on EvoChart-QA.

IJCAI Conference 2025 Conference Paper

fairGNN-WOD: Fair Graph Learning Without Complete Demographics

• Zichong Wang
• Fang Liu
• Shimei Pan
• Jun Liu
• Fahad Saeed
• Meikang Qiu
• Wenbin Zhang

Graph Neural Networks (GNNs) have excelled in diverse applications due to their outstanding predictive performance, yet they often overlook fairness considerations, prompting numerous recent efforts to address this societal concern. However, most fair GNNs assume complete demographics by design, which is impractical in most real-world socially sensitive applications due to privacy, legal, or regulatory restrictions. For example, the Consumer Financial Protection Bureau (CFPB) mandates that creditors ensure fairness without requesting or collecting information about an applicant’s race, religion, nationality, sex, or other demographics. To this end, this paper proposes fairGNN-WOD, a first-of-its-kind framework that considers mitigating unfairness in graph learning without using demographic information. In addition, this paper provides a theoretical perspective on analyzing bias in node representations and establishes the relationship between utility and fairness objectives. Experiments on three real-world graph datasets illustrate that fairGNN-WOD outperforms state-of-the-art baselines in achieving fairness but also maintains comparable prediction performance.

IJCAI Conference 2025 Conference Paper

FairSMOE: Mitigating Multi-Attribute Fairness Problem with Sparse Mixture-of-Experts

• Changdi Yang
• Zheng Zhan
• Ci Zhang
• Yifan Gong
• Yize Li
• Zichong Meng
• Jun Liu
• Xuan Shen

Real‐world datasets usually contain multiple attributes, making it essential to ensure fairness across all of them simultaneously. However, different attributes may vary in difficulty, and no existing approaches have effectively addressed this issue. Consequently, an attribute‐adaptive strategy is needed to achieve fairness for all attributes. Multi‐task Learning (MTL) leverages shared information to optimize multiple tasks concurrently, while Sparsely‐Gated Mixture‐of‐Experts (SMoE) can dynamically allocate computational resources to the most needed tasks. In this work, we formulate multi‐attribute fairness issue as an MTL problem and employ SMoE to achieve desirable performance across all attributes simultaneously. We first analyze the feasibility and find the potentiality by formalizing multi-attribute fairness problem into a MTL problem and mitigating it by using SMoE. However, vanilla SMoE could lead to over-utilization problem which causes sub-optimal performance. We then proposed an innovative SMoE framework for multi-attribute fair image classification, which further improves multi-attribute fairness by redesigning the MoE layer and routing policy with fairness consideration. Extensive experiments demonstrated the effectiveness. Taking a DeiT-Small as the backbone, we achieve 77. 25% and 86. 01% accuracy on the ISIC2019 and CelebA dataset respectively with Multi-attribute Predictive Quality Disparity (PQD) score of 0. 801 and 0. 787, beating current state-of-the-art methods Muffin, InfoFair and MultiFair.

EAAI Journal 2025 Journal Article

Fast inversion of parameters on Jiles–Atherton hysteresis model based on physics-guided deep learning network

• Saibo She
• Xiaochu Pang
• Jun Liu
• Xinnan Zheng
• Kuohai Yu
• Xun Zou
• Ruoxuan Zhu
• Rui Guo

The Jiles–Atherton (J–A) hysteresis model is widely used to illustrate the properties of ferromagnetic materials. However, in the inverse J–A hysteresis model, careful selection of the initial seed value is crucial for accurately obtaining the model parameters. This selection impacts both the calculation time and the model’s adaptability. To address this problem, the physics-guided Kolmogorov–Arnold Networks (KAN)-EfficientNet deep learning (DL) model is proposed to achieve fast and accurate J–A model parameters estimation. In the innovative approach, the physics constraints from the inversed J–A hysteresis model and the KAN-Linear layer are utilized to accelerate the convergence of the deep learning model and enhance its accuracy. The theoretical framework of the J–A hysteresis model and its inversed model is illustrated, alongside the development of an experimental platform for data collection to verify the proposed method’s feasibility. Data from classical J–A hysteresis numerical model and simulations are utilized to train the weights of proposed DL model. The compared algorithms are also introduced to illustrate the performance of the proposed model. The experimental data are fed into the analytical J–A hysteresis model and the pre-trained physics-guided KAN-EfficientNet model, respectively. The results show that the proposed method provides high accuracy (error less than 2. 41%) in less calculation time (0. 3 s) to determine the J–A hysteresis model parameters. The work introduces an innovative physics-guided DL model that significantly enhances the efficiency of J–A hysteresis model parameters estimation, thereby promoting the broader application of J–A hysteresis model in study of ferromagnetic materials.

EAAI Journal 2025 Journal Article

Formal verification for multi-agent path execution in stochastic environments

• Xia Wang
• Jun Liu
• Chris D. Nugent
• Shaobing Xu
• Yang Xu

Multi-agent pathfinding aims to determine conflict-free paths for multiple agents in a shared environment. However, real-world uncertainties can disrupt preplanned paths, leading to delays and new conflicts. Addressing these challenges requires robust strategies for path execution and adjustment. While many multi-agent pathfinding algorithms have been proposed, this work does not introduce a new algorithm. Instead, it presents an adjustment solution based on a set of constraint rules and a priority strategy to avoid conflicts and deadlocks. Additionally, a Markov decision process model is developed, derived from the preplanned paths, and integrated with the adjustment solution to account for stochastic environmental uncertainties. A novel integrated framework is proposed for formally analyze and verify the reliability of multi-agent path execution and the robustness of the adjustment solution in stochastic environments, with formal verification achieved through a logic-based probabilistic model checker. The performance of the proposed framework is validated through various scenarios on the Flatland platform. Results demonstrate that the adjustment solution, based on the constraint rules, effectively mitigates conflicts and deadlocks, improving robustness. Furthermore, formal verification proves effective in assessing the reliability and robustness of multi-agent path execution under uncertainty.

AAAI Conference 2025 Conference Paper

Global-Semantic Alignment Distillation for Partial Multi-view Classification

• Xiaoli Wang
• Anqi Huang
• Yongli Wang
• Guanzhou Ke
• Xiaobin Hong
• Jun Liu

Partial multi-view classification (PMvC) poses a significant challenge due to the incomplete nature of multi-view data, which complicates effective information fusion and accurate classification. Existing PMvC methods typically rely on heuristic evaluations of view informativeness to achieve global alignment for downstream classification tasks. However, these approaches suffer from two critical issues: information redundancy and semantic misalignment. The complexity of missing data not only leads to over-reliance on redundant or less informative views but also exacerbates semantic misalignment across views, making it difficult for existing methods to effectively capture and discriminate the class-related features. To address these issues, this work proposes a novel GLobal-semantic Alignment Distillation (GLAD) model for partial multi-view classification without requiring imputation. Our approach incorporates a self-distillation mechanism that enables the model to extract informative features and achieve global semantic alignment across views. The key insight of GLAD is leveraging labels as semantic anchors to guide the alignment of partial multi-view features. By integrating labels with extracted features via a cross-attention mechanism, we generate ideal embeddings that consistently capture global semantics across views. These embeddings then serve as intermediate supervision for distilling the student model, ensuring robust semantic alignment even with missing views. We further introduce a margin-aware weighting strategy to enhance the model's discriminative ability. Extensive experimental results validate the effectiveness and superiority of the proposed method, showcasing significant improvements in classification performance over existing techniques.

NeurIPS Conference 2025 Conference Paper

Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning

• Qitao Tan
• Jun Liu
• Zheng Zhan
• Caiwen Ding
• Yanzhi Wang
• Xiaolong Ma
• Jaewoo Lee
• Jin Lu

Large language models (LLMs) excel across various tasks, but standard first-order (FO) fine-tuning demands considerable memory, significantly limiting real-world deployment. Recently, zeroth-order (ZO) optimization stood out as a promising memory-efficient training paradigm, avoiding backward passes and relying solely on forward passes for gradient estimation, making it attractive for resource-constrained scenarios. However, ZO method lags far behind FO method in both convergence speed and accuracy. To bridge the gap, we introduce a novel layer-wise divergence analysis that uncovers the distinct update pattern of FO and ZO optimization. Aiming to resemble the learning capacity of FO method from the findings, we propose \textbf{Di}vergence-driven \textbf{Z}eroth-\textbf{O}rder (\textbf{DiZO}) optimization. DiZO conducts divergence-driven layer adaptation by incorporating projections to ZO updates, generating diverse-magnitude updates precisely scaled to layer-wise individual optimization needs. Our results demonstrate that DiZO significantly reduces the needed iterations for convergence without sacrificing throughput, cutting training GPU hours by up to 48\% on various datasets. Moreover, DiZO consistently outperforms the representative ZO baselines in fine-tuning RoBERTa-large, OPT-series, and Llama-series on downstream tasks and, in some cases, even surpasses memory-intensive FO fine-tuning. Our code is released at \url{https: //github. com/Skilteee/DiZO}.

AAAI Conference 2025 Conference Paper

Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension

• Yaxian Wang
• Henghui Ding
• Shuting He
• Xudong Jiang
• Bifan Wei
• Jun Liu

In this work, we address the challenging task of Generalized Referring Expression Comprehension (GREC). Compared to the classic Referring Expression Comprehension (REC) that focuses on single-target expressions, GREC extends the scope to a more practical setting by further encompassing no-target and multi-target expressions. Existing REC methods face challenges in handling the complex cases encountered in GREC, primarily due to their fixed output and limitations in multi-modal representations. To address these issues, we propose a Hierarchical Alignment-enhanced Adaptive Grounding Network (HieA2G) for GREC, which can flexibly deal with various types of referring expressions. First, a Hierarchical Multi-modal Semantic Alignment (HMSA) module is proposed to incorporate three levels of alignments, including word-object, phrase-object, and text-image alignment. It enables hierarchical cross-modal interactions across multiple levels to achieve comprehensive and robust multi-modal understanding, greatly enhancing grounding ability for complex cases. Then, to address the varying number of target objects in GREC, we introduce an Adaptive Grounding Counter (AGC) to dynamically determine the number of output targets. Additionally, an auxiliary contrastive loss is employed in AGC to enhance object-counting ability by pulling in multi-modal features with the same counting and pushing away those with different counting. Extensive experimental results show that HieA2G achieves new state-of-the-art performance on the challenging GREC task and also the other 4 tasks, including REC, Phrase Grounding, Referring Expression Segmentation (RES), and Generalized Referring Expression Segmentation (GRES), demonstrating the remarkable superiority and generalizability of the proposed HieA2G.

NeurIPS Conference 2025 Conference Paper

Hierarchical Optimization via LLM-Guided Objective Evolution for Mobility-on-Demand Systems

• Yi Zhang
• Yushen Long
• Yun Ni
• Liping Huang
• Xiaohong Wang
• Jun Liu

Online ride-hailing platforms aim to deliver efficient mobility-on-demand services, often facing challenges in balancing dynamic and spatially heterogeneous supply and demand. Existing methods typically fall into two categories: reinforcement learning (RL) approaches, which suffer from data inefficiency, oversimplified modeling of real-world dynamics, and difficulty enforcing operational constraints; or decomposed online optimization methods, which rely on manually designed high-level objectives that lack awareness of low-level routing dynamics. To address this issue, we propose a novel hybrid framework that integrates large language model (LLM) with mathematical optimization in a dynamic hierarchical system: (1) it is training-free, removing the need for large-scale interaction data as in RL, and (2) it leverages LLM to bridge cognitive limitations caused by problem decomposition by adaptively generating high-level objectives. Within this framework, LLM serves as a meta-optimizer, producing semantic heuristics that guide a low-level optimizer responsible for constraint enforcement and real-time decision execution. These heuristics are refined through a closed-loop evolutionary process, driven by harmony search, which iteratively adapts the LLM prompts based on feasibility and performance feedback from the optimization layer. Extensive experiments based on scenarios derived from both the New York and Chicago taxi datasets demonstrate the effectiveness of our approach, achieving an average improvement of 16% compared to state-of-the-art baselines.

NeurIPS Conference 2025 Conference Paper

JADE: Joint Alignment and Deep Embedding for Multi-Slice Spatial Transcriptomics

• Yuanchuan Guo
• Jun Liu
• Huimin Cheng
• Ying Ma

As spatial transcriptomics (ST) datasets increasingly span multiple adjacent or replicated slices, effective joint analysis across slices is needed to reconstruct tissue structures and identify consistent spatial gene expression patterns. This requires resolving spatial correspondences between slices while capturing shared transcriptomic features, two tasks that are typically addressed in isolation. Multi-slice analysis remains challenging due to physical distortions, technical variability, and batch effects. To address these challenges, we introduce Joint Alignment and Deep Embedding for multi-slice ST (JADE), a unified computational framework that simultaneously learns spot-wise alignments and shared low-dimensional embeddings across tissue slices. Unlike existing methods, JADE adopts a roundtrip framework in which each iteration alternates between alignment and embedding refinement. To infer alignment, we employ attention mechanisms that dynamically assess and weight the importance of different embedding dimensions, allowing the model to focus on the most alignment-relevant features while suppressing noise. To the best of our knowledge, JADE is the first method that jointly optimizes alignment and representation learning in a shared latent space, enabling robust multi-slice integration. We demonstrate that JADE outperforms existing alignment and embedding methods across multiple evaluation metrics in the 10x Visium human dorsolateral prefrontal cortex (DLPFC) and Stereo-seq axolotl brain datasets. By bridging spatial alignment and feature integration, JADE provides a scalable and accurate solution for cross-slice analysis of ST data.

IJCAI Conference 2025 Conference Paper

Metapath and Hypergraph Structure-based Multi-Channel Graph Contrastive Learning for Student Performance Prediction

• Lingyun Song
• Xiaofan Sun
• Xinbiao Gan
• Yudai Pan
• Xiaolin Han
• Jie Ma
• Jun Liu
• Xuequn Shang

Considerable attention has been paid to predicting student performance on exercises. The performance of prior studies is determined by the quality of the trait features of students and exercises. Nevertheless, most of the prior study primarily examines simple pairwise interactions in learning trait features, like those between students and exercises or exercises and concepts, while disregarding the complex higher-order interactions that typically exist among these components, which in turn hinders the prediction results. In this paper, we using an innovative Multi-Channel Graph Contrastive Learning (MCGCL) framework that integrates various high-order interactions for predicting student performance. MCGCL characterizes graph structures reflecting various high-order relationships among students, exercises, and concepts through multiple channels, thereby enhancing the trait features of both students and exercises. Moreover, graph contrastive learning is employed to enhance the representation of trait features acquired from high-order graph structures in diverse views. Extensive experiments on real-world datasets show that MCGCL achieves state-of-the-art results on the task of predicting student performance. The code is available at https: //github. com/sunlitsong/MCGCL.

YNIMG Journal 2025 Journal Article

Neural connectivity and balance control in aging: Insights from directed cortical networks during sensory conflict

• Guozheng Wang
• Yi Yang
• Xiaoxia Liu
• Anke Hua
• Xin Luo
• Yiming Cai
• Yanhua Song
• Jian Wang

Balance control is crucial for stability during daily activities, relying on the integration of sensory inputs from the visual, vestibular, and somatosensory systems. Aging impairs the efficiency of these systems, leading to an increased risk of falls; however, the neural mechanisms underlying this decline, particularly under sensory conflict, are not fully understood. This study investigated the effects of aging on neural connectivity and sensory integration during balance tasks. Ninety-six participants (47 older adults and 49 young adults) were subjected to balance perturbation tasks under sensory-congruent and sensory-conflict conditions using a virtual reality headset and rotating platform. Behavioral measures, including postural sway and perceptual accuracy, were recorded. Electroencephalography (EEG) data were analyzed using generalized partial directed coherence (GPDC) to assess the directed functional connectivity and network efficiency. Older adults exhibited significantly greater postural sway, reduced perceptual accuracy, and a diminished ability to detect sensory conflicts than young adults, particularly under conflict conditions. As demonstrated by connectivity analysis, young adults showed adaptive shifts in connectivity from the visual to somatosensory regions during sensory conflict. In contrast, older adults demonstrated a less adaptable mode of connectivity. At the same time, global efficiency and clustering coefficients of young adults were higher, suggesting more effective and modular brain networks. Correlation analyses in older adults revealed that higher visual cortex efficiency was linked to lower postural sway specifically during sensory conflict, whereas higher motor cortex efficiency was associated with greater sway only under sensory-congruent conditions. In short, neural adaptability is vital in sensory integration and balance control. Due to decreased neural flexibility and network efficiency in older adults, their sensory reweighting was undermined and instability increased during the sensory conflict. These findings establish a foundation for development of targeted interventions to strengthen balance and lower the risks of falls in older adults.

AAAI Conference 2025 Conference Paper

RMath: A Logic Reasoning-Focused Datasets Toward Mathematical Multistep Reasoning Tasks

• Ziyi Hu
• Jun Liu
• Zhongzhi Liu
• Yuzhong Liu
• Zheng Xie
• Yiping Song

Mathematical reasoning ability objectively reflects a language model's understanding of implicit knowledge in contexts, with logic being a prerequisite for exploring, articulating and establishing effective reasoning. Large language models (LLMs) have shown great potential in complex reasoning tasks represented by mathematical reasoning. However, existing mathematical datasets either focus on commonsense reasoning, assessing the model's knowledge application ability, or arithmetic problems with fixed calculation rules, evaluating the model's rapid learning capability. There is a lack of datasets that require solving problems solely through logical reasoning. As a result, the performance of LLMs in accurately understanding the implicit logical relationships in problems and deriving conclusions based solely on given conditions is hindered. To address this challenge, we construct a dataset specifically for multiple step reasoning tasks: Reasoning-Math (RMath). This dataset focuses on evaluating logical reasoning abilities with mathematical reasoning problems, covering typical problem types, including direct reasoning problems, hypothetical reasoning problems, and nested reasoning problems. Additionally, we design a standardized annotation scheme that transforms natural language descriptions of conditions into formal propositions. Other annotation contents include problem categories, proposition truth values, and proposition relationship types. This not only reduces biases caused by semantic misunderstandings during problem-solving, but also facilitates the incorporation of theoretically grounded logical reasoning methods to enhance reasoning abilities. Furthermore, we propose a normalization problem-solving framework based on propositional logic for RMath and design the problem-solving process for prompt tuning to guide LLMs to absorb mathematical logical theories and improving reasoning abilities. Finally, we evaluate RMath on several popular LLMs and present the corresponding results.

IROS Conference 2025 Conference Paper

STEAD: Spatio-Temporal Efficient Anomaly Detection for Time and Compute Sensitive Applications

• Andrew Gao
• Jun Liu

This paper presents a new method for anomaly detection in automated systems with time and compute sensitive requirements with unparalleled efficiency. As systems like autonomous driving become increasingly popular, ensuring their safety has become more important than ever. With this motivation, this paper focuses on how to quickly and effectively detect various anomalies in the aforementioned systems. Many detection systems have been developed with great success under spatial contexts. However, there is still significant room for improvement when it comes to temporal context. While there is substantial work regarding this task, there is minimal work done regarding the efficiency of models and their ability to be applied to scenarios that require real-time inference. To address this gap, we propose STEAD (Spatio-Temporal Efficient Anomaly Detection), whose backbone is developed using (2+1)D Convolutions and Performer Linear Attention, which ensures computational efficiency without sacrificing performance. When evaluated on the UCF-Crime benchmark, our base model achieves an AUC of 91. 34%, outperforming the previous SOTA (state of the art), and our fast version achieves an AUC of 88. 87%, while having 99. 70% less parameters and outperforming the previous SOTA as well. The code and pretrained models are made publicly available at https://github.com/agao8/STEAD.

IROS Conference 2025 Conference Paper

Throwing Planning Diffusion: A Solution to Learning and Planning of Robotic Throwing

• Ziqi Xu
• Haodu Li
• Lihao Liu
• Jun Liu
• Xuechao Duan

Dynamic manipulation enables efficient interaction tasks, such as throwing, which rely on finding one or more high-quality trajectories from the initial state to the goal state. While model-free learning methods have been used to acquire efficient robot manipulation configurations, traditional planning algorithms often struggle with multi-task specifications, high-dimensional, and multi-modal trajectory data. Prior generative model-based approaches, have made significant progress in the field of motion planning. Diffusion models, as an emerging generative model, have been widely applied to planning tasks in various environments and have gained attention for their ability in encoding multidimensional and multimodal trajectories. Here we propose our method that combines the diffusion model and model-free throwing methods. Specifically, we use a backward reachable tube to search for throwing configurations, and sample from posterior trajectory distribution conditioned on the throwing configurations. Several trajectory optimization methods are used to ensure the generation of effective throwing trajectories. Experimental results show that our method is effective in generating feasible, smooth, and collision-free throwing trajectories in both simulated and real-world tasks. Additionally, different trajectories are provided to enhance the multimodality of the throwing task.

AAAI Conference 2025 Conference Paper

Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment

• Jun Liu
• Zhenglun Kong
• Pu Zhao
• Changdi Yang
• Xuan Shen
• Hao Tang
• Geng Yuan
• Wei Niu

Structured pruning for large language models (LLMs) has garnered significant academic interest due to its ability to efficiently compress and accelerate LLMs by eliminating redundant weight groups at a coarse-grained granularity. Current structured pruning methods for LLMs typically depend on a singular granularity for assessing weight importance, resulting in notable performance degradation in downstream tasks. Intriguingly, our empirical investigations reveal that utilizing unstructured pruning, which achieves better performance retention by pruning weights at a finer granularity, \emph{i.e.}, individual weights, yields significantly varied sparse LLM structures when juxtaposed to structured pruning. This suggests that evaluating both holistic and individual assessments for weight importance are essential for LLM pruning. Building on this insight, we introduce the Hybrid-grained Weight Importance Assessment (HyWIA), a novel method that merges fine-grained and coarse-grained evaluations of weight importance for the pruning of LLMs. Leveraging an attention mechanism, HyWIA adaptively determines the optimal blend of granularity in weight importance assessments in an end-to-end pruning manner. Extensive experiments on LLaMA-V1/V2, Vicuna, Baichuan, and Bloom across various benchmarks demonstrate the effectiveness of HyWIA in pruning LLMs. For example, HyWIA surpasses the cutting-edge LLM-Pruner by an average margin of 2.82% in accuracy across seven downstream tasks when pruning LLaMA-7B by 50%.

YNIMG Journal 2025 Journal Article

Transcranial vibration stimulation at 40 Hz induced neural activity and promoted the coupling of global brain activity and cerebrospinal fluid flow

• Linghan Kong
• Wei Jin
• Yue Jiang
• Fuhua Yan
• Jun Liu
• Eric C. Leuthardt
• Guang-Zhong Yang
• Yuan Feng

BACKGROUND: Neuroscience advances have highlighted the potential of non-invasive brain stimulation in influencing cognitive and emotional processes. Conventional stimulation methods such as electrical, magnetic, and ultrasound have been studied intensively, but little is known about the mechanical stimulation. OBJECTIVE: To investigate the effects of 40 Hz transcranial vibration stimulation (TVS) on human brain activity, specifically focusing on changes in the Amplitude of Low-Frequency Fluctuation (ALFF), fractional ALFF (fALFF) and Regional Homogeneity (ReHo) as measures of spontaneous brain activity. Additionally, this study investigates alterations in the global blood-oxygen-level-dependent (gBOLD) signal and cerebrospinal fluid (CSF) inflow coupling, which serve as indicators of glymphatic system function. METHODS: A custom-built head actuator was used to apply 40 Hz TVS to human brain. Functional magnetic resonance imaging (fMRI) were performed before and after 5 mins TVS to explore the changes in ALFF and fALFF and the coupling of global brain activity with cerebrospinal fluid flow (CSF), which is related to the glymphatic clearance. RESULTS: Significant increases were observed in both ALFF and fALFF metrics, indicating that 40 Hz TVS effectively enhanced spontaneous brain activity. Additionally, 40 Hz TVS promoted the synchronization of overall brain activity with CSF, suggesting an improvement in glymphatic clearance processes, an effect that 30 Hz or 50 Hz TVS did not replicate. CONCLUSION: Non-invasive brain stimulation using TVS provided important implications for modulating brain physiology and showed prospective therapeutic benefits for neurological diseases.

AAAI Conference 2025 Conference Paper

VA-AR: Learning Velocity-Aware Action Representations with Mixture of Window Attention

• Jiangning Wei
• Lixiong Qin
• Bo Yu
• Tianjian Zou
• Chuhan Yan
• Dandan Xiao
• Yang Yu
• Lan Yang

Action recognition is a crucial task in artificial intelligence, with significant implications across various domains. We initially perform a comprehensive analysis of seven prominent action recognition methods across five widely-used datasets. This analysis reveals a critical, yet previously overlooked, observation: as the velocity of actions increases, the performance of these methods variably declines, undermining their robustness. This decline in performance poses significant challenges for their application in real-world scenarios. Building on these findings, we introduce the Velocity-Aware Action Recognition (VA-AR) framework to obtain robust action representations across different velocities. Our principal insight is that rapid actions (e.g., the giant circle backward in uneven bars or a smash in badminton) occur within short time intervals, necessitating smaller temporal attention windows to accurately capture intricate changes. Conversely, slower actions (e.g., drinking water or wiping face) require larger windows to effectively encompass the broader context. VA-AR employs a Mixture of Window Attention (MoWA) strategy, dynamically adjusting its attention window size based on the action's velocity. This adjustment enables VA-AR to obtain a velocity-aware representation, thereby enhancing the accuracy of action recognition. Extensive experiments confirm that VA-AR achieves state-of-the-art performance on the same five datasets, demonstrating VA-AR's effectiveness across a broad spectrum of action recognition scenarios.

AAAI Conference 2025 Conference Paper

VProChart: Answering Chart Question Through Visual Perception Alignment Agent and Programmatic Solution Reasoning

• Muye Huang
• Lingling Zhang
• Han Lai
• Wenjun Wu
• Xinyu Zhang
• Jun Liu

Charts are widely used for data visualization across various fields, including education, research, and business. Chart Question Answering (CQA) is an emerging task focused on the automatic interpretation and reasoning of data presented in charts. However, chart images are inherently difficult to interpret, and chart-related questions often involve complex logical and numerical reasoning, which hinders the performance of existing models. This paper introduces VProChart, a novel framework designed to address these challenges in CQA by integrating a lightweight Visual Perception Alignment Agent (VPAgent) and a Programmatic Solution Reasoning approach. VPAgent aligns and models chart elements based on principles of human visual perception, enhancing the understanding of chart context. The Programmatic Solution Reasoning approach leverages large language models (LLMs) to transform natural language reasoning questions into structured solution programs, facilitating precise numerical and logical reasoning. Extensive experiments on benchmark datasets such as ChartQA and PlotQA demonstrate that VProChart significantly outperforms existing methods, highlighting its capability in understanding and reasoning with charts.

JMLR Journal 2024 Journal Article

Almost Sure Convergence Rates Analysis and Saddle Avoidance of Stochastic Gradient Methods

• Jun Liu
• Ye Yuan

The vast majority of convergence rates analysis for stochastic gradient methods in the literature focus on convergence in expectation, whereas trajectory-wise almost sure convergence is clearly important to ensure that any instantiation of the stochastic algorithms would converge with probability one. Here we provide a unified almost sure convergence rates analysis for stochastic gradient descent (SGD), stochastic heavy-ball (SHB), and stochastic Nesterov's accelerated gradient (SNAG) methods. We show, for the first time, that the almost sure convergence rates obtained for these stochastic gradient methods on strongly convex functions, are arbitrarily close to their optimal convergence rates possible. For non-convex objective functions, we not only show that a weighted average of the squared gradient norms converges to zero almost surely, but also the last iterates of the algorithms. We further provide last-iterate almost sure convergence rates analysis for stochastic gradient methods on general convex smooth functions, in contrast with most existing results in the literature that only provide convergence in expectation for a weighted average of the iterates. The last-iterate almost sure convergence results also enable us to obtain almost sure avoidance of any strict saddle manifold by stochastic gradient methods with or without momentum. To the best of our knowledge, this is the first time such results are obtained for SHB and SNAG methods. [abs] [ pdf ][ bib ] &copy JMLR 2024. ( edit, beta )

AAAI Conference 2024 Conference Paper

Correlation Matching Transformation Transformers for UHD Image Restoration

• Cong Wang
• Jinshan Pan
• Wei Wang
• Gang Fu
• Siyuan Liang
• Mengzhu Wang
• Xiao-ming Wu
• Jun Liu

This paper proposes UHDformer, a general Transformer for Ultra-High-Definition (UHD) image restoration. UHDformer contains two learning spaces: (a) learning in high-resolution space and (b) learning in low-resolution space. The former learns multi-level high-resolution features and fuses low-high features and reconstructs the residual images, while the latter explores more representative features learning from the high-resolution ones to facilitate better restoration. To better improve feature representation in low-resolution space, we propose to build feature transformation from the high-resolution space to the low-resolution one. To that end, we propose two new modules: Dual-path Correlation Matching Transformation module (DualCMT) and Adaptive Channel Modulator (ACM). The DualCMT selects top C/r (r is greater or equal to 1 which controls the squeezing level) correlation channels from the max-pooling/mean-pooling high-resolution features to replace low-resolution ones in Transformers, which can effectively squeeze useless content to improve the feature representation in low-resolution space to facilitate better recovery. The ACM is exploited to adaptively modulate multi-level high-resolution features, enabling to provide more useful features to low-resolution space for better learning. Experimental results show that our UHDformer reduces about ninety-seven percent model sizes compared with most state-of-the-art methods while significantly improving performance under different training sets on 3 UHD image restoration tasks, including low-light image enhancement, image dehazing, and image deblurring. The source codes will be made available at https://github.com/supersupercong/UHDformer.

NeurIPS Conference 2024 Conference Paper

COSMIC: Compress Satellite Image Efficiently via Diffusion Compensation

• Ziyuan Zhang
• Han Qiu
• Maosen Zhang
• Jun Liu
• Bin Chen
• Tianwei Zhang
• Hewu Li

With the rapidly increasing number of satellites in space and their enhanced capabilities, the amount of earth observation images collected by satellites is exceeding the transmission limits of satellite-to-ground links. Although existing learned image compression solutions achieve remarkable performance by using a sophisticated encoder to extract fruitful features as compression and using a decoder to reconstruct. It is still hard to directly deploy those complex encoders on current satellites' embedded GPUs with limited computing capability and power supply to compress images in orbit. In this paper, we propose COSMIC, a simple yet effective learned compression solution to transmit satellite images. We first design a lightweight encoder (i. e. reducing FLOPs by 2. 5~5X) on satellite to achieve a high image compression ratio to save satellite-to-ground links. Then, for reconstructions on the ground, to deal with the feature extraction ability degradation due to simplifying encoders, we propose a diffusion-based model to compensate image details when decoding. Our insight is that satellite's earth observation photos are not just images but indeed multi-modal data with a nature of Text-to-Image pairing since they are collected with rich sensor data (e. g. coordinates, timestep, etc. ) that can be used as the condition for diffusion generation. Extensive experiments show that COSMIC outperforms state-of-the-art baselines on both perceptual and distortion metrics.

JBHI Journal 2024 Journal Article

Deep Learning-Based Microscopic Cell Detection Using Inverse Distance Transform and Auxiliary Counting

• Rui Liu
• Wei Dai
• Cong Wu
• Tianyi Wu
• Min Wang
• Junxian Zhou
• Xiaozhen Zhang
• Wen Jung Li

Microscopic cell detection is a challenging task due to significant inter-cell occlusions in dense clusters and diverse cell morphologies. This paper introduces a novel framework designed to enhance automated cell detection. The proposed approach integrates a deep learning model that produces an inverse distance transform-based detection map from the given image, accompanied by a secondary network designed to regress a cell density map from the same input. The inverse distance transform-based map effectively highlights each cell instance in the densely populated areas, while the density map accurately estimates the total cell count in the image. Then, a custom counting-aided cell center extraction strategy leverages the cell count obtained by integrating over the density map to refine the detection process, significantly reducing false responses and thereby boosting overall accuracy. The proposed framework demonstrated superior performance with F-scores of 96. 93%, 91. 21%, and 92. 00% on the VGG, MBM, and ADI datasets, respectively, surpassing existing state-of-the-art methods. It also achieved the lowest distance error, further validating the effectiveness of the proposed approach. These results demonstrate significant potential for automated cell analysis in biomedical applications.

JBHI Journal 2024 Journal Article

Deeply Supervised Skin Lesions Diagnosis With Stage and Branch Attention

• Wei Dai
• Rui Liu
• Tianyi Wu
• Min Wang
• Jianqin Yin
• Jun Liu

Accurate and unbiased examinations of skin lesions are critical for the early diagnosis and treatment of skin diseases. Visual features of skin lesions vary significantly because the images are collected from patients with different lesion colours and morphologies by using dissimilar imaging equipment. Recent studies have reported that ensembled convolutional neural networks (CNNs) are practical to classify the images for early diagnosis of skin disorders. However, the practical use of these ensembled CNNs is limited as these networks are heavyweight and inadequate for processing contextual information. Although lightweight networks (e. g. , MobileNetV3 and EfficientNet) were developed to achieve parameter reduction for implementing deep neural networks on mobile devices, insufficient depth of feature representation restricts the performance. To address the existing limitations, we develop a new lite and effective neural network, namely HierAttn. The HierAttn applies a novel deep supervision strategy to learn the local and global features by using multi-stage and multi-branch attention mechanisms with only one training loss. The efficacy of HierAttn was evaluated by using the dermoscopy images dataset ISIC2019 and smartphone photos dataset PAD-UFES-20 (PAD2020). The experimental results show that HierAttn achieves the best accuracy and area under the curve (AUC) among the state-of-the-art lightweight networks.

AAAI Conference 2024 Conference Paper

DifAttack: Query-Efficient Black-Box Adversarial Attack via Disentangled Feature Space

• Jun Liu
• Jiantao Zhou
• Jiandian Zeng
• Jinyu Tian

This work investigates efficient score-based black-box adversarial attacks with high Attack Success Rate (ASR) and good generalizability. We design a novel attack method based on a Disentangled Feature space, called DifAttack, which differs significantly from the existing ones operating over the entire feature space. Specifically, DifAttack firstly disentangles an image's latent feature into an adversarial feature and a visual feature, where the former dominates the adversarial capability of an image, while the latter largely determines its visual appearance. We train an autoencoder for the disentanglement by using pairs of clean images and their Adversarial Examples (AEs) generated from available surrogate models via white-box attack methods. Eventually, DifAttack iteratively optimizes the adversarial feature according to the query feedback from the victim model until a successful AE is generated, while keeping the visual feature unaltered. In addition, due to the avoidance of using surrogate models' gradient information when optimizing AEs for black-box models, our proposed DifAttack inherently possesses better attack capability in the open-set scenario, where the training dataset of the victim model is unknown. Extensive experimental results demonstrate that our method achieves significant improvements in ASR and query efficiency simultaneously, especially in the targeted attack and open-set scenarios. The code is available The code is available at https://github.com/csjunjun/DifAttack.git.

NeurIPS Conference 2024 Conference Paper

DisC-GS: Discontinuity-aware Gaussian Splatting

• Haoxuan Qu
• Zhuoling Li
• Hossein Rahmani
• Yujun Cai
• Jun Liu

Recently, Gaussian Splatting, a method that represents a 3D scene as a collection of Gaussian distributions, has gained significant attention in addressing the task of novel view synthesis. In this paper, we highlight a fundamental limitation of Gaussian Splatting: its inability to accurately render discontinuities and boundaries in images due to the continuous nature of Gaussian distributions. To address this issue, we propose a novel framework enabling Gaussian Splatting to perform discontinuity-aware image rendering. Additionally, we introduce a B\'ezier-boundary gradient approximation strategy within our framework to keep the ``differentiability'' of the proposed discontinuity-aware rendering process. Extensive experiments demonstrate the efficacy of our framework.

YNIMG Journal 2024 Journal Article

Distinct patterns of electrophysiologic-neuroimaging correlations between Parkinson's disease and multiple system atrophy

• Puyu Li
• Xinyi Zhou
• Ningdi Luo
• Ruinan Shen
• Xue Zhu
• Min Zhong
• Sijia Huang
• Naying He

Due to a high degree of symptom overlap in the early stages, with movement disorders predominating, Parkinson's disease (PD) and multiple system atrophy (MSA) may exhibit a similar decline in motor areas, yet they differ in their spread throughout the brain, ultimately resulting in two distinct diseases. Drawing upon neuroimaging analyses and altered motor cortex excitability, potential diffusion mechanisms were delved into, and comparisons of correlations across distinct disease groups were conducted in a bid to uncover significant pathological disparities. We recruited thirty-five PD, thirty-seven MSA, and twenty-eight matched controls to conduct clinical assessments, electromyographic recording, and magnetic resonance imaging scanning during the "on medication" state. Patients with neurodegeneration displayed a widespread decrease in electrophysiology in bilateral M1. Brain function in early PD was still in the self-compensatory phase and there was no significant change. MSA patients demonstrated an increase in intra-hemispheric function coupled with a decrease in diffusivity, indicating a reduction in the spread of neural signals. The level of resting motor threshold in healthy aged showed broad correlations with both clinical manifestations and brain circuits related to left M1, which was absent in disease states. Besides, ICF exhibited distinct correlations with functional connections between right M1 and left middle temporal gyrus in all groups. The present study identified subtle differences in the functioning of PD and MSA related to bilateral M1. By combining clinical information, cortical excitability, and neuroimaging intuitively, we attempt to bring light on the potential mechanisms that may underlie the development of neurodegenerative disease.

NeurIPS Conference 2024 Conference Paper

Generated and Pseudo Content guided Prototype Refinement for Few-shot Point Cloud Segmentation

• Lili Wei
• Congyan Lang
• Ziyi Chen
• Tao Wang
• Yidong Li
• Jun Liu

Few-shot 3D point cloud semantic segmentation aims to segment query point clouds with only a few annotated support point clouds. Existing prototype-based methods learn prototypes from the 3D support set to guide the segmentation of query point clouds. However, they encounter the challenge of low prototype quality due to constrained semantic information in the 3D support set and class information bias between support and query sets. To address these issues, in this paper, we propose a novel framework called Generated and Pseudo Content guided Prototype Refinement (GPCPR), which explicitly leverages LLM-generated content and reliable query context to enhance prototype quality. GPCPR achieves prototype refinement through two core components: LLM-driven Generated Content-guided Prototype Refinement (GCPR) and Pseudo Query Context-guided Prototype Refinement (PCPR). Specifically, GCPR integrates diverse and differentiated class descriptions generated by large language models to enrich prototypes with comprehensive semantic knowledge. PCPR further aggregates reliable class-specific pseudo-query context to mitigate class information bias and generate more suitable query-specific prototypes. Furthermore, we introduce a dual-distillation regularization term, enabling knowledge transfer between early-stage entities (prototypes or pseudo predictions) and their deeper counterparts to enhance refinement. Extensive experiments demonstrate the superiority of our method, surpassing the state-of-the-art methods by up to 12. 10% and 13. 75% mIoU on S3DIS and ScanNet, respectively.

EAAI Journal 2024 Journal Article

High-resolution cross-scale transformer: A deep learning model for bolt loosening detection based on monocular vision measurement

• Tianyi Wu
• Ke Shang
• Wei Dai
• Min Wang
• Rui Liu
• Junxian Zhou
• Jun Liu

The reliability of bolt connections significantly impacts the operational state and lifespan of industrial equipment. Vision-based noncontact methods exhibit high efficiency in bolt loosening detection. However, limited image features hinder measurement accuracy. To improve bolt loosening detection performance, this paper proposes a novel deep learning backbone, the high-resolution cross-scale transformer, to extract high precision keypoints for bolt three-dimensional model construction. Simultaneously, a monocular vision measurement model is established to get the bolt exposed length and evaluate the connection loosening state. The proposed backbone hybridizes the advantages of high-resolution architecture and transformer, realizing global information aggregation and fine-grained image details. A simplified module, dual-scale multi-head self-attention, is designed to reduce the computational redundancy caused by the implementation of high-resolution multi-branch architecture. In the experiment section, the high-resolution cross-scale transformer outperforms other keypoint detection baselines, achieving the top one performance with 91. 6 average precision and 84. 9 average recall. The monocular vision measurement model realizes a 0. 053 mm error with a 0. 028 mm standard deviation, satisfying the industrial implementation requirement. Additionally, the model is tested on different industrial situations and an additional outside dataset, indicating the model’s robustness and actual environment adaptability.

EAAI Journal 2024 Journal Article

Intelligent identification of power grid parameters based on dynamic weighting

• Gang Zou
• Min Xia
• Liudong Zhang
• Zhen Lei
• Zhiqiang Peng
• Jun Liu

Transmission line parameter identification plays a crucial role in power systems. However, existing methods for identifying transmission line parameters face several challenges: (1) when dealing with non-Euclidean data such as power grid data, a considerable amount of noise is introduced, (2) inability to treat the power grid as a global system, limiting calculations to individual transmission lines and overlooking inter-branch correlations, (3) high sensitivity to data contamination. To address these issues, this paper proposes a multi-task noisy graph attention network (MNGAN) for power grid parameter identification, which leverages the spatial structure of the power grid and considers the inter-branch correlations. By computing the homogeneity and average degree of the graph data, we introduce an adaptive attention mechanism to the model. This attention mechanism dynamically adjusts the model’s focus on key information, mitigating the impact of data contamination and improving parameter identification accuracy. Additionally, to achieve simultaneous recognition of multiple branch parameters and enhance model training speed and accuracy, we incorporate a multi-task loss function based on homoscedastic uncertainty. Experimental results demonstrate the superiority of our proposed model over other machine learning and deep learning methods, providing more accurate predictions of branch parameters.

AAAI Conference 2024 Conference Paper

LAFA: Multimodal Knowledge Graph Completion with Link Aware Fusion and Aggregation

• Bin Shang
• Yinliang Zhao
• Jun Liu
• Di Wang

Recently, an enormous amount of research has emerged on multimodal knowledge graph completion (MKGC), which seeks to extract knowledge from multimodal data and predict the most plausible missing facts to complete a given multimodal knowledge graph (MKG). However, existing MKGC approaches largely ignore that visual information may introduce noise and lead to uncertainty when adding them to the traditional KG embeddings due to the contribution of each associated image to entity is different in diverse link scenarios. Moreover, treating each triple independently when learning entity embeddings leads to local structural and the whole graph information missing. To address these challenges, we propose a novel link aware fusion and aggregation based multimodal knowledge graph completion model named LAFA, which is composed of link aware fusion module and link aware aggregation module. The link aware fusion module alleviates noise of irrelevant visual information by calculating the importance between an entity and its associated images in different link scenarios, and fuses the visual and structural embeddings according to the importance through our proposed modality embedding fusion mechanism. The link aware aggregation module assigns neighbor structural information to a given central entity by calculating the importance between the entity and its neighbors, and aggregating the fused embeddings through linear combination according to the importance. Extensive experiments on standard datasets validate that LAFA can obtain state-of-the-art performance.

AAAI Conference 2024 Conference Paper

Learning Task-Aware Language-Image Representation for Class-Incremental Object Detection

• Hongquan Zhang
• Bin-Bin Gao
• Yi Zeng
• Xudong Tian
• Xin Tan
• Zhizhong Zhang
• Yanyun Qu
• Jun Liu

Class-incremental object detection (CIOD) is a real-world desired capability, requiring an object detector to continuously adapt to new tasks without forgetting learned ones, with the main challenge being catastrophic forgetting. Many methods based on distillation and replay have been proposed to alleviate this problem. However, they typically learn on a pure visual backbone, neglecting the powerful representation capabilities of textual cues, which to some extent limits their performance. In this paper, we propose task-aware language-image representation to mitigate catastrophic forgetting, introducing a new paradigm for language-image-based CIOD. First of all, we demonstrate the significant advantage of language-image detectors in mitigating catastrophic forgetting. Secondly, we propose a learning task-aware language-image representation method that overcomes the existing drawback of directly utilizing the language-image detector for CIOD. More specifically, we learn the language-image representation of different tasks through an insulating approach in the training stage, while using the alignment scores produced by task-specific language-image representation in the inference stage. Through our proposed method, language-image detectors can be more practical for CIOD. We conduct extensive experiments on COCO 2017 and Pascal VOC 2007 and demonstrate that the proposed method achieves state-of-the-art results under the various CIOD settings.

NeurIPS Conference 2024 Conference Paper

Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering

• Jie Ma
• Min Hu
• Pinghui Wang
• Wangchun Sun
• Lingyun Song
• Hongbin Pei
• Jun Liu
• Youtian Du

Audio-Visual Question Answering (AVQA) is a complex multi-modal reasoning task, demanding intelligent systems to accurately respond to natural language queries based on audio-video input pairs. Nevertheless, prevalent AVQA approaches are prone to overlearning dataset biases, resulting in poor robustness. Furthermore, current datasets may not provide a precise diagnostic for these methods. To tackle these challenges, firstly, we propose a novel dataset, MUSIC-AVQA-R, crafted in two steps: rephrasing questions within the test split of a public dataset ( MUSIC-AVQA ) and subsequently introducing distribution shifts to split questions. The former leads to a large, diverse test space, while the latter results in a comprehensive robustness evaluation on rare, frequent, and overall questions. Secondly, we propose a robust architecture that utilizes a multifaceted cycle collaborative debiasing strategy to overcome bias learning. Experimental results show that this architecture achieves state-of-the-art performance on MUSIC-AVQA-R, notably obtaining a significant improvement of 9. 32\%. Extensive ablation experiments are conducted on the two datasets mentioned to analyze the component effectiveness within the debiasing strategy. Additionally, we highlight the limited robustness of existing multi-modal QA methods through the evaluation on our dataset. We also conduct experiments combining various baselines with our proposed strategy on two datasets to verify its plug-and-play capability. Our dataset and code are available at https: //github. com/reml-group/MUSIC-AVQA-R.

AAAI Conference 2024 Conference Paper

MatchDet: A Collaborative Framework for Image Matching and Object Detection

• Jinxiang Lai
• Wenlong Wu
• Bin-Bin Gao
• Jun Liu
• Jiawei Zhan
• Congchong Nie
• Yi Zeng
• Chengjie Wang

Image matching and object detection are two fundamental and challenging tasks, while many related applications consider them two individual tasks (i.e. task-individual). In this paper, a collaborative framework called MatchDet (i.e. task-collaborative) is proposed for image matching and object detection to obtain mutual improvements. To achieve the collaborative learning of the two tasks, we propose three novel modules, including a Weighted Spatial Attention Module (WSAM) for Detector, and Weighted Attention Module (WAM) and Box Filter for Matcher. Specifically, the WSAM highlights the foreground regions of target image to benefit the subsequent detector, the WAM enhances the connection between the foreground regions of pair images to ensure high-quality matches, and Box Filter mitigates the impact of false matches. We evaluate the approaches on a new benchmark with two datasets called Warp-COCO and miniScanNet. Experimental results show our approaches are effective and achieve competitive improvements.

AAAI Conference 2024 Conference Paper

Mixed Geometry Message and Trainable Convolutional Attention Network for Knowledge Graph Completion

• Bin Shang
• Yinliang Zhao
• Jun Liu
• Di Wang

Knowledge graph completion (KGC) aims to study the embedding representation to solve the incompleteness of knowledge graphs (KGs). Recently, graph convolutional networks (GCNs) and graph attention networks (GATs) have been widely used in KGC tasks by capturing neighbor information of entities. However, Both GCNs and GATs based KGC models have their limitations, and the best method is to analyze the neighbors of each entity (pre-validating), while this process is prohibitively expensive. Furthermore, the representation quality of the embeddings can affect the aggregation of neighbor information (message passing). To address the above limitations, we propose a novel knowledge graph completion model with mixed geometry message and trainable convolutional attention network named MGTCA. Concretely, the mixed geometry message function generates rich neighbor message by integrating spatially information in the hyperbolic space, hypersphere space and Euclidean space jointly. To complete the autonomous switching of graph neural networks (GNNs) and eliminate the necessity of pre-validating the local structure of KGs, a trainable convolutional attention network is proposed by comprising three types of GNNs in one trainable formulation. Furthermore, a mixed geometry scoring function is proposed, which calculates scores of triples by novel prediction function and similarity function based on different geometric spaces. Extensive experiments on three standard datasets confirm the effectiveness of our innovations, and the performance of MGTCA is significantly improved compared to the state-of-the-art approaches.

AAAI Conference 2024 Conference Paper

MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving

• Guangfeng Jiang
• Jun Liu
• Yuzhi Wu
• Wenlong Liao
• Tao He
• Pai Peng

Instance segmentation is a fundamental research in computer vision, especially in autonomous driving. However, manual mask annotation for instance segmentation is quite time-consuming and costly. To address this problem, some prior works attempt to apply weakly supervised manner by exploring 2D or 3D boxes. However, no one has ever successfully segmented 2D and 3D instances simultaneously by only using 2D box annotations, which could further reduce the annotation cost by an order of magnitude. Thus, we propose a novel framework called Multimodal Weakly Supervised Instance Segmentation (MWSIS), which incorporates various fine-grained label correction modules for both 2D and 3D modalities, along with a new multimodal cross-supervision approach. In the 2D pseudo label generation branch, the Instance-based Pseudo Mask Generation (IPG) module utilizes predictions for self-supervised correction. Similarly, in the 3D pseudo label generation branch, the Spatial-based Pseudo Label Generation (SPG) module generates pseudo labels by incorporating the spatial prior information of the point cloud. To further refine the generated pseudo labels, the Point-based Voting Label Correction (PVC) module utilizes historical predictions for correction. Additionally, a Ring Segment-based Label Correction (RSC) module is proposed to refine the predictions by leveraging the depth prior information from the point cloud. Finally, the Consistency Sparse Cross-modal Supervision (CSCS) module reduces the inconsistency of multimodal predictions by response distillation. Particularly, transferring the 3D backbone to downstream tasks not only improves the performance of the 3D detectors, but also outperforms fully supervised instance segmentation with only 5% fully supervised annotations. On the Waymo dataset, the proposed framework demonstrates significant improvements over the baseline, especially achieving 2.59% mAP and 12.75% mAP increases for 2D and 3D instance segmentation tasks, respectively. The code is available at https://github.com/jiangxb98/mwsis-plugin.

ICML Conference 2024 Conference Paper

Network Tight Community Detection

• Jiayi Deng
• Xiaodong Yang
• Jun Yu
• Jun Liu
• Zhaiming Shen
• Danyang Huang
• Huimin Cheng

Conventional community detection methods often categorize all nodes into clusters. However, the presumed community structure of interest may only be valid for a subset of nodes (named as ‘tight nodes’), while the rest of the network may consist of noninformative “scattered nodes”. For example, a protein-protein network often contains proteins that do not belong to specific biological functional modules but are involved in more general processes, or act as bridges between different functional modules. Forcing each of these proteins into a single cluster introduces unwanted biases and obscures the underlying biological implication. To address this issue, we propose a tight community detection (TCD) method to identify tight communities excluding scattered nodes. The algorithm enjoys a strong theoretical guarantee of tight node identification accuracy and is scalable for large networks. The superiority of the proposed method is demonstrated by various synthetic and real experiments.

YNIMG Journal 2024 Journal Article

Neuro-immune communication at the core of craving-associated brain structural network reconfiguration in methamphetamine users

• Yanyao Du
• Jiaqi Zhang
• Dan Cao
• Wenhan Yang
• Jin Li
• Deying Li
• Ming Song
• Zhengyi Yang

Methamphetamine (MA) use disorder is a chronic neurotoxic brain disease characterized by a high risk of relapse driven by intense cravings. However, the neurobiological signatures of cravings remain unclear, limiting the effectiveness of various treatment methods. Diffusion MRI (dMRI) scans from 62 MA users and 57 healthy controls (HC) were used in this study. The MA users were longitudinally followed up during their period of long-term abstinence (duration of long-term abstinence: 347. 52±99. 25 days). We systematically quantified the control ability of each brain region for craving-associated state transitions using network control theory from a causal perspective. Craving-associated structural alterations (CSA) were investigated through multivariate group comparisons and biological relevance analysis. The neural mechanisms underlying CSA were elucidated using transcriptomic and neurochemical analyses. We observed that long-term abstinence-induced structural alterations significantly influenced the state transition energy involved in the cognitive control response to external information, which correlated with changes in craving scores (r ∼ 0. 35, P <0. 01). Our causal network analysis further supported the crucial role of the prefrontal cortex (PFC) in craving mechanisms. Notably, while the PFC is central to the craving, the CSAs were distributed widely across multiple brain regions (PFDR <0. 05), with strong alterations in somatomotor regions (PFDR <0. 05) and moderate alterations in high-level association networks (PFDR <0. 05). Additionally, transcriptomic, chemical compounds, cell-type analyses, and molecular imaging collectively highlight the influence of neuro-immune communication on human craving modulation. Our results offer an integrative, multi-scale perspective on unraveling the neural underpinnings of craving and suggest that neuro-immune signaling may be a promising target for future human addiction therapeutics.

TCS Journal 2024 Journal Article

Universal enzymatic numerical P systems with small number of enzymatic rules

• Jun Liu
• Leiya Wang
• Gexiang Zhang
• Sergey Verlan
• Ming Zhu

Enzymatic Numerical P Systems (ENPSs) are a model of membrane computing that is well-suited for the simulation of physical processes and that has been used for the design and the implementation of motion controllers for wheeled robots and flying drones. The ENPSs model has been proven to be Turing universal and the theoretical effort was focused on minimizing various descriptional complexity parameters. In this paper, we explore the minimum number of enzymatic rules needed to achieve universality in ENPSs, specifically focusing on the all-parallel derivation mode where all applicable rules are applied at the same time. We show that in the case of a linear restriction for production functions, the universality can be obtained using 21 enzymatic rules, substantially improving previously known results. If production functions are allowed to be polynomials of degree 2, we show that a single enzymatic rule is sufficient to achieve universality. To obtain these results, a new proof method is introduced based on the translation of ENPSs to systems of conditional recurrences.

IJCAI Conference 2023 Conference Paper

Clustered-patch Element Connection for Few-shot Learning

• Jinxiang Lai
• Siqian Yang
• Junhong Zhou
• Wenlong Wu
• Xiaochen Chen
• Jun Liu
• Bin-Bin Gao
• Chengjie Wang

Weak feature representation problem has influenced the performance of few-shot classification task for a long time. To alleviate this problem, recent researchers build connections between support and query instances through embedding patch features to generate discriminative representations. However, we observe that there exists semantic mismatches (foreground/ background) among these local patches, because the location and size of the target object are not fixed. What is worse, these mismatches result in unreliable similarity confidences, and complex dense connection exacerbates the problem. According to this, we propose a novel Clustered-patch Element Connection (CEC) layer to correct the mismatch problem. The CEC layer leverages Patch Cluster and Element Connection operations to collect and establish reliable connections with high similarity patch features, respectively. Moreover, we propose a CECNet, including CEC layer based attention module and distance metric. The former is utilized to generate a more discriminative representation benefiting from the global clustered-patch features, and the latter is introduced to reliably measure the similarity between pair-features. Extensive experiments demonstrate that our CECNet outperforms the state-of-the-art methods on classification benchmark. Furthermore, our CEC approach can be extended into few-shot segmentation and detection tasks, which achieves competitive performances.

IJCAI Conference 2023 Conference Paper

Diagram Visual Grounding: Learning to See with Gestalt-Perceptual Attention

• Xin Hu
• Lingling Zhang
• Jun Liu
• Xinyu Zhang
• Wenjun Wu
• Qianying Wang

Diagram visual grounding aims to capture the correlation between language expression and local objects in the diagram, and plays an important role in the applications like textbook question answering and cross-modal retrieval. Most diagrams consist of several colors and simple geometries. This results in sparse low-level visual features, which further aggravates the gap between low-level visual and high-level semantic features of diagrams. The phenomenon brings challenges to the diagram visual grounding. To solve the above issues, we propose a gestalt-perceptual attention model to align the diagram objects and language expressions. For low-level visual features, inspired by the gestalt that simulates human visual system, we build a gestalt-perception graph network to make up the features learned by the traditional backbone network. For high-level semantic features, we design a multi-modal context attention mechanism to facilitate the interaction between diagrams and language expressions, so as to enhance the semantics of diagrams. Finally, guided by diagram features and linguistic embedding, the target query is gradually decoded to generate the coordinates of the referred object. By conducting comprehensive experiments on diagrams and natural images, we demonstrate that the proposed model achieves superior performance over the competitors. Our code will be released at https: //github. com/AIProCode/GPA.

EAAI Journal 2023 Journal Article

Enhancing fuzzy evidential reasoning approach using dynamic adjustment mechanism and new rule-based transformation for engineering emergency response evaluation

• Yan Tu
• Zhuang Ma
• Jun Liu
• Xiaoyang Zhou
• Benjamin Lev

With the frequent occurrence of various emergency events in engineering field, engineering emergency response (EER) evaluation plays an increasingly significant role in handling such situations and provides great challenges to research since the uncertain information and the urgent response time. Aiming at achieving timely and effective emergency response, an enhanced evidential reasoning (ER) approach based on the dynamic adjustment mechanism and new rule-based transformation is proposed. First, the linguistic terms to represent various preference information provided by experts are encoded into the trapezoidal interval type-2 fuzzy sets (TrIT2FSs) with different granularities. Second, for ensuring the validity of the information, based on the definition of the expert decision risk preference coefficients, a dynamic adjustment mechanism is constructed to identify and adjust the preference information. Meanwhile, combined with social network, the experts’ weights can be calculated and revised several times to obtain group information. Then, a new rule-based transformation and related optimization models are proposed to convert the TrIT2FSs into interval belief structures. Furthermore, considering the importance of attributes, the relative weights and interval belief structures are combined. Finally, according to integrated interval belief structures obtained by the analytical ER algorithm, a new ranking approach with the optimism degree and decision tendency degree is constructed to rank the interval expected utility and score utility of each alternative. To further show the effectiveness, superiorities, and stability of the proposed method, a case study on the EER evaluation is preformed and some comparisons and discussions are provided.

AAAI Conference 2023 Conference Paper

GPTR: Gestalt-Perception Transformer for Diagram Object Detection

• Xin Hu
• Lingling Zhang
• Jun Liu
• Jinfu Fan
• Yang You
• Yaqiang Wu

Diagram object detection is the key basis of practical applications such as textbook question answering. Because the diagram mainly consists of simple lines and color blocks, its visual features are sparser than those of natural images. In addition, diagrams usually express diverse knowledge, in which there are many low-frequency object categories in diagrams. These lead to the fact that traditional data-driven detection model is not suitable for diagrams. In this work, we propose a gestalt-perception transformer model for diagram object detection, which is based on an encoder-decoder architecture. Gestalt perception contains a series of laws to explain human perception, that the human visual system tends to perceive patches in an image that are similar, close or connected without abrupt directional changes as a perceptual whole object. Inspired by these thoughts, we build a gestalt-perception graph in transformer encoder, which is composed of diagram patches as nodes and the relationships between patches as edges. This graph aims to group these patches into objects via laws of similarity, proximity, and smoothness implied in these edges, so that the meaningful objects can be effectively detected. The experimental results demonstrate that the proposed GPTR achieves the best results in the diagram object detection task. Our model also obtains comparable results over the competitors in natural image object detection.

NeurIPS Conference 2023 Conference Paper

Joint Attribute and Model Generalization Learning for Privacy-Preserving Action Recognition

• Duo Peng
• Li Xu
• Qiuhong Ke
• Ping Hu
• Jun Liu

Privacy-Preserving Action Recognition (PPAR) aims to transform raw videos into anonymous ones to prevent privacy leakage while maintaining action clues, which is an increasingly important problem in intelligent vision applications. Despite recent efforts in this task, it is still challenging to deal with novel privacy attributes and novel privacy attack models that are unavailable during the training phase. In this paper, from the perspective of meta-learning (learning to learn), we propose a novel Meta Privacy-Preserving Action Recognition (MPPAR) framework to improve both generalization abilities above (i. e. , generalize to novel privacy attributes and novel privacy attack models ) in a unified manner. Concretely, we simulate train/test task shifts by constructing disjoint support/query sets w. r. t. privacy attributes or attack models. Then, a virtual training and testing scheme is applied based on support/query sets to provide feedback to optimize the model's learning toward better generalization. Extensive experiments demonstrate the effectiveness and generalization of the proposed framework compared to state-of-the-arts.

NeurIPS Conference 2023 Conference Paper

LMC: Large Model Collaboration with Cross-assessment for Training-Free Open-Set Object Recognition

• Haoxuan Qu
• Xiaofei Hui
• Yujun Cai
• Jun Liu

Open-set object recognition aims to identify if an object is from a class that has been encountered during training or not. To perform open-set object recognition accurately, a key challenge is how to reduce the reliance on spurious-discriminative features. In this paper, motivated by that different large models pre-trained through different paradigms can possess very rich while distinct implicit knowledge, we propose a novel framework named Large Model Collaboration (LMC) to tackle the above challenge via collaborating different off-the-shelf large models in a training-free manner. Moreover, we also incorporate the proposed framework with several novel designs to effectively extract implicit knowledge from large models. Extensive experiments demonstrate the efficacy of our proposed framework. Code is available \href{https: //github. com/Harryqu123/LMC}{here}.

AAAI Conference 2023 Conference Paper

SpatialFormer: Semantic and Target Aware Attentions for Few-Shot Learning

• Jinxiang Lai
• Siqian Yang
• Wenlong Wu
• Tao Wu
• Guannan Jiang
• Xi Wang
• Jun Liu
• Bin-Bin Gao

Recent Few-Shot Learning (FSL) methods put emphasis on generating a discriminative embedding features to precisely measure the similarity between support and query sets. Current CNN-based cross-attention approaches generate discriminative representations via enhancing the mutually semantic similar regions of support and query pairs. However, it suffers from two problems: CNN structure produces inaccurate attention map based on local features, and mutually similar backgrounds cause distraction. To alleviate these problems, we design a novel SpatialFormer structure to generate more accurate attention regions based on global features. Different from the traditional Transformer modeling intrinsic instance-level similarity which causes accuracy degradation in FSL, our SpatialFormer explores the semantic-level similarity between pair inputs to boost the performance. Then we derive two specific attention modules, named SpatialFormer Semantic Attention (SFSA) and SpatialFormer Target Attention (SFTA), to enhance the target object regions while reduce the background distraction. Particularly, SFSA highlights the regions with same semantic information between pair features, and SFTA finds potential foreground object regions of novel feature that are similar to base categories. Extensive experiments show that our methods are effective and achieve new state-of-the-art results on few-shot classification benchmarks.

NeurIPS Conference 2022 Conference Paper

Decoupling Classifier for Boosting Few-shot Object Detection and Instance Segmentation

• Bin-Bin Gao
• Xiaochen Chen
• Zhongyi Huang
• Congchong Nie
• Jun Liu
• Jinxiang Lai
• Guannan Jiang
• Xi Wang

This paper focus on few-shot object detection~(FSOD) and instance segmentation~(FSIS), which requires a model to quickly adapt to novel classes with a few labeled instances. The existing methods severely suffer from bias classification because of the missing label issue which naturally exists in an instance-level few-shot scenario and is first formally proposed by us. Our analysis suggests that the standard classification head of most FSOD or FSIS models needs to be decoupled to mitigate the bias classification. Therefore, we propose an embarrassingly simple but effective method that decouples the standard classifier into two heads. Then, these two individual heads are capable of independently addressing clear positive samples and noisy negative samples which are caused by the missing label. In this way, the model can effectively learn novel classes while mitigating the effects of noisy negative samples. Without bells and whistles, our model without any additional computation cost and parameters consistently outperforms its baseline and state-of-the-art by a large margin on PASCAL VOC and MS-COCO benchmarks for FSOD and FSIS tasks. \footnote{\url{https: //csgaobb. github. io/Projects/DCFS}. }

YNICL Journal 2022 Journal Article

Disrupted coupling between salience network segregation and glucose metabolism is associated with cognitive decline in Alzheimer's disease – A simultaneous resting-state FDG-PET/fMRI study

• Miao Zhang
• Ziyun Guan
• Yaoyu Zhang
• Wanqing Sun
• Wenli Li
• Jialin Hu
• Binyin Li
• Guanyu Ye

The aberrant organization and functioning of three core neurocognitive networks (NCNs), i.e., default-mode network (DMN), central executive network (CEN), and salience network (SN), are among the prominent features in Alzheimer's disease (AD). The dysregulation of both intra- and inter-network functional connectivities (FCs) of the three NCNs contributed to AD-related cognitive and behavioral abnormalities. Brain functional network segregation, integrating intra- and inter-network FCs, is essential for maintaining the energetic efficiency of brain metabolism. The association of brain functional network segregation, together with glucose metabolism, with age-related cognitive decline was recently shown. Yet how these joint functional-metabolic biomarkers relate to cognitive decline along with mild cognitive impairment (MCI) and AD remains to be elucidated. In this study, under the framework of the triple-network model, we performed a hybrid FDG-PET/fMRI study to evaluate the concurrent changes of resting-state brain intrinsic FCs and glucose metabolism of the three NCNs across cognitively normal (CN) (N = 24), MCI (N = 21), and AD (N = 21) groups. Lower network segregation and glucose metabolism were observed in all three NCNs in patients with AD. More interestingly, in the SN, the coupled relationship between network segregation and glucose metabolism existed in the CN group (r = 0.523, p = 0.013) and diminished in patients with MCI (r = 0.431, p = 0.065) and AD (r = 0.079, p = 0.748). Finally, the glucose metabolism of the DMN (r = 0.380, p = 0.017) and the network segregation of the SN (r = 0.363, p = 0.023) were significantly correlated with the general cognitive status of the patients. Our findings suggest that the impaired SN segregation and its uncoupled relationship with glucose metabolism contribute to the cognitive decline in AD.

NeurIPS Conference 2022 Conference Paper

Heatmap Distribution Matching for Human Pose Estimation

• Haoxuan Qu
• Li Xu
• Yujun Cai
• Lin Geng Foo
• Jun Liu

For tackling the task of 2D human pose estimation, the great majority of the recent methods regard this task as a heatmap estimation problem, and optimize the heatmap prediction using the Gaussian-smoothed heatmap as the optimization objective and using the pixel-wise loss (e. g. MSE) as the loss function. In this paper, we show that optimizing the heatmap prediction in such a way, the model performance of body joint localization, which is the intrinsic objective of this task, may not be consistently improved during the optimization process of the heatmap prediction. To address this problem, from a novel perspective, we propose to formulate the optimization of the heatmap prediction as a distribution matching problem between the predicted heatmap and the dot annotation of the body joint directly. By doing so, our proposed method does not need to construct the Gaussian-smoothed heatmap and can achieve a more consistent model performance improvement during the optimization of the heatmap prediction. We show the effectiveness of our proposed method through extensive experiments on the COCO dataset and the MPII dataset.

JBHI Journal 2022 Journal Article

Multimodal Disentangled Variational Autoencoder With Game Theoretic Interpretability for Glioma Grading

• Jianhong Cheng
• Min Gao
• Jin Liu
• Hailin Yue
• Hulin Kuang
• Jun Liu
• Jianxin Wang

Effective fusion of multimodal magnetic resonance imaging (MRI) is of great significance to boost the accuracy of glioma grading thanks to the complementary information provided by different imaging modalities. However, how to extract the common and distinctive information from MRI to achieve complementarity is still an open problem in information fusion research. In this study, we propose a deep neural network model termed as multimodal disentangled variational autoencoder (MMD-VAE) for glioma grading based on radiomics features extracted from preoperative multimodal MRI images. Specifically, the radiomics features are quantized and extracted from the region of interest for each modality. Then, the latent representations of variational autoencoder for these features are disentangled into common and distinctive representations to obtain the shared and complementary data among modalities. Afterwards, cross-modality reconstruction loss and common-distinctive loss are designed to ensure the effectiveness of the disentangled representations. Finally, the disentangled common and distinctive representations are fused to predict the glioma grades, and SHapley Additive exPlanations (SHAP) is adopted to quantitatively interpret and analyze the contribution of the important features to grading. Experimental results on two benchmark datasets demonstrate that the proposed MMD-VAE model achieves encouraging predictive performance (AUC: 0. 9939) on a public dataset, and good generalization performance (AUC: 0. 9611) on a cross-institutional private dataset. These quantitative results and interpretations may help radiologists understand gliomas better and make better treatment decisions for improving clinical outcomes.

NeurIPS Conference 2022 Conference Paper

Neural Lyapunov Control of Unknown Nonlinear Systems with Stability Guarantees

• Ruikun Zhou
• Thanin Quartz
• Hans De Sterck
• Jun Liu

Learning for control of dynamical systems with formal guarantees remains a challenging task. This paper proposes a learning framework to simultaneously stabilize an unknown nonlinear system with a neural controller and learn a neural Lyapunov function to certify a region of attraction (ROA) for the closed-loop system with provable guarantees. The algorithmic structure consists of two neural networks and a satisfiability modulo theories (SMT) solver. The first neural network is responsible for learning the unknown dynamics. The second neural network aims to identify a valid Lyapunov function and a provably stabilizing nonlinear controller. The SMT solver verifies the candidate Lyapunov function satisfies the Lyapunov conditions. We further provide theoretical guarantees of the proposed learning framework and show that the obtained Lyapunov function indeed verifies for the unknown nonlinear system under mild assumptions. We illustrate the effectiveness of the results with a few numerical experiments.

AAAI Conference 2022 Conference Paper

REMOTE: Reinforced Motion Transformation Network for Semi-supervised 2D Pose Estimation in Videos

• Xianzheng Ma
• Hossein Rahmani
• Zhipeng Fan
• Bin Yang
• Jun Chen
• Jun Liu

Existing approaches for 2D pose estimation in videos often require a large number of dense annotations, which are costly and labor intensive to acquire. In this paper, we propose a semi-supervised REinforced MOtion Transformation nEtwork (REMOTE) to leverage a few labeled frames and temporal pose variations in videos, which enables effective learning of 2D pose estimation in sparsely annotated videos. Specifically, we introduce a Motion Transformer (MT) module to perform cross frame reconstruction, aiming to learn motion dynamic knowledge in videos. Besides, a novel reinforcement learning-based Frame Selection Agent (FSA) is designed within our framework, which is able to harness informative frame pairs on the fly to enhance the pose estimator under our cross reconstruction mechanism. We conduct extensive experiments that show the efficacy of our proposed REMOTE framework.

NeurIPS Conference 2022 Conference Paper

ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value Factorization

• Siqi Shen
• Mengwei Qiu
• Jun Liu
• Weiquan Liu
• Yongquan Fu
• Xinwang Liu
• Cheng Wang

The factorization of state-action value functions for Multi-Agent Reinforcement Learning (MARL) is important. Existing studies are limited by their representation capability, sample efficiency, and approximation error. To address these challenges, we propose, ResQ, a MARL value function factorization method, which can find the optimal joint policy for any state-action value function through residual functions. ResQ masks some state-action value pairs from a joint state-action value function, which is transformed as the sum of a main function and a residual function. ResQ can be used with mean-value and stochastic-value RL. We theoretically show that ResQ can satisfy both the individual global max (IGM) and the distributional IGM principle without representation limitations. Through experiments on matrix games, the predator-prey, and StarCraft benchmarks, we show that ResQ can obtain better results than multiple expected/stochastic value factorization methods.

YNIMG Journal 2021 Journal Article

Brain responses to drug cues predict craving changes in abstinent heroin users: A preliminary study

• Shuang Liu
• Shicong Wang
• Min Zhang
• Yan Xu
• Ziqiang Shao
• Longmao Chen
• Wenhan Yang
• Jun Liu

BACKGROUND: Loss of control over drug intake occurring in drug addiction is believed to result from disruption of reward circuits, including reduced responsiveness to natural rewards (e.g., monetary, sex) and heightened responsiveness to drug reward. Yet few studies have assessed reward deficiency and related brain responses in abstinent heroin users with opioid use disorder, and less is known whether the brain responses can predict cue-induced craving changes following by prolonged abstinence. METHOD: 31 heroin users (age: 44.13±7.68 years, male: 18 (58%), duration of abstinence: 85.2 ± 52.5 days) were enrolled at a mandatory detoxification center. By employing a cue-reactivity paradigm including three types of cues (drug, sexual, neutral), brain regional activations and circuit-level functional coupling were extracted. Among the 31 heroin users, 15 were followed up longitudinally to assess cue induced craving changes in the ensuing 6 months. RESULTS: One way analysis of variance results showed that heroin users have differential brain activations to the three cues (neutral, drug and sexual) in the left dorsolateral prefrontal cortex (DLPFC), insula, orbiotofrontal cortex (OFC) and the bilateral thalamus. Drug cue induced greater activations in left DLPFC, insula and OFC compared to sexual cue. The psychophysiological interactions (PPI) analysis revealed negative couplings of the left DLPFC and the left OFC, bilateral thalamus, putamen in heroin users during drug cue exposure. In the 6-month follow-up study, both drug cue induced activation of the left DLPFC and the functional coupling of the left DLPFC-bilateral thalamus at baseline was correlated with craving reductions, which were not found for sexual cues. CONCLUSION: Our preliminary study provided novel evidence for the reward deficiency theory of opioid use disorder. Our findings also have clinical implications, as drug cue induced activation of the left DLPFC and functional coupling of left DLPFC-bilateral thalamus may be potential neuroimaging markers for craving changes during prolonged abstinence. Evidently, the findings in the current preliminary study should be confirmed by large sample size in the future.

JBHI Journal 2021 Journal Article

Multiscale Attention Guided Network for COVID-19 Diagnosis Using Chest X-Ray Images

• Jingxiong Li
• Yaqi Wang
• Shuai Wang
• Jun Wang
• Jun Liu
• Qun Jin
• Lingling Sun

Coronavirus disease 2019 (COVID-19) is one of the most destructive pandemic after millennium, forcing the world to tackle a health crisis. Automated lung infections classification using chest X-ray (CXR) images could strengthen diagnostic capability when handling COVID-19. However, classifying COVID-19 from pneumonia cases using CXR image is a difficult task because of shared spatial characteristics, high feature variation and contrast diversity between cases. Moreover, massive data collection is impractical for a newly emerged disease, which limited the performance of data thirsty deep learning models. To address these challenges, Multiscale Attention Guided deep network with Soft Distance regularization ( MAG-SD ) is proposed to automatically classify COVID-19 from pneumonia CXR images. In MAG-SD, MA-Net is used to produce prediction vector and attention from multiscale feature maps. To improve the robustness of trained model and relieve the shortage of training data, attention guided augmentations along with a soft distance regularization are posed, which aims at generating meaningful augmentations and reduce noise. Our multiscale attention model achieves better classification performance on our pneumonia CXR image dataset. Plentiful experiments are proposed for MAG-SD which demonstrates its unique advantage in pneumonia classification over cutting-edge models. The code is available at https://github.com/JasonLeeGHub/MAG-SD.

EAAI Journal 2020 Journal Article

A heuristic algorithm combining Pareto optimization and niche technology for multi-objective unequal area facility layout problem

• Jingfa Liu
• Jun Liu
• Xueming Yan
• Bitao Peng

The unequal area facility layout problem (UA-FLP) is the problem of placing departments with different areas in a facility so that departments satisfy some given objectives and constraints. In this paper, two objectives including the material handling cost and the closeness rating are optimized. Based on the quasi-physical strategy, we introduce an extrusive elastic potential energy based on the overlapping distance between departments into the layout system. After a novel handling approach of the non-overlapping constraint formed by executing the gradient method with an adaptive step length and subsequent department deformation strategy is developed to deal with the interference among departments and between any department and the facility, the problem is first converted into an optimization problem without the non-overlapping constraint. Then, we use a new heuristic algorithm that combines the local search based on the Pareto optimization and the global optimum search based on the niche technology to obtain Pareto-optimal solutions of the problem. In the proposed heuristic algorithm, in order to overcome the shortcomings of low efficient search toward the diversity of solutions in classical Pareto optimization method, we propose a heuristic layout updating strategy and a niche technology. To improve the convergence of the algorithm to the Pareto front, a mechanism of evolution of population named a feasible layout bank in the algorithm based on the local search and global optimum search is proposed. Two sets of representative instances from the literature with the size of the problem up to 62 departments are tested. The experimental results show that the proposed heuristic algorithm is an effective method for solving the multi-objective UA-FLP.

AAAI Conference 2020 Conference Paper

D-SPIDER-SFO: A Decentralized Optimization Algorithm with Faster Convergence Rate for Nonconvex Problems

• Taoxing Pan
• Jun Liu
• Jie Wang

Decentralized optimization algorithms have attracted intensive interests recently, as it has a balanced communication pattern, especially when solving large-scale machine learning problems. Stochastic Path Integrated Differential Estimator Stochastic First-Order method (SPIDER-SFO) nearly achieves the algorithmic lower bound in certain regimes for nonconvex problems. However, whether we can find a decentralized algorithm which achieves a similar convergence rate to SPIDER-SFO is still unclear. To tackle this problem, we propose a decentralized variant of SPIDER-SFO, called decentralized SPIDER-SFO (D-SPIDER-SFO). We show that D-SPIDER-SFO achieves a similar gradient computation cost—that is, O( −3 ) for finding an -approximate first-order stationary point—to its centralized counterpart. To the best of our knowledge, D-SPIDER-SFO achieves the state-of-the-art performance for solving nonconvex optimization problems on decentralized networks in terms of the computational cost. Experiments on different network configurations demonstrate the efficiency of the proposed method.

IJCAI Conference 2020 Conference Paper

Disentangled Feature Learning Network for Vehicle Re-Identification

• Yan Bai
• Yihang Lou
• Yongxing Dai
• Jun Liu
• Ziqian Chen
• Ling-Yu Duan

Vehicle Re-Identification (ReID) has attracted lots of research efforts due to its great significance to the public security. In vehicle ReID, we aim to learn features that are powerful in discriminating subtle differences between vehicles which are visually similar, and also robust against different orientations of the same vehicle. However, these two characteristics are hard to be encapsulated into a single feature representation simultaneously with unified supervision. Here we propose a Disentangled Feature Learning Network (DFLNet) to learn orientation specific and common features concurrently, which are discriminative at details and invariant to orientations, respectively. Moreover, to effectively use these two types of features for ReID, we further design a feature metric alignment scheme to ensure the consistency of the metric scales. The experiments show the effectiveness of our method that achieves state-of-the-art performance on three challenging datasets.

IJCAI Conference 2020 Conference Paper

pbSGD: Powered Stochastic Gradient Descent Methods for Accelerated Non-Convex Optimization

• Beitong Zhou
• Jun Liu
• Weigao Sun
• Ruijuan Chen
• Claire Tomlin
• Ye Yuan

We propose a novel technique for improving the stochastic gradient descent (SGD) method to train deep networks, which we term pbSGD. The proposed pbSGD method simply raises the stochastic gradient to a certain power elementwise during iterations and introduces only one additional parameter, namely, the power exponent (when it equals to 1, pbSGD reduces to SGD). We further propose pbSGD with momentum, which we term pbSGDM. The main results of this paper present comprehensive experiments on popular deep learning models and benchmark datasets. Empirical results show that the proposed pbSGD and pbSGDM obtain faster initial training speed than adaptive gradient methods, comparable generalization ability with SGD, and improved robustness to hyper-parameter selection and vanishing gradients. pbSGD is essentially a gradient modifier via a nonlinear transformation. As such, it is orthogonal and complementary to other techniques for accelerating gradient-based optimization such as learning rate schedules. Finally, we show convergence rate analysis for both pbSGD and pbSGDM methods. The theoretical rates of convergence match the best known theoretical rates of convergence for SGD and SGDM methods on nonconvex functions.

YNICL Journal 2020 Journal Article

Prediction of clinical and biomarker conformed Alzheimer’s disease and mild cognitive impairment from multi-feature brain structural MRI using age-correction from a large independent lifespan sample

• Binyin Li
• Miao Zhang
• Joost Riphagen
• Kathryn Morrison Yochim
• Biao Li
• Jun Liu
• David H. Salat

Structural neuroimaging has been applied to the identification of individuals with Alzheimer's disease (AD) and mild cognitive impairment (MCI). However, these methods are greatly impacted by age limiting their utility for detection of preclinical pathology. We built linear models for age based on multiple combined structural features using a large independent lifespan sample of 272 healthy adults across a wide age range from the Human Connectome Project Aging study. These models were then used to create a new support vector machine (SVM) training model in 136 AD and 268 control participants based on residues of fit from the expected age-effects relationship. Subsequent validation assessed the accuracy of the SVM model in new datasets. Finally, we applied the classifier to 276 individuals with MCI to evaluate prediction for early impairment and longitudinal cognitive change. The optimal 10-fold cross-validation accuracy was 93.07%, compared to 91.83% without age detrending. In the validation dataset, the classifier for AD obtained an accuracy of 84.85% (56/66), sensitivity of 85.36% (35/41) and specificity of 84% (21/25). Classification accuracy was improved when using the lifespan sample as opposed to the classification sample. Importantly, we observed cross-sectional greater AD specific biomarkers, as well as faster cognitive decline in MCI who were classified as more 'AD-like' (MCI-AD), and these effects were pronounced in individuals who were late MCI. The top five contributive features were volumes of left hippocampus, right hippocampus, left amygdala, the thickness of left and right middle temporal & parahippocampus gyrus. Linear detrending for age in SVM for combined structural features resulted in good performance for recognition of AD and AD-specific biomarkers, as well as prediction of MCI progression. Such procedures may be used in future work to enhance prediction in samples with atypical age distributions.

JBHI Journal 2019 Journal Article

Automated Layer Segmentation of Retinal Optical Coherence Tomography Images Using a Deep Feature Enhanced Structured Random Forests Classifier

• Xiaoming Liu
• Tianyu Fu
• Zhifang Pan
• Dong Liu
• Wei Hu
• Jun Liu
• Kai Zhang

Optical coherence tomography (OCT) is a high-resolution and noninvasive imaging modality that has become one of the most prevalent techniques for ophthalmic diagnosis. Retinal layer segmentation is very crucial for doctors to diagnose and study retinal diseases. However, manual segmentation is often a time-consuming and subjective process. In this work, we propose a new method for automatically segmenting retinal OCT images, which integrates deep features and hand-designed features to train a structured random forests classifier. The deep convolutional features are learned from deep residual network. With the trained classifier, we can get the contour probability graph of each layer; finally, the shortest path is employed to achieve the final layer segmentation. The experimental results show that our method achieves good results with the mean layer contour error of 1. 215 pixels, whereas that of the state of the art was 1. 464 pixels, and achieves an F1-score of 0. 885, which is also better than 0. 863 that is obtained by the state of the art method.

YNICL Journal 2019 Journal Article

Changes in default mode network connectivity in different glucose metabolism status and diabetes duration

• Huanghui Liu
• Jun Liu
• Limin Peng
• Zhichao Feng
• Lu Cao
• Huasheng Liu
• Hui Shen
• Dewen Hu

AIMS/HYPOTHESES: It is now generally accepted that diabetes increases the risk for cognitive impairment, but the precise mechanisms are poorly understood. In recent years, resting-state functional magnetic resonance imaging (rs-fMRI) is increasingly used to investigate the neural basis of cognitive dysfunction in type 2 diabetes (T2D) patients. Alterations in brain functional connectivity may underlie diabetes-related cognitive dysfunction and brain damage. The aim of this study was to investigate the changes in default mode network (DMN) connectivity in different glucose metabolism status and diabetes duration. METHODS: We used a seed-based fMRI analysis to investigate positive and negative DMN connectivity in four groups (39 subjects with normal glucose metabolism [NGM], 23 subjects with impaired glucose metabolism [IGM; i.e., prediabetes], 59 T2D patients with a diabetes duration of <10 years, and 24 T2D patients with a diabetes duration of ≥10 years). RESULTS: Negative DMN connectivity increased and then regressed with deteriorating glucose metabolism status and extending diabetes duration. DMN connectivity showed a significant correlation with diabetes duration. CONCLUSION/INTERPRETATION: This study suggests that DMN connectivity may exhibit distinct patterns in different glucose metabolism status and diabetes duration, providing some potential neuroimaging evidence for early diagnosis and further understanding of the pathophysiological mechanisms of diabetic brain damage.

AAAI Conference 2019 Conference Paper

Connecting Language to Images: A Progressive Attention-Guided Network for Simultaneous Image Captioning and Language Grounding

• Lingyun Song
• Jun Liu
• Buyue Qian
• Yihe Chen

Image captioning and visual language grounding are two important tasks for image understanding, but are seldom considered together. In this paper, we propose a Progressive Attention-Guided Network (PAGNet), which simultaneously generates image captions and predicts bounding boxes for caption words. PAGNet mainly has two distinctive properties: i) It can progressively refine the predictive results of image captioning, by updating the attention map with the predicted bounding boxes. ii) It learns bounding boxes of the words using a weakly supervised strategy, which combines the frameworks of Multiple Instance Learning (MIL) and Markov Decision Process (MDP). By using the attention map generated in the captioning process, PAGNet significantly reduces the search space of the MDP. We conduct experiments on benchmark datasets to demonstrate the effectiveness of PAGNet and results show that PAGNet achieves the best performance.

ICML Conference 2019 Conference Paper

Greedy Orthogonal Pivoting Algorithm for Non-Negative Matrix Factorization

• Kai Zhang
• Sheng Zhang
• Jun Liu
• Jun Wang
• Jie Zhang

Non-negative matrix factorization is a powerful tool for learning useful representations in the data and has been widely applied in many problems such as data mining and signal processing. Orthogonal NMF, which can improve the locality of decomposition, has drawn considerable interest in solving clustering problems in recent years. However, imposing simultaneous non-negative and orthogonal structure can be quite difficult, and so existing algorithms can only solve it approximately. To address this challenge, we propose an innovative procedure called Greedy Orthogonal Pivoting Algorithm (GOPA). The GOPA algorithm fully exploits the sparsity of non-negative orthogonal solutions to break the global problem into a series of local optimizations, in which an adaptive subset of coordinates are updated in a greedy, closed-form manner. The biggest advantage of GOPA is that it promotes exact orthogonality and provides solid empirical evidence that stronger orthogonality does contribute favorably to better clustering performance. On the other hand, we further design randomized and parallel version of GOPA, which can further reduce the computational cost and improve accuracy, making it suitable for large data.

YNICL Journal 2017 Journal Article

Early cortical biomarkers of longitudinal transcutaneous vagus nerve stimulation treatment success in depression

• Jiliang Fang
• Natalia Egorova
• Peijing Rong
• Jun Liu
• Yang Hong
• Yangyang Fan
• Xiaoling Wang
• Honghong Wang

Transcutaneous vagus nerve stimulation (tVNS), a non-invasive method of brain stimulation through the auricular branch of the vagus nerve, has shown promising results in treating major depressive disorder (MDD) in several pilot studies. However, the neural mechanism by which the effect on depression might be achieved has not been fully investigated, with only a few neuroimaging studies demonstrating tVNS-induced changes in the brains of healthy volunteers. Identifying specific neural pathways, which are influenced by tVNS compared with sham in depressed individuals, as well as determining neurobiomarkers of tVNS treatment success are needed to advance the application of tVNS for MDD. In order to address these questions, we measured fMRI brain activity of thirty-eight depressed patients assigned to undergo tVNS (n =17) or sham (n =21) treatment for 4weeks, during the first stimulation session. The results showed significant fMRI signal increases in the left anterior insula, revealed by a direct comparison of tVNS and sham stimulation. Importantly, the insula activation level during the first stimulation session in the tVNS group was significantly associated with the clinical improvement at the end of the four-week treatment, as indicated by the Hamilton Depression Rating Scale (HAM-D) score. Our findings suggest that anterior insula fMRI activity could serve as a potential cortical biomarker and an early predictor of tVNS longitudinal treatment success.

AIJ Journal 2017 Journal Article

Low-rank decomposition meets kernel learning: A generalized Nyström method

• Liang Lan
• Kai Zhang
• Hancheng Ge
• Wei Cheng
• Jun Liu
• Andreas Rauber
• Xiao-li Li
• Jun Wang

Low-rank matrix decomposition and kernel learning are two useful techniques in building advanced learning systems. Low-rank decomposition can greatly reduce the computational cost of manipulating large kernel matrices. However, existing approaches are mostly unsupervised and do not incorporate side information such as class labels, making the decomposition less effective for a specific learning task. On the other hand, kernel learning techniques aim at constructing kernel matrices whose structure is well aligned with the learning target, which improves the generalization performance of kernel methods. However, most kernel learning approaches are computationally very expensive. To obtain the advantages of both techniques and address their limitations, in this paper we propose a novel kernel low-rank decomposition formulation called the generalized Nyström method. Our approach inherits the linear time and space complexity via matrix decomposition, while at the same time fully exploits (partial) label information in computing task-dependent decomposition. In addition, the resultant low-rank factors can generalize to arbitrary new samples, rendering great flexibility in inductive learning scenarios. We further extend the algorithm to a multiple kernel learning setup. The experimental results on semi-supervised classification demonstrate the usefulness of the proposed method.

IS Journal 2015 Journal Article

Knowledge Engineering with Big Data

• Xindong Wu
• Huanhuan Chen
• Gongqing Wu
• Jun Liu
• Qinghua Zheng
• Xiaofeng He
• Aoying Zhou
• Zhong-Qiu Zhao

In the era of big data, knowledge engineering faces fundamental challenges induced by fragmented knowledge from heterogeneous, autonomous sources with complex and evolving relationships. The knowledge representation, acquisition, and inference techniques developed in the 1970s and 1980s, driven by research and development of expert systems, must be updated to cope with both fragmented knowledge from multiple sources in the big data revolution and in-depth knowledge from domain experts. This article presents BigKE, a knowledge engineering framework that handles fragmented knowledge modeling and online learning from multiple information sources, nonlinear fusion on fragmented knowledge, and automated demand-driven knowledge navigation.

NeurIPS Conference 2014 Conference Paper

A Safe Screening Rule for Sparse Logistic Regression

• Jie Wang
• Jiayu Zhou
• Jun Liu
• Peter Wonka
• Jieping Ye

The l1-regularized logistic regression (or sparse logistic regression) is a widely used method for simultaneous classification and feature selection. Although many recent efforts have been devoted to its efficient implementation, its application to high dimensional data still poses significant challenges. In this paper, we present a fast and effective sparse logistic regression screening rule (Slores) to identify the zero components in the solution vector, which may lead to a substantial reduction in the number of features to be entered to the optimization. An appealing feature of Slores is that the data set needs to be scanned only once to run the screening and its computational cost is negligible compared to that of solving the sparse logistic regression problem. Moreover, Slores is independent of solvers for sparse logistic regression, thus Slores can be integrated with any existing solver to improve the efficiency. We have evaluated Slores using high-dimensional data sets from different applications. Extensive experimental results demonstrate that Slores outperforms the existing state-of-the-art screening rules and the efficiency of solving sparse logistic regression is improved by one magnitude in general.

YNIMG Journal 2013 Journal Article

Cerebellum and integration of neural networks in dual-task processing

• Tao Wu
• Jun Liu
• Mark Hallett
• Zheng Zheng
• Piu Chan

Performing two tasks simultaneously (dual-task) is common in human daily life. The neural correlates of dual-task processing remain unclear. In the current study, we used a dual motor and counting task with functional MRI (fMRI) to determine whether there are any areas additionally activated for dual-task performance. Moreover, we investigated the functional connectivity of these added activated areas, as well as the training effect on brain activity and connectivity. We found that the right cerebellar vermis, left lobule V of the cerebellar anterior lobe and precuneus are additionally activated for this type of dual-tasking. These cerebellar regions had functional connectivity with extensive motor- and cognitive-related regions. Dual-task training induced less activation in several areas, but increased the functional connectivity between these cerebellar regions and numbers of motor- and cognitive-related areas. Our findings demonstrate that some regions within the cerebellum can be additionally activated with dual-task performance. Their role in dual motor and cognitive task processes is likely to integrate motor and cognitive networks, and may be involved in adjusting these networks to be more efficient in order to perform dual-tasking properly. The connectivity of the precuneus differs from the cerebellar regions. A possible role of the precuneus in dual-tasks may be to monitor the operation of active brain networks.

YNIMG Journal 2013 Journal Article

Modeling disease progression via multi-task learning

• Jiayu Zhou
• Jun Liu
• Vaibhav A. Narayan
• Jieping Ye

Alzheimer's disease (AD), the most common type of dementia, is a severe neurodegenerative disorder. Identifying biomarkers that can track the progress of the disease has recently received increasing attentions in AD research. An accurate prediction of disease progression would facilitate optimal decision-making for clinicians and patients. A definitive diagnosis of AD requires autopsy confirmation, thus many clinical/cognitive measures including Mini Mental State Examination (MMSE) and Alzheimer's Disease Assessment Scale cognitive subscale (ADAS-Cog) have been designed to evaluate the cognitive status of the patients and used as important criteria for clinical diagnosis of probable AD. In this paper, we consider the problem of predicting disease progression measured by the cognitive scores and selecting biomarkers predictive of the progression. Specifically, we formulate the prediction problem as a multi-task regression problem by considering the prediction at each time point as a task and propose two novel multi-task learning formulations. We have performed extensive experiments using data from the Alzheimer's Disease Neuroimaging Initiative (ADNI). Specifically, we use the baseline MRI features to predict MMSE/ADAS-Cog scores in the next 4years. Results demonstrate the effectiveness of the proposed multi-task learning formulations for disease progression in comparison with single-task learning algorithms including ridge regression and Lasso. We also perform longitudinal stability selection to identify and analyze the temporal patterns of biomarkers in disease progression. We observe that cortical thickness average of left middle temporal, cortical thickness average of left and right Entorhinal, and white matter volume of left Hippocampus play significant roles in predicting ADAS-Cog at all time points. We also observe that several MRI biomarkers provide significant information for predicting MMSE scores for the first 2years, however very few are shown to be significant in predicting MMSE score at later stages. The lack of predictable MRI biomarkers in later stages may contribute to the lower prediction performance of MMSE than that of ADAS-Cog in our study and other related studies.

YNIMG Journal 2012 Journal Article

MRI hippocampal and entorhinal cortex mapping in predicting conversion to Alzheimer's disease

• D.P. Devanand
• Ravi Bansal
• Jun Liu
• Xuejun Hao
• Gnanavalli Pradhaban
• Bradley S. Peterson

Objective Using MRI surface morphometry mapping, to evaluate local deformations of the hippocampus, parahippocampal gyrus, and entorhinal cortex in predicting conversion from mild cognitive impairment (MCI) to Alzheimer's disease (AD). Methods Baseline brain MRI with surface morphological analysis was performed in 130 outpatients with MCI, broadly defined, and 61 healthy controls followed for an average of 4years in a single site study. Results Patients with MCI differed from controls in several regions of the hippocampus and entorhinal cortex, and to a lesser extent in the parahippocampal gyrus. In the MCI sample, Cox regression models were conducted for time to conversion comparing converters to AD (n=31) and non-converters (n=99), controlling for age, sex and education. Converters showed greater atrophy in the head of the hippocampus, predominantly in the CA1 region and subiculum, and in the entorhinal cortex, especially in the anterior–inferior pole bilaterally. When distances of specific points representing localized inward deformation were entered together with the corresponding hippocampal or entorhinal cortex volume in the same Cox regression model, the distances remained highly significant whereas the volumes of the corresponding structures were either marginally significant or not significant. Inclusion of cognitive or memory measures or apolipoprotein E ε4 genotype as covariates, or restricting the sample to patients with amnestic MCI (24 converters and 81 non-converters) did not materially change the findings. In the 3-year follow-up sample of patients with MCI, logistic regression analyses using the same measures and covariates yielded similar results. Interpretation These findings indicate selective early involvement of the CA1 and subiculum regions of the hippocampus and provide new information on early anterior pole involvement in the entorhinal cortex in incipient AD. Fine-grained surface morphometry of medial temporal lobe structures may be superior to volumetric assessment in predicting conversion to AD in patients clinically diagnosed with MCI.

NeurIPS Conference 2011 Conference Paper

Efficient Methods for Overlapping Group Lasso

• Lei Yuan
• Jun Liu
• Jieping Ye

The group Lasso is an extension of the Lasso for feature selection on (predefined) non-overlapping groups of features. The non-overlapping group structure limits its applicability in practice. There have been several recent attempts to study a more general formulation, where groups of features are given, potentially with overlaps between the groups. The resulting optimization is, however, much more challenging to solve due to the group overlaps. In this paper, we consider the efficient optimization of the overlapping group Lasso penalized problem. We reveal several key properties of the proximal operator associated with the overlapping group Lasso, and compute the proximal operator by solving the smooth and convex dual problem, which allows the use of the gradient descent type of algorithms for the optimization. We have performed empirical evaluations using both synthetic and the breast cancer gene expression data set, which consists of 8, 141 genes organized into (overlapping) gene sets. Experimental results show that the proposed algorithm is more efficient than existing state-of-the-art algorithms.

NeurIPS Conference 2011 Conference Paper

Projection onto A Nonnegative Max-Heap

• Jun Liu
• Liang Sun
• Jieping Ye

We consider the problem of computing the Euclidean projection of a vector of length $p$ onto a non-negative max-heap---an ordered tree where the values of the nodes are all nonnegative and the value of any parent node is no less than the value(s) of its child node(s). This Euclidean projection plays a building block role in the optimization problem with a non-negative max-heap constraint. Such a constraint is desirable when the features follow an ordered tree structure, that is, a given feature is selected for the given regression/classification task only if its parent node is selected. In this paper, we show that such Euclidean projection problem admits an analytical solution and we develop a top-down algorithm where the key operation is to find the so-called \emph{maximal root-tree} of the subtree rooted at each node. A naive approach for finding the maximal root-tree is to enumerate all the possible root-trees, which, however, does not scale well. We reveal several important properties of the maximal root-tree, based on which we design a bottom-up algorithm with merge for efficiently finding the maximal root-tree. The proposed algorithm has a (worst-case) linear time complexity for a sequential list, and $O(p^2)$ for a general tree. We report simulation results showing the effectiveness of the max-heap for regression with an ordered tree structure. Empirical results show that the proposed algorithm has an expected linear time complexity for many special cases including a sequential list, a full binary tree, and a tree with depth 1.

EAAI Journal 2011 Journal Article

resolution method for a lattice-valued first-order logic

• Xingxing He
• Yang Xu
• Jun Liu
• Da Ruan

This paper focuses on resolution-based automated reasoning approaches in a lattice-valued first-order logic LF(X) with truth-values defined in a logical algebraic structure—lattice implication algebra (LIA), which aims at providing the logic foundation to represent and handle both imprecision and incomparability. In order to improve the efficiency of α - resolution approach proposed for LF(X), firstly the concepts of α - lock resolution principle and deduction are introduced for lattice-valued propositional logic LP(X) based on LIA, along with its soundness and weak completeness theorems. Then all the results are extended into LF(X) by using Lifting Lemma. Finally an α - lock resolution automated reasoning algorithm in LF(X) is proposed for the implementation purpose. This work provides a theoretical foundation for more efficient resolution-based automated reasoning algorithm in lattice-valued logic LF(X).

AAAI Conference 2010 Conference Paper

Learning to Surface Deep Web Content

• Zhaohui Wu
• Lu Jiang
• Qinghua Zheng
• Jun Liu

We propose a novel deep web crawling framework based on reinforcement learning. The crawler is regarded as an agent and deep web database as the environment. The agent perceives its current state and submits a selected action (query) to the environment according to Q-value. Based on the framework we develop an adaptive crawling method. Experimental results show that it outperforms the state of art methods in crawling capability and breaks through the assumption of full-text search implied by existing methods.

NeurIPS Conference 2010 Conference Paper

Moreau-Yosida Regularization for Grouped Tree Structure Learning

• Jun Liu
• Jieping Ye

We consider the tree structured group Lasso where the structure over the features can be represented as a tree with leaf nodes as features and internal nodes as clusters of the features. The structured regularization with a pre-defined tree structure is based on a group-Lasso penalty, where one group is defined for each node in the tree. Such a regularization can help uncover the structured sparsity, which is desirable for applications with some meaningful tree structures on the features. However, the tree structured group Lasso is challenging to solve due to the complex regularization. In this paper, we develop an efficient algorithm for the tree structured group Lasso. One of the key steps in the proposed algorithm is to solve the Moreau-Yosida regularization associated with the grouped tree structure. The main technical contributions of this paper include (1) we show that the associated Moreau-Yosida regularization admits an analytical solution, and (2) we develop an efficient algorithm for determining the effective interval for the regularization parameter. Our experimental results on the AR and JAFFE face data sets demonstrate the efficiency and effectiveness of the proposed algorithm.

IJCAI Conference 2009 Conference Paper

• Jun Liu
• Jianhui Chen
• Songcan Chen
• Jieping Ye

Kernel methods have been applied successfully in many applications. The kernel matrix plays an important role in kernel-based learning methods, but the “ideal” kernel matrix is usually unknown in practice and needs to be estimated. In this paper, we propose to directly learn the “ideal” kernel matrix (called the optimal neighborhood kernel matrix) from a pre-specified kernel matrix for improved classification performance. We assume that the prespecified kernel matrix generated from the specific application is a noisy observation of the ideal one. The resulting optimal neighborhood kernel matrix is shown to be the summation of the pre-specified kernel matrix and a rank-one matrix. We formulate the problem of learning the optimal neighborhood kernel as a constrained quartic problem, and propose to solve it using two methods: level method and constrained gradient descent. Empirical results on several benchmark data sets demonstrate the efficiency and effectiveness of the proposed algorithms.

NeurIPS Conference 2009 Conference Paper

Efficient Recovery of Jointly Sparse Vectors

• Liang Sun
• Jun Liu
• Jianhui Chen
• Jieping Ye

We consider the reconstruction of sparse signals in the multiple measurement vector (MMV) model, in which the signal, represented as a matrix, consists of a set of jointly sparse vectors. MMV is an extension of the single measurement vector (SMV) model employed in standard compressive sensing (CS). Recent theoretical studies focus on the convex relaxation of the MMV problem based on the $(2, 1)$-norm minimization, which is an extension of the well-known $1$-norm minimization employed in SMV. However, the resulting convex optimization problem in MMV is significantly much more difficult to solve than the one in SMV. Existing algorithms reformulate it as a second-order cone programming (SOCP) or semidefinite programming (SDP), which is computationally expensive to solve for problems of moderate size. In this paper, we propose a new (dual) reformulation of the convex optimization problem in MMV and develop an efficient algorithm based on the prox-method. Interestingly, our theoretical analysis reveals the close connection between the proposed reformulation and multiple kernel learning. Our simulation studies demonstrate the scalability of the proposed algorithm.

NeurIPS Conference 2009 Conference Paper

Learning Brain Connectivity of Alzheimer's Disease from Neuroimaging Data

• Shuai Huang
• Jing Li
• Liang Sun
• Jun Liu
• Teresa Wu
• Kewei Chen
• Adam Fleisher
• Eric Reiman

Recent advances in neuroimaging techniques provide great potentials for effective diagnosis of Alzheimer’s disease (AD), the most common form of dementia. Previous studies have shown that AD is closely related to alternation in the functional brain network, i. e. , the functional connectivity among different brain regions. In this paper, we consider the problem of learning functional brain connectivity from neuroimaging, which holds great promise for identifying image-based markers used to distinguish Normal Controls (NC), patients with Mild Cognitive Impairment (MCI), and patients with AD. More specifically, we study sparse inverse covariance estimation (SICE), also known as exploratory Gaussian graphical models, for brain connectivity modeling. In particular, we apply SICE to learn and analyze functional brain connectivity patterns from different subject groups, based on a key property of SICE, called the “monotone property” we established in this paper. Our experimental results on neuroimaging PET data of 42 AD, 116 MCI, and 67 NC subjects reveal several interesting connectivity patterns consistent with literature findings, and also some new patterns that can help the knowledge discovery of AD.