Li Zhang

AAAI Conference 2026 Conference Paper

Collaboratively “Copy & Paste” 2D-3D Features for Complex Video-to-Video Motion Editing

• Jia-Xing Zhong
• Shijie Zhao
• Junlin Li
• Li Zhang

Video-to-video human motion editing aims to transfer motion from a driving video to a reference video while preserving the background dynamics and the protagonist's original appearance. We identify critical limitations in existing methods that fail to capture the full complexity of human motions, particularly regarding: 1) location changes, 2) orientation variations, and 3) complicated non-upright poses. To address these challenges, we propose a framework that collaboratively "copies and pastes" 2D and 3D features across spatio-temporal dimensions into a shared representation space for motion guidance. Our approach achieves this through: 1) a mutual distillation mechanism that enhances the robustness and capability of individual encoders, and 2) a selective fusion module that adaptively weights and combines complementary information from spatio-temporal representations. To evaluate motion editing algorithms under challenging scenarios, we introduce a comprehensive benchmark dataset comprising real-world video clips from artistic gymnastics and figure skating competitions. These sports disciplines naturally encompass the three aforementioned aspects of motion complexity. Extensive experiments demonstrate that our approach significantly outperforms existing methods, particularly in handling intricate human motions.

AAAI Conference 2026 Conference Paper

ELSPR: Evaluator LLM Training Data Self-Purification on Non-Transitive Preferences via Tournament Graph Reconstruction

• Yan Yu
• Yilun Liu
• Minggui He
• Shimin Tao
• Weibin Meng
• Xinhua Yang
• Li Zhang
• Hongxia Ma

Pairwise evaluation of large language models (LLMs) has become the dominant paradigm for benchmarking open-ended tasks, yet non-transitive preferences—where evaluators prefer A over B, B over C, but C over A—fundamentally undermine ranking reliability. We show that this critical issue stems largely from low-quality data that contains inherently ambiguous preference pairs. To address this challenge, we propose ELSPR, a principled graph-theoretic framework that models pairwise preferences as tournament graphs and systematically identifies problematic training data. ELSPR quantifies non-transitivity through strongly connected components (SCCs) analysis and measures overall preference clarity using a novel normalized directed graph structural entropy metric. Our filtering methodology selectively removes preference data that induce non-transitivity while preserving transitive preferences. Extensive experiments on the AlpacaEval benchmark demonstrate that models fine-tuned on ELSPR-filtered data achieve substantial improvements: a 13.8% reduction in non-transitivity, a 0.088 decrease in structural entropy, and significantly enhanced discriminative power in real-world evaluation systems. Human validation confirms that discarded data exhibit dramatically lower inter-annotator agreement (34.4% vs. 52.6%) and model-human consistency (51.2% vs. 80.6%) compared to cleaned data. These findings establish ELSPR as an effective data self-purification approach for developing more robust, consistent, and human-aligned LLM evaluation systems.

AAAI Conference 2026 Conference Paper

EvoFMVC: Trusted Federated Multi-View Clustering with Evolutionary Fusion

• Li Zhang
• Pinhan Fu
• Li Lv
• Qian Guo
• Liang Du
• Xinyan Liang

With the growing demand for decentralized collaborative analysis of privacy-sensitive data, federated multi-view clustering (FMVC) has attracted widespread attention due to its ability to balance privacy protection and collaborative modeling. However, current methods still face the following challenges: (1) Clients need to frequently upload high-dimensional data such as model parameters or graph structures, resulting in high communication costs; (2) The structured data uploaded often contains semantic features and has a high risk of being inverted; (3) The server usually merges the data from all clients with the fixed fusion rule, which may result in a suboptimized clustering result when there exist low-quality clients. To address the issues, we propose a new trusted federated multi-view clustering framework (EvoFMVC) that introduces three key innovations: First, lightweight trusted evidence serves as a compact communication medium, significantly reducing overhead compared to conventional model parameters or graph structures. Second, trusted evidences express clustering results in the form of probability distribution, which avoids the risk of structured information being easily inverted. Lastly, we formalize the server-side aggregation process as a neural architecture search (NAS) task where the server flexibly uses different fusion operators to filter and fuse necessary views through evolutionary algorithms, which significantly improves the fusion effect and model performance. Experimental results on multiple datasets show that our method is superior to existing FMVC methods in terms of clustering accuracy and communication efficiency.

AAAI Conference 2026 Conference Paper

Exploring Category-level Articulated Object Pose Tracking on SE(3) Manifolds

• Xianhui Meng
• Yukang Huo
• Li Zhang
• Liu Liu
• Haonan Jiang
• Yan Zhong
• Pingrui Zhang
• Cewu Lu

Articulated objects are prevalent in daily life and robotic manipulation tasks. However, compared to rigid objects, pose tracking for articulated objects remains an underexplored problem due to their inherent kinematic constraints. To address these challenges, this work proposes a novel point-pair-based pose tracking framework, termed PPF-Tracker. The proposed framework first performs quasi-canonicalization of point clouds in the SE(3) Lie group space, and then models articulated objects using Point Pair Features (PPF) to predict pose voting parameters by leveraging the invariance properties of SE(3). Finally, semantic information of joint axes is incorporated to impose unified kinematic constraints across all parts of the articulated object. PPF-Tracker is systematically evaluated on both synthetic datasets and real-world scenarios, demonstrating strong generalization across diverse and challenging environments. Experimental results highlight the effectiveness and robustness of PPF-Tracker in multi-frame pose tracking of articulated objects. We believe this work can foster advances in robotics, embodied intelligence, and augmented reality.

AAAI Conference 2026 Conference Paper

F2SST: Frequency-to-Spatial Semantic Transfer for Few-Shot Image Classification

• Xueyi Chen
• Bangjun Wang
• Jiaqing Fan
• Li Zhang
• Fanzhang Li

Few-shot image classification (FSIC) aims to recognize novel categories from only a few labeled examples, making it inherently challenging under limited supervision. Existing approaches have attempted to alleviate this issue by incorporating explicit semantics like class names or knowledge graphs to guide learning. However, such methods often encounter semantic ambiguity due to their dependence on either overly simplistic semantic priors or resource-intensive external knowledge sources, which limits their potential. In this paper, we explore the frequency domain as an implicit and task-adaptive source of semantic information. We propose F2SST, a Frequency-to-Spatial Semantic Transfer framework that enhances feature learning by leveraging spectral signals as hidden semantics. Specifically, F2SST applies Fast Fourier Transform (FFT) to extract phase-invariant global frequency descriptors, followed by a lightweight Gated Spectral Attention (GSA) module that selectively emphasizes class-relevant frequency components. These enhanced spectral cues are then integrated into the spatial stream through a class-guided fusion mechanism, enabling more robust and semantically aligned representations. Extensive experiments on four standard benchmarks (miniImageNet, tieredImageNet, CIFAR-FS and FC100) demonstrate that F2SST consistently improves performance, validating the effectiveness of frequency-domain semantics in FSIC.

AAAI Conference 2026 Conference Paper

MIDB: Multilingual Instruction Data Booster for Enhancing Cultural Equality in Multilingual Instruction Synthesis

• Yilun Liu
• Chunguang Zhao
• Xinhua Yang
• Hongyong Zeng
• Shimin Tao
• Weibin Meng
• Minggui He
• Yan Yu

Despite doubts on data quality, instruction synthesis has been widely applied into instruction tuning (IT) of LLMs as an economic and rapid alternative. Recent endeavors focus on improving data quality for synthesized instruction pairs in English and have facilitated IT of English-centric LLMs. However, data quality issues in multilingual synthesized instruction pairs are even more severe, since the common synthesizing practice is to translate English synthesized data into other languages using machine translation (MT). Besides the known content errors in these English synthesized data, multilingual synthesized instruction data are further exposed to defects introduced by MT and face insufficient localization of the target languages, leading to cultural inequality in trained LLMs. In this paper, we propose MIDB, a Multilingual Instruction Data Booster to automatically address the quality issues in multilingual synthesized data. MIDB is trained on around 36.8k revision examples across 16 languages by human linguistic experts, thereby can boost the low-quality data by addressing content errors and MT defects, and improving localization in these synthesized data. Both automatic and human evaluation indicate that not only MIDB steadily improved instruction data quality in 16 languages, but also the instruction-following and cultural-understanding abilities of multilingual LLMs fine-tuned on MIDB-boosted data were significantly enhanced, suggesting an improved linguistic and cultural equality.

EAAI Journal 2026 Journal Article

Multi-label feature selection using adaptive heterogeneous graph-based learning and self-adaptive evolutionary optimization with local refinement

• Xingming Liu
• Li Zhang
• Ning Pan

In many real-world applications, multi-label feature selection is a difficult challenge due to high-dimensional data, intricate feature–label interactions, and strong label correlations. In order to overcome these problems, this paper suggests a unique hybrid framework for efficient multi-label feature selection that combines self-adaptive evolutionary optimization, local refinement, and adaptive heterogeneous graph-based learning. In order to enable the dynamic learning of feature relevance and explicitly capture label dependencies via attention-driven relational learning, the suggested method builds an adaptive heterogeneous graph that simultaneously models instances, features, and labels. Without the need for human parameter tweaking, a self-adaptive evolutionary optimization method that balances global exploration and convergence stability is guided by the learnt graph-based representations. The whole architecture is made to be resilient to redundancy and noisy correlations while choosing feature subsets that are both compact and highly discriminative. Six benchmark multi-label datasets including bioinformatics, text analysis, picture annotation, and music emotion identification are used to assess the suggested approach. While using far fewer features, experimental findings show consistent gains over previous state-of-the-art approaches across a variety of assessment criteria, including Hamming Loss, One-Error, Ranking Loss, Average Precision, and Macro-F1. The combined contributions of adaptive graph modeling, self-adaptive evolutionary optimization, and local refinement are further supported by ablation investigations. This paper shows how to effectively reduce dimensionality for multi-label learning problems in high-dimensional domains by combining evolutionary optimization with graph-based learning to provide a tangible artificial intelligence implementation and its artificial intelligence application in real-world multi-label learning scenarios.

JBHI Journal 2026 Journal Article

MVTT-GMamba: A Multimodal Graph Reasoning Framework for Anti-VEGF Efficacy Prediction in Diabetic Macular Edema

• Shijing Wu
• Yifan Zheng
• Tianyu Chen
• Gang Wu
• Qinkun Zhi
• Ailing Sui
• Haixia Bai
• Junhui Shen

Accurately predicting individual responses to Anti-Vascular Endothelial Growth Factor (Anti-VEGF) efficacy in diabetic macular edema (DME) remains a critical challenge in personalized ophthalmic care. Existing methods often rely on unimodal data or suffer from ineffective multimodal feature extraction and fusion, leading to modality redundancy and performance degradation. To address these limitations, we propose MVTT-GMamba, a novel multimodal learning framework that integrates optical coherence tomography (OCT) images and structured clinical indicators for early and accurate Anti-VEGF efficacy prediction. At the core of MVTT-GMamba is a feature-wise heterogeneous graph reasoning paradigm that explicitly models inter-patient and inter-feature relations, together with an adaptive, graph-guided prediction head that progressively anneals structural priors into the classifier. Building on this core, we adopt domain-tailored MambaVision and TabTransformer encoders and an early cross-attention fusion module to realize fine-grained multimodal representation learning. Extensive experiments on both a private clinical dataset (DMETHERA-ECSAHZU) and the public APTOS2021 benchmark demonstrate that MVTT-GMamba consistently outperforms state-of-the-art methods across all evaluation metrics. In addition, Grad-CAM visualizations reveal that the model attends to clinically relevant retinal regions, providing enhanced interpretability. Code is available at: https://github.com/DME666/DME.

AAAI Conference 2026 Conference Paper

Perception in Plan: Coupled Perception and Planning for End-to-End Autonomous Driving

• Bozhou Zhang
• Jingyu Li
• Nan Song
• Li Zhang

End-to-end autonomous driving has achieved remarkable advancements in recent years. Existing methods primarily follow a perception–planning paradigm, where perception and planning are executed sequentially within a fully differentiable framework for planning-oriented optimization. We further advance this paradigm through a "perception-in-plan'' framework design, which integrates perception into the planning process. This design facilitates targeted perception guided by evolving planning objectives over time, ultimately enhancing planning performance. Building on this insight, we introduce VeteranAD, a coupled perception and planning framework for end-to-end autonomous driving. By incorporating multi-mode anchored trajectories as planning priors, the perception module is specifically designed to gather traffic elements along these trajectories, enabling comprehensive and targeted perception. Planning trajectories are then generated based on both the perception results and the planning priors. To make perception fully serve planning, we adopt an autoregressive strategy that progressively predicts future trajectories while focusing on relevant regions for targeted perception at each step. With this simple yet effective design, VeteranAD fully unleashes the potential of planning-oriented end-to-end methods, leading to more accurate and reliable driving behavior. Extensive experiments on the NAVSIM and Bench2Drive datasets demonstrate that our VeteranAD achieves state-of-the-art performance.

AAAI Conference 2026 Conference Paper

SIAM: Towards Generalizable Articulated Object Modeling via Single Robot-Object Interaction

• Yuyan Liu
• Li Zhang
• Di Wu
• Yan Zhang
• Anran Huang
• Zhi Wang
• Liu Liu
• Dan Guo

Articulated object modeling, which represents interconnected rigid bodies with their geometry, part segmentation, articulation tree, and physical properties, is crucial for robotic perception and manipulation. Recently existing methods like SAGCI leverage Interactive Perception (IP) to refine models through robot interaction. However, SAGCI suffers from prior-dependency (requiring initialization), neglects kinematic/dynamic constraints, and generates non-watertight meshes. To overcome these limitations, we propose SIAM, a novel framework for efficient and generalizable Single-Interaction Articulated Modeling. Given an initial point cloud, SIAM first enables minimal robot interaction to trigger object motion. It then precisely segments parts by analyzing point cloud differences pre- and post-interaction. For joint parameter estimation, we introduce an optimization incorporating novel kinematic energy constraints, enhancing physical consistency. Finally, we reconstruct a high-quality, topologically watertight mesh by learning 3D Gaussian Primitives from multi-view RGB-D observations under deformation. Extensive experiments on the PartNet-Mobility benchmark demonstrate state-of-the-art articulation modeling performance. Successful real-world deployment with an xArm robot further validates the framework's practicality and transferability. SIAM achieves accurate, prior-free modeling with significantly reduced interaction cost.

AAAI Conference 2026 Conference Paper

Spatio-Temporal Distortion Aware Omnidirectional Video Super-Resolution

• Hongyu An
• Xinfeng Zhang
• Shijie Zhao
• Li Zhang
• Ruiqin Xiong

Omnidirectional videos (ODVs) provide an immersive visual experience by capturing the 360° scene. With the rapid advancements in virtual/augmented reality, metaverse, and generative artificial intelligence, the demand for high-quality ODVs is surging. However, ODVs often suffer from low resolution due to their wide field of view and limitations in capturing devices and transmission bandwidth. Although video super-resolution (SR) is a capable video quality enhancement technique, the performance ceiling and practical generalization of existing methods are limited when applied to ODVs due to their unique attributes. To alleviate spatial projection distortions and temporal flickering of ODVs, we propose a Spatio-Temporal Distortion Aware Network (STDAN) with joint spatio-temporal alignment and reconstruction. Specifically, we incorporate a spatio-temporal continuous alignment (STCA) to mitigate discrete geometric artifacts in parallel with temporal alignment. Subsequently, we introduce an interlaced multi-frame reconstruction (IMFR) to enhance temporal consistency. Furthermore, we employ latitude-saliency adaptive (LSA) weights to focus on regions with higher texture complexity and human-watching interest. By exploring a spatio-temporal jointly framework and real-world viewing strategies, STDAN effectively reinforces spatio-temporal coherence on a novel ODV-SR dataset and ensures affordable computational costs. Extensive experimental results demonstrate that STDAN outperforms state-of-the-art methods in improving visual fidelity and dynamic smoothness of ODVs.

AAAI Conference 2026 Conference Paper

TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Models

• Li Zhang
• Zhongxuan Han
• XiaoHua Feng
• Jiaming Zhang
• Yuyuan Li
• Linbo Jiang
• Jianan Lin
• Chaochao Chen

Efficient and lightweight adaptation of pre-trained Vision-Language Models (VLMs) to downstream tasks through collaborative interactions between local clients and a central server is a rapidly emerging research topic in federated learning. Existing adaptation algorithms are typically trained iteratively, which incur significant communication costs and increase the susceptibility to potential attacks. Motivated by the one-shot federated training techniques that reduce client-server exchanges to a single round, developing a lightweight one-shot federated VLM adaptation method to alleviate these issues is particularly attractive. However, current one-shot approaches face certain challenges in adapting VLMs within federated settings: (1) insufficient exploitation of the rich multimodal information inherent in VLMs; (2) lack of specialized adaptation strategies to systematically handle the severe data heterogeneity; and (3) requiring additional training resource of clients or server. To bridge these gaps, we propose a novel Training-free One-shot Federated Adaptation framework for VLMs, named TOFA. To fully leverage the generalizable multimodal features in pre-trained VLMs, TOFA employs both visual and textual pipelines to extract task-relevant representations. In the visual pipeline, a hierarchical Bayesian model learns personalized, class-specific prototype distributions. For the textual pipeline, TOFA evaluates and globally aligns the generated local text prompts for robustness. An adaptive weight calibration mechanism is also introduced to combine predictions from both modalities, balancing personalization and robustness to handle data heterogeneity. Our method is training-free, not relying on additional training resources on either the client or server side. Extensive experiments across 9 datasets in various federated settings demonstrate the effectiveness of the proposed TOFA method.

AAAI Conference 2026 Conference Paper

Uncertainty-Guided View-Strength-Aware Feature Utilization for Multi-View Classification

• Li Lv
• Qian Guo
• Li Zhang
• Liang Du
• Bingbing Jiang
• Lu Chen
• Xinyan Liang

In multi-view classification tasks (MVC), each view provides an unique perspective on the data, offering complementary information that can improve classification performance when properly integrated. However, traditional methods typically adopt a uniform processing strategy for all views before fusion, overlooking the fact that different views may require different treatments due to variations in their quality and informativeness. To address this limitation, we propose a novel framework called Uncertainty-Guided View-Strength-Aware Feature Utilization (UVF) for multi-view classification. Our approach introduces a view uncertainty estimation module to quantify the discriminative strength of each view. Based on this estimation, a Differentiated Feature Selector (DFS) adaptively selects features, retaining informative dimensions in weak views while preserving original features in strong views. Furthermore, we employ an uncertainty-guided fusion strategy that assigns dynamic weights to each view's contribution based on its uncertainty score, enhancing the robustness and reliability of the final decision. Experimental results on benchmark datasets demonstrate that our method significantly outperforms conventional approaches, achieving better classification accuracy and interpretability through strength-aware feature processing and fusion.

AAAI Conference 2026 Conference Paper

VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learning

• Xuanyu Zhang
• Weiqi Li
• Shijie Zhao
• Junlin Li
• Li Zhang
• Jian Zhang

Recent advances in AI-generated content (AIGC) have led to the emergence of powerful text-to-video generation models. Despite these successes, evaluating the quality of AIGC-generated videos remains challenging due to limited generalization, lack of temporal awareness, heavy reliance on large-scale annotated datasets, and the lack of effective interaction with generation models. Most current approaches rely on supervised fine-tuning of vision-language models (VLMs), which often require large-scale annotated datasets and tend to decouple understanding and generation. To address these shortcomings, we propose VQ-Insight, a novel reasoning-style VLM framework for AIGC video quality assessment. Our approach features: (1) a progressive video quality learning scheme that combines image quality warm-up, general task-specific temporal learning, and joint optimization with the video generation model; (2) the design of multi-dimension scoring rewards, preference comparison rewards, and temporal modeling rewards to enhance both generalization and specialization in video quality evaluation. Extensive experiments demonstrate that VQ-Insight consistently outperforms state-of-the-art baselines in preference comparison, multi-dimension scoring, and natural video scoring, bringing significant improvements for video generation tasks.

NeurIPS Conference 2025 Conference Paper

4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration

• Jiahui Zhang
• Yurui Chen
• Yueming Xu
• Ze Huang
• Yanpeng Zhou
• Yu-Jie Yuan
• Xinyue Cai
• Guowei Huang

Leveraging diverse robotic data for pretraining remains a critical challenge. Existing methods typically model the dataset’s action distribution using simple observations as inputs. However, these inputs are often incomplete, resulting in a dispersed conditional action distribution—an issue we refer to as coordinate system chaos and state chaos. This inconsistency significantly hampers pretraining efficiency. To address this, we propose 4D-VLA, a novel approach that effectively integrates 4D information into the input to mitigate these sources of chaos. Our model introduces depth and temporal information into visual features with sequential RGB-D inputs, aligning the coordinate systems of the robot and the scene. This alignment endows the model with strong spatiotemporal reasoning capabilities while minimizing training overhead. Additionally, we introduce Memory bank sampling, a frame sampling strategy designed to extract informative frames from historical images, further improving effectiveness and efficiency. Experimental results demonstrate that our pretraining method and architectural components substantially enhance model performance. In both simulated and real-world experiments, our model achieves a significant increase in success rate over OpenVLA. To further assess spatial perception and generalization to novel views, we introduce MV-Bench, a multi-view simulation benchmark. Our model consistently outperforms existing methods, demonstrating stronger spatial understanding and adaptability.

EAAI Journal 2025 Journal Article

A dynamic graph convolutional network-based framework for the unsteady operating states recognition of multi-product pipeline systems

• Li Zhang
• Lin Fan
• Jianjun Liu
• Dingyu Jiao
• Yuxuan He
• Jing Zhou
• Karine Zeitouni
• Huai Su

Considering that the existing methods lack spatial and temporal information mining of pipeline multidimensional operation data, it is unable to accurately recognize the unsteady operation conditions among pipeline stations. In this study, a dynamic graph convolutional network classification model is proposed for the recognition of unsteady operating states in multi-product pipeline systems. Firstly, dynamic graph convolutional network of multi-pipeline system (DPipeNet) is constructed based on the visibility graph algorithm, mutual information and long and short-term memory network model. Secondly, static graph convolutional network of multi-pipeline system (SPipeNet) is constructed by using the real geographic location information of each station of multi-pipeline. Then, the input subgraph of the graph convolutional network is used to construct the multi-pipeline system operational state relationship network (OSRN), and the vulnerable state nodes of the system are evaluated using complex network centrality metrics. Finally, the proposed model is applied to real operational data of a multi-pipeline system in China. The results show that in the two-classification scenario, both DPipeNet and SPipeNet have higher accuracies, but DPipeNet has a lower missed rate. In the multi-classification scenario, DPipeNet has the highest precision, which can reach more than 85%, and the recall rate is improved by 13%–25% compared with the neural network models in recent literature and SPipeNet. In the vulnerability analysis scenario, the intermediate station pump startup/stoppage of multi-pipeline has higher vulnerability. The proposed method also provides decision support for managers in pipeline system operation and maintenance management.

EAAI Journal 2025 Journal Article

A Lightweight Vision Mamba Coding UNet for medical image segmentation

• Yuanyuan Li
• Yifei Duan
• Guanqiu Qi
• Baisen Cong
• Li Zhang
• Zhiqin Zhu

The segmentation of medical images is vital for advancing research in medicine and supporting precise clinical diagnoses. Over the past few years, neural network-based methods have become a major focus of research and have been broadly adopted in medical image segmentation. However, mainstream methods like Transformer have excessive computational cost requirements, which makes them impractical for mobile medical applications. Therefore, this paper proposes a Lightweight Vision Mamba Coding UNet (LVMC-UNet), which integrates the Rotation-based Vision Mamba module (RVM) and Correlation Space Fusion module (CSF) in a lightweight manner. The RVM module processes the input images in parallel and introduces rotary positional encoding (RPE) on the Vision Mamba architecture to address the limitation of the Mamba module’s insufficient ability to capture local details. The CSF module integrates multi-stage and multi-scale feature maps and introduces blueprint separable convolutions (BSConvs) to enhance intra-kernel correlation. Comprehensive experiments on three datasets demonstrate that LVMC-UNet outperforms current lightweight segmentation methods and strikes a balance between segmentation accuracy and computational requirements.

EAAI Journal 2025 Journal Article

Adaptive adversarial pattern contrast algorithm for black-box model and domain attack

• Weidong Wang
• Yi Wang
• Zhi Li
• Long Zheng
• Li Zhang

The transferability of adversarial attacks in deep neural networks (DNNs) is a significant challenge, especially for achieving effective attacks across models and data domains. Unfortunately, existing attack approaches primarily focus on cross-model transferability, often overlooking the potential for black-box attacks across diverse data domains. This paper proposes the Adaptive adversarial Pattern Contrast (APEC) algorithm, designed to achieve cross-model and domain adversarial attacks with high transferability. Firstly, APEC generates transferable adversarial examples by leveraging spatial characteristics such as regional homogeneity, repetition, and density, thereby increasing classifier misclassification rates. Secondly, a key innovation in APEC is the similarity contrast loss inspired by contrastive learning. It guides the model to learn discriminative adversarial features by aligning adversarial examples with adversarial patterns and distancing them from clean examples. Importantly, this optimization is performed label-free, enhancing APEC’s practicality in real-world black-box scenarios. Additionally, we introduce a Gaussian low-pass filter in APEC to generate adversarial perturbation patterns adaptively. This operation suppresses high-frequency information while preserving the low-frequency characteristics of natural examples, enhancing APEC’s attack capabilities. The APEC algorithm shows relative improvement across models and data domains compared to state-of-the-art transferability attacks. Our code is available at https: //github. com/cs-igps/APEC-TransferAttack.

YNIMG Journal 2025 Journal Article

Altered brain network dynamics during rumination in remitted depression

• Su Shu
• Wenwen Ou
• Mohan Ma
• Hairuo He
• Qianqian Zhang
• Mei Huang
• Wentao Chen
• Aoqian Deng

Rumination is a known risk factor for depression relapse. Understanding its neurobiological mechanisms during depression remission can inform strategies to prevent relapse, yet the temporal dynamics of brain networks during rumination in remitted depression remain unclear. Here, we collected rumination induction fMRI data from 42 patients with remitted depression and 41 healthy controls (HCs). Using an energy landscape approach, we investigated the temporal dynamics of brain networks during rumination. The appearance frequency (AF) and transition frequency (TF) metrics were defined to quantify the dynamic properties of brain states. Patients during remission showed higher levels of rumination than HCs. Both groups exhibited four brain states during rumination, which consisted of complementary network group activation (states 1 and 2, states 3 and 4). In patients, the AFs of and reciprocal TFs between states 1 and 2 during rumination were significantly increased, while AFs of states 3 and 4 and reciprocal TFs involving states 1-3, 1-4, 2-3, and 2-4 were decreased, both when compared to HCs and relative to patients themselves during distraction. Moreover, we found that for patients, the AF of state 1 was negatively correlated with rumination levels and marginally positively associated with attention, while the AF of state 2 was negatively associated with performance on attention tasks. Our study revealed altered dynamic characteristics of brain states composed of network groups during rumination in remitted depression. Additionally, the findings suggest that heightened self-focus linked to rumination may impair the brain's ability to efficiently allocate attentional resources.

I&C Journal 2025 Journal Article

An iterative scheme of hybrid controller synthesis for nonlinear systems subject to safety constraints

• Niuniu Qi
• Xia Zeng
• Banglong Liu
• Zhengfeng Yang
• Xiaochao Tang
• Li Zhang
• Chao Peng
• Zhenbing Zeng

For many safety-critical systems, there is a pressing need for learning the controllers to endow systems with properties of safety. Reinforcement learning (RL) has emerged as a promising approach to synthesizing controllers that satisfy safety requirements by leveraging user-defined reward functions to encode desired system behavior. However, it remains a significant challenge in synthesizing provably correct controllers with safety requirements. To address this issue, we try to design a special hybrid polynomial-NN controller that is easy to verify without losing its expressiveness and flexibility. This paper proposes an iterative framework to synthesize a hybrid controller based on RL, low-degree polynomial fitting and knowledge distillation. By formulating and solving a constrained optimization problem in which the verification conditions produce the barrier certificates, a computational method is given to ensure that every trajectory starting from the initial set of the system with the resulting controller satisfies the given safety requirement. In addition, we have implemented a tool named SynHC and evaluated its performance over a set of benchmark examples. The experimental results demonstrate that our approach efficiently synthesizes safe DNN controllers.

AAMAS Conference 2025 Conference Paper

Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors

• Lang Feng
• Jiahao Lin
• Dong Xing
• Li Zhang
• De Ma
• Gang Pan

Population-population generalization is a challenging problem in multi-agent reinforcement learning (MARL), particularly when agents encounter unseen co-players. However, existing self-playbased methods are constrained by the limitation of inside-space generalization. In this study, we propose Bidirectional Distillation (BiDist), a novel mixed-play framework, to overcome this limitation in MARL. BiDist leverages knowledge distillation in two alternating directions: forward distillation, which emulates the historical policies’ space and creates an implicit self-play, and reverse distillation, which systematically drives agents towards novel distributions outside the known policy space in a non-self-play manner. Our results highlight its remarkable generalization ability across a variety of cooperative, competitive, and social dilemma tasks, and reveal that BiDist significantly diversifies the policy distribution space.

YNICL Journal 2025 Journal Article

Brain network dynamics during rumination relate to relapse of depression

• Su Shu
• Yumeng Ju
• Mi Wang
• Wenwen Ou
• Mohan Ma
• Qianqian Zhang
• Mei Huang
• Hairuo He

BACKGROUND: Rumination is a maladaptive cognitive style and a risk factor for relapse of depression. However, the clinically relevant pattern of dynamic network reconfiguration during rumination in remitted depression and its implication in relapse remained unclear. METHODS: We employed a rumination induction neuroimaging paradigm in which subjects would be guided into an active rumination state and a distraction state. Forty-two patients with remitted depression were involved. Participants underwent assessments of rumination behavior and imaging tasks, and were then monitored for two year to assess the potential relapse of depression. A time-resolved community detection approach was applied to investigate the temporal dynamics of brain networks, and the dynamic network properties including flexibility and integration were analyzed. RESULTS: = 0.036). Moreover, elastic net regression indicated that dynamic network features could predict two-year relapse outcomes with moderate accuracy (AUC = 0.70). CONCLUSIONS: Our findings reveal a potential mechanistic link between the brain network dynamics during rumination and relapse of depression, shedding light on the intricate relationship between cognitive-affective processes, neural dynamics, and the potential vulnerability to depression recurrence.

AAAI Conference 2025 Conference Paper

Calibrating Large Language Models with Sample Consistency

• Qing Lyu
• Kumar Shridhar
• Chaitanya Malaviya
• Li Zhang
• Yanai Elazar
• Niket Tandon
• Marianna Apidianaki
• Mrinmaya Sachan

Accurately gauging the confidence level of Large Language Models' (LLMs) predictions is pivotal for their reliable application. However, LLMs are often uncalibrated inherently and elude conventional calibration techniques due to their proprietary nature and massive scale. In this work, we derive model confidence from the distribution of multiple randomly sampled generations, using three measures of consistency. We extensively evaluate eleven open and closed-source models on nine reasoning datasets. Results show that consistency-based calibration methods outperform existing post-hoc approaches in terms of calibration error. Meanwhile, we find that factors such as intermediate explanations, model scaling, and larger sample sizes enhance calibration, while instruction-tuning makes calibration more difficult. Moreover, confidence scores obtained from consistency can potentially enhance model performance. Finally, we offer guidance on choosing suitable consistency metrics for calibration, tailored to model characteristics such as the exposure to instruction-tuning and RLHF.

ICLR Conference 2025 Conference Paper

Controllable Unlearning for Image-to-Image Generative Models via ϵ-Constrained Optimization

• Xiaohua Feng 0002
• Yuyuan Li 0001
• Chaochao Chen 0001
• Li Zhang
• Longfei Li
• Jun Zhou 0011
• Xiaolin Zheng

While generative models have made significant advancements in recent years, they also raise concerns such as privacy breaches and biases. Machine unlearning has emerged as a viable solution, aiming to remove specific training data, e.g., containing private information and bias, from models. In this paper, we study the machine unlearning problem in Image-to-Image (I2I) generative models. Previous studies mainly treat it as a single objective optimization problem, offering a solitary solution, thereby neglecting the varied user expectations towards the trade-off between complete unlearning and model utility. To address this issue, we propose a controllable unlearning framework that uses a control coefficient $\epsilon$ to control the trade-off. We reformulate the I2I generative model unlearning problem into a $\epsilon$-constrained optimization problem and solve it with a gradient-based method to find optimal solutions for unlearning boundaries. These boundaries define the valid range for the control coefficient. Within this range, every yielded solution is theoretically guaranteed with Pareto optimality. We also analyze the convergence rate of our framework under various control functions. Extensive experiments on two benchmark datasets across three mainstream I2I models demonstrate the effectiveness of our controllable unlearning framework.

NeurIPS Conference 2025 Conference Paper

FedFACT: A Provable Framework for Controllable Group-Fairness Calibration in Federated Learning

• Li Zhang
• Zhongxuan Han
• XiaoHua Feng
• Jiaming Zhang
• Yuyuan Li
• Chaochao Chen

With emerging application of Federated Learning (FL) in decision-making scenarios, it is imperative to regulate model fairness to prevent disparities across sensitive groups (e. g. , female, male). Current research predominantly focuses on two concepts of group fairness within FL: Global Fairness (overall model disparity across all clients) and Local Fairness (the disparity within each client). However, the non-decomposable, non-differentiable nature of fairness criteria pose two fundamental, unresolved challenges for fair FL: (i) Harmonizing global and local fairness, especially in multi-class classification; (ii) Enabling a controllable, optimal accuracy-fairness trade-off. To tackle the aforementioned challenges, we propose a novel controllable federated group-fairness calibration framework, named FedFACT. FedFACT identifies the Bayes-optimal classifiers under both global and local fairness constraints in multi-class case, yielding models with minimal performance decline while guaranteeing fairness. To effectively realize an adjustable, optimal accuracy-fairness balance, we derive specific characterizations of the Bayes-optimal fair classifiers for reformulating fair FL as personalized cost-sensitive learning problem for in-processing, and bi-level optimization for post-processing. Theoretically, we provide convergence and generalization guarantees for FedFACT to approach the near-optimal accuracy under given fairness levels. Extensive experiments on multiple datasets across various data heterogeneity demonstrate that FedFACT consistently outperforms baselines in balancing accuracy and global-local fairness.

NeurIPS Conference 2025 Conference Paper

From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D

• Jiahui Zhang
• Yurui Chen
• Yueming Xu
• Ze Huang
• Jilin Mei
• Chunhui Chen
• Yanpeng Zhou
• Yu-Jie Yuan

Recent advances in LVLMs have improved vision-language understanding, but they still struggle with spatial perception, limiting their ability to reason about complex 3D scenes. Unlike previous approaches that incorporate 3D representations into models to improve spatial understanding, we aim to unlock the potential of VLMs by leveraging spatially relevant image data. To this end, we introduce a novel 2D spatial data generation and annotation pipeline built upon scene data with 3D ground-truth. This pipeline enables the creation of a diverse set of spatial tasks, ranging from basic perception tasks to more complex reasoning tasks. Leveraging this pipeline, we construct SPAR-7M, a large-scale dataset generated from thousands of scenes across multiple public datasets. In addition, we introduce SPAR-Bench, a benchmark designed to offer a more comprehensive evaluation of spatial capabilities compared to existing spatial benchmarks, supporting both single-view and multi-view inputs. Training on both SPAR-7M and large-scale 2D datasets enables our models to achieve state-of-the-art performance on 2D spatial benchmarks. Further fine-tuning on 3D task-specific datasets yields competitive results, underscoring the effectiveness of our dataset in enhancing spatial reasoning.

NeurIPS Conference 2025 Conference Paper

Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution

• Bozhou Zhang
• Nan Song
• Jingyu Li
• Xiatian Zhu
• Jiankang Deng
• Li Zhang

End-to-end autonomous driving methods aim to directly map raw sensor inputs to future driving actions such as planned trajectories, bypassing traditional modular pipelines. While these approaches have shown promise, they often operate under a one-shot paradigm that relies heavily on the current scene context, potentially underestimating the importance of scene dynamics and their temporal evolution. This limitation restricts the model’s ability to make informed and adaptive decisions in complex driving scenarios. We propose a new perspective: the future trajectory of an autonomous vehicle is closely intertwined with the evolving dynamics of its environment, and conversely, the vehicle’s own future states can influence how the surrounding scene unfolds. Motivated by this bidirectional relationship, we introduce SeerDrive, a novel end-to-end framework that jointly models future scene evolution and trajectory planning in a closed-loop manner. Our method first predicts future bird’s-eye view (BEV) representations to anticipate the dynamics of the surrounding scene, then leverages this foresight to generate future-context-aware trajectories. Two key components enable this: (1) future-aware planning, which injects predicted BEV features into the trajectory planner, and (2) iterative scene modeling and vehicle planning, which refines both future scene prediction and trajectory generation through collaborative optimization. Extensive experiments on the NAVSIM and nuScenes benchmarks show that SeerDrive significantly outperforms existing state-of-the-art methods.

EAAI Journal 2025 Journal Article

Hyperspectral imaging for rapid impurity detection in power system liquids

• Liang Xue
• Li Zhang
• Zhuoyue Yang
• Youhua Jiang
• Chao Jiang
• Haoyang Cui

The chemical integrity of power system liquids, such as coolants and transformer oils, is critical for the reliable operation of energy systems. Contaminants such as carbon, iron, copper, and tin can compromise cooling efficiency, increase failure risks, reduce equipment lifespan, and cause electrical malfunctions, thereby threatening the safety and stability of these systems. This study presents an innovative approach that integrates hyperspectral imaging (HSI) with machine learning (ML) algorithms to identify and quantify impurities in these liquids. A weighted ensemble model, referred to as the WeightedEnsemble_L2 model, has been developed and optimized. This model utilizes thirteen advanced machine-learning algorithms to identify impurities by analyzing spectral signatures across a broad wavelength range. The implemented artificial intelligence (AI) model demonstrates 90 % accuracy on the training set and 87. 53 % on the validation set. This novel approach offers a robust solution for impurity detection in power system liquids, supporting predictive maintenance and enhancing the safety and stability of energy systems through the practical application of AI technology.

AAAI Conference 2025 Conference Paper

LoGoFair: Post-Processing for Local and Global Fairness in Federated Learning

• Li Zhang
• Chaochao Chen
• Zhongxuan Han
• Qiyong Zhong
• Xiaolin Zheng

Federated learning (FL) has garnered considerable interest for its capability to learn from decentralized data sources. Given the increasing application of FL in decision-making scenarios, addressing fairness issues across different sensitive groups (e.g., female, male) in FL is crucial. Current research typically focus on facilitating fairness at each client's data (local fairness) or within the entire dataset across all clients (global fairness). However, existing approaches that focus exclusively on either global or local fairness fail to address two key challenges: (CH1) Under statistical heterogeneity, global fairness does not imply local fairness, and vice versa. (CH2) Achieving fairness under model-agnostic setting. To tackle the aforementioned challenges, this paper proposes a novel post-processing framework for achieving both Local and Global Fairness in the FL context, namely LoGoFair. To address CH1, LoGoFair endeavors to seek the Bayes optimal classifier under local and global fairness constraints, which strikes the optimal accuracy-fairness balance in the probabilistic sense. To address CH2, LoGoFair employs a model-agnostic federated post-processing procedure that enables clients to collaboratively optimize global fairness while ensuring local fairness, thereby achieving the optimal fair classifier within FL. Experimental results on three real-world datasets further illustrate the effectiveness of the proposed LoGoFair framework.

JBHI Journal 2025 Journal Article

M3D: Manifold-Based Domain Adaptation With Dynamic Distribution for Non-Deep Transfer Learning in Cross-Subject and Cross-Session EEG-Based Emotion Recognition

• Ting Luo
• Jing Zhang
• Yingwei Qiu
• Li Zhang
• Yaohua Hu
• Zhuliang Yu
• Zhen Liang

Emotion decoding using Electroencephalography (EEG)-based affective brain-computer interfaces (aBCIs) is crucial for affective computing but is hindered by EEG’s non-stationarity, individual variability, and the high cost of large-scale labeled data. Deep learning-based approaches, while effective, require substantial computational resources and large datasets, limiting their practicality. To address these challenges, we propose Manifold-based Domain Adaptation with Dynamic Distribution (M3D), a lightweight non-deep transfer learning framework. M3D includes four main modules: manifold feature transformation, dynamic distribution alignment, classifier learning, and ensemble learning. The data undergoes a transformation onto an optimal Grassmann manifold space, enabling dynamic alignment of the source and target domains. This process prioritizes both marginal and conditional distributions according to their significance, ensuring enhanced adaptation efficiency across various types of data. In the classifier learning, the principle of structural risk minimization is integrated to develop robust classification models. This is complemented by dynamic distribution alignment, which refines the classifier iteratively. Additionally, the ensemble learning module aggregates the classifiers obtained at different stages of the optimization process, which leverages the diversity of the classifiers to enhance the overall prediction accuracy. The proposed M3D framework is evaluated on three benchmark EEG emotion recognition datasets using two validation protocols (cross-subject single-session and cross-subject cross-session), as well as on a clinical EEG dataset of Major Depressive Disorder (MDD). Experimental results demonstrate that M3D outperforms traditional non-deep learning methods, achieving an average improvement of 6. 67%, while achieving deep learning-comparable performance with significantly lower data and computational requirements. These findings highlight the potential of M3D to enhance the practicality and applicability of aBCIs in real-world scenarios.

IROS Conference 2025 Conference Paper

MPDG-SLAM: Motion Probability-Based 3DGS-SLAM in Dynamic Environment

• Conghao Huang
• Li Zhang
• Tianchen Deng
• Kangxu Wang
• Mingrui Li

We present MPDG-SLAM, a novel 3D Gaussian point cloud rendering SLAM method based on Motion Probability (MP) for dynamic interference handling. Current 3DGSSLAM approaches for dynamic environments often rely on optical flow estimation masks. However, these deep learning-based optical flow models are computationally intensive and limited by processing speed, posing challenges for deployment on mobile devices in real-world scenarios. Moreover, existing systems depend on precise mask segmentation and corresponding loss functions for artifact removal, yet the pixel accuracy of optical flow estimation is constrained by real-world lighting conditions. To address these issues, we introduce a mobile-deployable Yolo and a mathematically derived Motion Probability (MP) attribute to label Gaussian points, which are then inversely mapped to the front-end feature tracking system to correct for dynamic object influences. By incorporating an MP-based penalty term, dynamic Gaussians corresponding to moving entities are explicitly removed to minimize their effect. Additionally, we design an edge warp loss based on MP estimation, enabling accurate artifact removal even with coarse segmentation masks. The experiments show that our approach notably improves the reconstruction quality of dynamic scenes, surpassing baseline methods and reaching speeds over 30 FPS on high-end GPUs, which suggests its potential for real-time use on mobile platforms after further optimization.

JBHI Journal 2025 Journal Article

NSSI-Net: A Multi-Concept GAN for Non-Suicidal Self-Injury Detection Using High-Dimensional EEG in a Semi-Supervised Framework

• Zhen Liang
• Weishan Ye
• Qile Liu
• Li Zhang
• Gan Huang
• Yongjie Zhou

Non-suicidal self-injury (NSSI) is a serious threat to the physical and mental health of adolescents, significantly increasing the risk of suicide and attracting widespread public concern. Electroencephalography (EEG), as an objective tool for identifying brain disorders, holds great promise. However, extracting meaningful and reliable features from high-dimensional EEG data, especially by integrating spatiotemporal brain dynamics into informative representations, remains a major challenge. In this study, we introduce an advanced semi-supervised adversarial network, NSSI-Net, to effectively model EEG features related to NSSI. NSSI-Net consists of two key modules: a spatial-temporal feature extraction module and a multi-concept discriminator. In the spatial-temporal feature extraction module, an integrated 2D convolutional neural network (2D-CNN) and a bi-directional Gated Recurrent Unit (BiGRU) are used to capture both spatial and temporal dynamics in EEG data. In the multi-concept discriminator, signal, gender, domain, and disease levels are fully explored to extract meaningful EEG features, considering individual, demographic, disease variations across a diverse population. Based on self-collected NSSI data (n=114), the model's effectiveness and reliability are demonstrated, with a 5. 44% improvement in performance compared to existing machine learning and deep learning methods. This study advances the understanding and early diagnosis of NSSI in adolescents with depression, enabling timely intervention.

NeurIPS Conference 2025 Conference Paper

On the Stability of Graph Convolutional Neural Networks: A Probabilistic Perspective

• Ning Zhang
• Henry Kenlay
• Li Zhang
• Mihai Cucuringu
• Xiaowen Dong

Graph convolutional neural networks (GCNNs) have emerged as powerful tools for analyzing graph-structured data, achieving remarkable success across diverse applications. However, the theoretical understanding of the stability of these models, i. e. , their sensitivity to small changes in the graph structure, remains in rather limited settings, hampering the development and deployment of robust and trustworthy models in practice. To fill this gap, we study how small perturbations in the graph topology affect GCNN outputs and propose a novel formulation for analyzing model stability. Unlike prior studies that focus only on worst-case perturbations, our distribution-aware formulation characterizes output perturbations across a broad range of input data. This way, our framework enables, for the first time, a probabilistic perspective on the interplay between the statistical properties of the node data and perturbations in the graph topology. We conduct extensive experiments to validate our theoretical findings and demonstrate their benefits over existing baselines, in terms of both representation stability and adversarial attacks on downstream tasks. Our results demonstrate the practical significance of the proposed formulation and highlight the importance of incorporating data distribution into stability analysis.

IJCAI Conference 2025 Conference Paper

Pre-defined Keypoints Promote Category-level Articulation Pose Estimation via Multi-Modal Alignment

• Wenbo Xu
• Li Zhang
• Liu Liu
• Yan Zhong
• Haonan Jiang
• Xue Wang
• Rujing Wang

Articulations are essential in everyday interactions, yet traditional RGB-based pose estimation methods often struggle with issues such as lighting variations and shadows. To overcome these challenges, we propose a novel Pre-defined keypoint based framework for category-level articulation pose estimation via multi-modal Alignment, coined PAGE. Specifically, we first propose a customized keypoint estimation method, aiming to avoid the divergent distance pattern between heuristically generated keypoints and visible points. In addition, to reduce the mutual information redundancy between point clouds and RGB images, we design the geometry-color alignment, which fuses the features after aligning two modalities. This is followed by decoding the radius for each visible point, and applying our proposal integration scoring strategy to predict keypoints. Ultimately, the framework outputs the per-part 6D pose of the articulation. We conduct extensive experiments to evaluate PAGE across a variety of datasets, from synthetic to real-world scenarios, demonstrating its robustness and superior performance.

NeurIPS Conference 2025 Conference Paper

Q-Insight: Understanding Image Quality via Visual Reinforcement Learning

• Weiqi Li
• Xuanyu Zhang
• Shijie Zhao
• Yabin ZHANG
• Junlin Li
• Li Zhang
• Jian Zhang

Image quality assessment (IQA) focuses on the perceptual visual quality of images, playing a crucial role in downstream tasks such as image reconstruction, compression, and generation. The rapid advancement of multi-modal large language models (MLLMs) has significantly broadened the scope of IQA, moving toward comprehensive image quality understanding that incorporates content analysis, degradation perception, and comparison reasoning beyond mere numerical scoring. Previous MLLM-based methods typically either generate numerical scores lacking interpretability or heavily rely on supervised fine-tuning (SFT) using large-scale annotated datasets to provide descriptive assessments, limiting their flexibility and applicability. In this paper, we propose Q-Insight, a reinforcement learning-based model built upon group relative policy optimization (GRPO), which demonstrates strong visual reasoning capability for image quality understanding while requiring only a limited amount of rating scores and degradation labels. By jointly optimizing score regression and degradation perception tasks with carefully designed reward functions, our approach effectively exploits their mutual benefits for enhanced performance. Extensive experiments demonstrate that Q-Insight substantially outperforms existing state-of-the-art methods on both score regression and degradation perception tasks, while exhibiting impressive zero-shot generalization and superior comparison reasoning capability. The code and models are available at https: //github. com/bytedance/Q-Insight.

AAAI Conference 2025 Conference Paper

R^2-Art: Category-Level Articulation Pose Estimation from Single RGB Image via Cascade Render Strategy

• Li Zhang
• Haonan Jiang
• Yukang Huo
• Yan Zhong
• Jianan Wang
• Xue Wang
• Rujing Wang
• Liu Liu

Human life is filled with articulated objects. Previous works for estimating the pose of category-level articulated objects rely on costly 3D point clouds or RGB-D images. In this paper, our goal is to estimate category-level articulation poses from a single RGB image, where we propose R2-Art, a novel category-level Articulation pose estimation framework from a single RGB image and a cascade Render strategy. Given an RGB image as input, R2-Art estimates per-part 6D pose for the articulation. Specifically, we design parallel regression branches tailored to generate camera-to-root translation and rotation. Using the predicted joint states, we perform PC prior transformation and deformation with a joint-centric modeling approach. For further refinement, a cascade render strategy is proposed for projecting the 3D deformed prior onto the 2D mask. Extensive experiments are provided to validate our R2-Art on various datasets ranging from synthetic datasets to real-world scenarios, demonstrating the superior performance and robustness of the R2-Art. We believe that this work has the potential to be applied in many fields including robotics, embodied intelligence, and augmented reality.

IROS Conference 2025 Conference Paper

Reinforcement Learning-Based Microrobotic Swarm Navigation and Obstacle Avoidance in Partially Observable Environments

• Shengming Luo
• Xuanyu An
• Qijun Yang
• Haoyu Zhang
• Li Zhang
• Qianqian Wang

Microrobotic swarms have shown promising features due to their collective and flexible behaviours, while achieving precise swarm control and autonomous navigation in complex environments remains a challenge. Here, we propose a Transformer-based reinforcement learning strategy that integrates Proximal Policy Optimization for autonomous swarm control in obstacle environments. By combining domain randomization, this strategy enables direct transfer from simulation to real-world without fine tuning. Experimental results demonstrate robust control performance in avoiding static obstacles and tracking the dynamic target, which is not validated in training. The swarm autonomously navigates and adjusts its velocity and trajectory in obstacle environments with an intact swarm pattern. Our work presents a scalable strategy for the deployment of microrobotic swarms with adaptive navigation capability through complex, constrained environments.

NeurIPS Conference 2025 Conference Paper

Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization

• Ming Nie
• Chunwei Wang
• Jianhua Han
• Hang Xu
• Li Zhang

Unified vision-language models have made significant progress in multimodal understanding and generation, yet they largely fall short in producing multimodal interleaved outputs, which is a crucial capability for tasks like visual storytelling and step-by-step visual reasoning. In this work, we propose a reinforcement learning-based post-training strategy to unlock this capability in existing unified models, without relying on large-scale multimodal interleaved datasets. We begin with a warm-up stage using a hybrid dataset comprising curated interleaved sequences and limited data for multimodal understanding and text-to-image generation, which exposes the model to interleaved generation patterns while preserving its pretrained capabilities. To further refine interleaved generation, we propose a unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the multimodal setting. Our approach jointly models text and image generation within a single decoding trajectory and optimizes it with our novel hybrid rewards covering textual relevance, visual-text alignment, and structural fidelity. Additionally, we incorporate process-level rewards to provide step-wise guidance, enhancing training efficiency in complex multimodal tasks. Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of multimodal interleaved generation.

NeurIPS Conference 2025 Conference Paper

UniMotion: A Unified Motion Framework for Simulation, Prediction and Planning

• Nan Song
• Junzhe Jiang
• Jingyu Li
• Xiatian Zhu
• Li Zhang

Motion simulation, prediction and planning are foundational tasks in autonomous driving, each essential for modeling and reasoning about dynamic traffic scenarios. While often addressed in isolation due to their differing objectives, such as generating diverse motion states or estimating optimal trajectories, these tasks inherently depend on shared capabilities: understanding multi-agent interactions, modeling motion behaviors, and reasoning over temporal and spatial dynamics. Despite this underlying commonality, existing approaches typically adopt specialized model designs, which hinders cross-task generalization and system scalability. More critically, this separation overlooks the potential mutual benefits among tasks. Motivated by these observations, we propose UniMotion, a unified motion framework that captures shared structures across motion tasks while accommodating their individual requirements. Built on a decoder-only Transformer architecture, UniMotion employs dedicated interaction modes and tailored training strategies to simultaneously support these motion tasks. This unified design not only enables joint optimization and representation sharing but also allows for targeted fine-tuning to specialize in individual tasks when needed. Extensive experiments on the Waymo Open Motion Dataset (WOMD) demonstrate that joint training leads to robust generalization and effective task integration. With further fine-tuning, UniMotion achieves state-of-the-art performance across a range of motion tasks, establishing it as a versatile and scalable solution for autonomous driving.

JBHI Journal 2024 Journal Article

A Two-Stage Differential Privacy Scheme for Federated Learning Based on Edge Intelligence

• Li Zhang
• Jianbo Xu
• Audithan Sivaraman
• Jegatha Deborah Lazarus
• Pradip Kumar Sharma
• Vijayakumar Pandi

The issue of data privacy protection must be considered in distributed federated learning (FL) so as to ensure that sensitive information is not leaked. In this article, we propose a two-stage differential privacy (DP) framework for FL based on edge intelligence. Various levels of privacy preservation can be provided according to the degree of data sensitivity. In the first stage, the randomized response mechanism is used to perturb the original feature data by the user terminal for data desensitization, and the user can self-regulate the level of privacy preservation. In the second stage, noise is added to the local models by the edge server to further guarantee the privacy of the models. Finally, the model updates are aggregated in the cloud. In order to evaluate the performance of the proposed end-edge-cloud FL framework in terms of training accuracy and convergence, extensive experiments are conducted on a real electrocardiogram (ECG) signal dataset. Bi-directional long-short-term memory (BiLSTM) neural network is adopted to training classification model. The effect of different combinations of feature perturbation and noise addition on the model accuracy is analyzed depending on different privacy budgets and parameters. The experimental results demonstrate that the proposed privacy-preserving framework provides good accuracy and convergence while ensuring privacy.

JBHI Journal 2024 Journal Article

An Efficient Multi-Task Synergetic Network for Polyp Segmentation and Classification

• Miao Wang
• Xingwei An
• Zhengcun Pei
• Ning Li
• Li Zhang
• Gang Liu
• Dong Ming

Colonoscopy is considered the best diagnostic tool for early detection and resection of polyps, which can effectively prevent consequential colorectal cancer. In clinical practice, segmenting and classifying polyps from colonoscopic images have a great significance since they provide precious information for diagnosis and treatment. In this study, we propose an efficient multi-task synergetic network (EMTS-Net) for concurrent polyp segmentation and classification, and we introduce a polyp classification benchmark for exploring the potential correlations of the above-mentioned two tasks. This framework is composed of an enhanced multi-scale network (EMS-Net) for coarse-grained polyp segmentation, an EMTS-Net (Class) for accurate polyp classification, and an EMTS-Net (Seg) for fine-grained polyp segmentation. Specifically, we first obtain coarse segmentation masks by using EMS-Net. Then, we concatenate these rough masks with colonoscopic images to assist EMTS-Net (Class) in locating and classifying polyps precisely. To further enhance the segmentation performance of polyps, we propose a random multi-scale (RMS) training strategy to eliminate the interference caused by redundant information. In addition, we design an offline dynamic class activation mapping (OFLD CAM) generated by the combined effect of EMTS-Net (Class) and RMS strategy, which optimizes bottlenecks between multi-task networks efficiently and elegantly and helps EMTS-Net (Seg) to perform more accurate polyp segmentation. We evaluate the proposed EMTS-Net on the polyp segmentation and classification benchmarks, and it achieves an average mDice of 0. 864 in polyp segmentation and an average AUC of 0. 913 with an average accuracy of 0. 924 in polyp classification. Quantitative and qualitative evaluations on the polyp segmentation and classification benchmarks demonstrate that our EMTS-Net achieves the best performance and outperforms previous state-of-the-art methods in terms of both efficiency and generalization.

ICML Conference 2024 Conference Paper

BLO-SAM: Bi-level Optimization Based Finetuning of the Segment Anything Model for Overfitting-Preventing Semantic Segmentation

• Li Zhang
• Youwei Liang
• Ruiyi Zhang
• Amirhosein Javadi
• Pengtao Xie

The Segment Anything Model (SAM), a foundation model pretrained on millions of images and segmentation masks, has significantly advanced semantic segmentation, a fundamental task in computer vision. Despite its strengths, SAM encounters two major challenges. Firstly, it struggles with segmenting specific objects autonomously, as it relies on users to manually input prompts like points or bounding boxes to identify targeted objects. Secondly, SAM faces challenges in excelling at specific downstream tasks, like medical imaging, due to a disparity between the distribution of its pretraining data, which predominantly consists of general-domain images, and the data used in downstream tasks. Current solutions to these problems, which involve finetuning SAM, often lead to overfitting, a notable issue in scenarios with very limited data, like in medical imaging. To overcome these limitations, we introduce BLO-SAM, which finetunes SAM based on bi-level optimization (BLO). Our approach allows for automatic image segmentation without the need for manual prompts, by optimizing a learnable prompt embedding. Furthermore, it significantly reduces the risk of overfitting by training the model’s weight parameters and the prompt embedding on two separate subsets of the training dataset, each at a different level of optimization. We apply BLO-SAM to diverse semantic segmentation tasks in general and medical domains. The results demonstrate BLO-SAM’s superior performance over various state-of-the-art image semantic segmentation methods. The code of BLO-SAM is available at https: //github. com/importZL/BLO-SAM.

AAAI Conference 2024 Conference Paper

CatmullRom Splines-Based Regression for Image Forgery Localization

• Li Zhang
• Mingliang Xu
• Dong Li
• Jianming Du
• Rujing Wang

IFL (Image Forgery Location) helps secure digital media forensics. However, many methods suffer from false detections (i.e., FPs) and inaccurate boundaries. In this paper, we proposed the CatmullRom Splines-based Regression Network (CSR-Net), which first rethinks the IFL task from the perspective of regression to deal with this problem. Specifically speaking, we propose an adaptive CutmullRom splines fitting scheme for coarse localization of the tampered regions. Then, for false positive cases, we first develop a novel re-scoring mechanism, which aims to filter out samples that cannot have responses on both the classification branch and the instance branch. Later on, to further restrict the boundaries, we design a learnable texture extraction module, which refines and enhances the contour representation by decoupling the horizontal and vertical forgery features to extract a more robust contour representation, thus suppressing FPs. Compared to segmentation-based methods, our method is simple but effective due to the unnecessity of post-processing. Extensive experiments show the superiority of CSR-Net to existing state-of-the-art methods, not only on standard natural image datasets but also on social media datasets.

NeurIPS Conference 2024 Conference Paper

CURE4Rec: A Benchmark for Recommendation Unlearning with Deeper Influence

• Chaochao Chen
• Jiaming Zhang
• Yizhao Zhang
• Li Zhang
• Lingjuan Lyu
• Yuyuan Li
• Biao Gong
• Chenggang Yan

With increasing privacy concerns in artificial intelligence, regulations have mandated the right to be forgotten, granting individuals the right to withdraw their data from models. Machine unlearning has emerged as a potential solution to enable selective forgetting in models, particularly in recommender systems where historical data contains sensitive user information. Despite recent advances in recommendation unlearning, evaluating unlearning methods comprehensively remains challenging due to the absence of a unified evaluation framework and overlooked aspects of deeper influence, e. g. , fairness. To address these gaps, we propose CURE4Rec, the first comprehensive benchmark for recommendation unlearning evaluation. CURE4Rec covers four aspects, i. e. , unlearning Completeness, recommendation Utility, unleaRning efficiency, and recommendation fairnEss, under three data selection strategies, i. e. , core data, edge data, and random data. Specifically, we consider the deeper influence of unlearning on recommendation fairness and robustness towards data with varying impact levels. We construct multiple datasets with CURE4Rec evaluation and conduct extensive experiments on existing recommendation unlearning methods. Our code is released at https: //github. com/xiye7lai/CURE4Rec.

NeurIPS Conference 2024 Conference Paper

DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States

• Bozhou Zhang
• Nan Song
• Li Zhang

Accurate motion forecasting for traffic agents is crucial for ensuring the safety and efficiency of autonomous driving systems in dynamically changing environments. Mainstream methods adopt a one-query-one-trajectory paradigm, where each query corresponds to a unique trajectory for predicting multi-modal trajectories. While straightforward and effective, the absence of detailed representation of future trajectories may yield suboptimal outcomes, given that the agent states dynamically evolve over time. To address this problem, we introduce DeMo, a framework that decouples multi-modal trajectory queries into two types: mode queries capturing distinct directional intentions and state queries tracking the agent's dynamic states over time. By leveraging this format, we separately optimize the multi-modality and dynamic evolutionary properties of trajectories. Subsequently, the mode and state queries are integrated to obtain a comprehensive and detailed representation of the trajectories. To achieve these operations, we additionally introduce combined Attention and Mamba techniques for global information aggregation and state sequence modeling, leveraging their respective strengths. Extensive experiments on both the Argoverse 2 and nuScenes benchmarks demonstrate that our DeMo achieves state-of-the-art performance in motion forecasting. In addition, we will make our code and models publicly available.

NeurIPS Conference 2024 Conference Paper

DG-SLAM: Robust Dynamic Gaussian Splatting SLAM with Hybrid Pose Optimization

• Yueming Xu
• Haochen Jiang
• Zhongyang Xiao
• Jianfeng Feng
• Li Zhang

Achieving robust and precise pose estimation in dynamic scenes is a significant research challenge in Visual Simultaneous Localization and Mapping (SLAM). Recent advancements integrating Gaussian Splatting into SLAM systems have proven effective in creating high-quality renderings using explicit 3D Gaussian models, significantly improving environmental reconstruction fidelity. However, these approaches depend on a static environment assumption and face challenges in dynamic environments due to inconsistent observations of geometry and photometry. To address this problem, we propose DG-SLAM, the first robust dynamic visual SLAM system grounded in 3D Gaussians, which provides precise camera pose estimation alongside high-fidelity reconstructions. Specifically, we propose effective strategies, including motion mask generation, adaptive Gaussian point management, and a hybrid camera tracking algorithm to improve the accuracy and robustness of pose estimation. Extensive experiments demonstrate that DG-SLAM delivers state-of-the-art performance in camera pose estimation, map reconstruction, and novel-view synthesis in dynamic scenes, outperforming existing methods meanwhile preserving real-time rendering ability.

NeurIPS Conference 2024 Conference Paper

EfficientCAPER: An End-to-End Framework for Fast and Robust Category-Level Articulated Object Pose Estimation

• Xinyi Yu
• Haonan Jiang
• Li Zhang
• Lin Y. Wu
• Linin Ou
• Liu Liu

Human life is populated with articulated objects. Pose estimation for category-level articulated objects is a significant challenge due to their inherent complexity and diverse kinematic structures. Current methods for this task usually meet the problems of insufficient consideration of kinematic constraints, self-occlusion, and optimization requirements. In this paper, we propose EfficientCAPER, an end-to-end Category-level Articulated object Pose EstimatoR, eliminating the need for optimization functions as post-processing and utilizing the kinematic structure for joint-centric pose modeling, thus enhancing the efficiency and applicability. Given a partial point cloud as input, the EfficientCAPER firstly estimates the pose for the free part of an articulated object using decoupled rotation representation. Next, we canonicalize the input point cloud to estimate constrained parts' poses by predicting the joint parameters and states as replacements. Evaluations on three diverse datasets, ArtImage, ReArtMix, and RobotArm, show EfficientCAPER's effectiveness and generalization ability to real-world scenarios. The framework exhibits excellent static pose estimation performance for articulated objects, contributing to the advancement of category-level pose estimation. Codes will be made publicly available.

ICML Conference 2024 Conference Paper

FrameQuant: Flexible Low-Bit Quantization for Transformers

• Harshavardhan Adepu
• Zhanpeng Zeng
• Li Zhang
• Vikas Singh

Transformers are the backbone of powerful foundation models for many Vision and Natural Language Processing tasks. But their compute and memory/storage footprint is large, and so, serving such models is expensive often requiring high-end hardware. To mitigate this difficulty, Post-Training Quantization seeks to modify a pre-trained model and quantize it to eight bits or lower, significantly boosting compute/memory/latency efficiency. Such models have been successfully quantized to four bits with some performance loss. In this work, we outline a simple scheme to quantize Transformer-based models to just two bits (plus some overhead) with only a small drop in accuracy. Key to our formulation is a concept borrowed from Harmonic analysis called Fusion Frames. Our main finding is that the quantization must take place not in the original weight space, but instead in the Fusion Frame representations. If quantization is interpreted as the addition of noise, our casting of the problem allows invoking an extensive body of known consistent recovery and noise robustness guarantees. Further, if desired, de-noising filters are known in closed form. We show empirically, via a variety of experiments, that (almost) two-bit quantization for Transformer models promises sizable efficiency gains. The code is available at https: //github. com/vsingh-group/FrameQuant

EAAI Journal 2024 Journal Article

Global superpixel-merging via set maximum coverage

• Xubing Yang
• Zhengxiao Zhang
• Li Zhang
• Xijian Fan
• Qiaolin Ye
• Liyong Fu

Due to better boundary adherence and low computational cost, the superpixel segmentation algorithm SLIC (simple linear iterative clustering) has been widely applied in vision-based applications. However, limit to unavoidable over-segmentation problem, one has to consider region-merging to reconstruct entire objects from the segmented superpixels (or called regions). The existing region-merging methods are generated from data clustering, and avoidably suffer from error-merging, slow convergence speed, or easily dropping in LOCAL optimal problems, especially for high-resolution RS (remote sensing) images. In this paper, instead of data clustering, we propose a fast GLOBAL method based on Set Maximum Coverage, termed as MaxCov-merging. Theoretically, the existence of the maximum coverage is proved by using Bayes optimal decision principle. To speed up MaxCov-merging, some heuristic strategies are also provided. Finally, extensive verification and comparison are carried on the public and our collected high-resolution images. Compared with the state-of-the-art methods, the comparison shows the superiority of our MaxCov in terms of the performance of globality, ease of use and fast region-merging speed.

AAAI Conference 2024 Conference Paper

LaneGraph2Seq: Lane Topology Extraction with Language Model via Vertex-Edge Encoding and Connectivity Enhancement

• Renyuan Peng
• Xinyue Cai
• Hang Xu
• Jiachen Lu
• Feng Wen
• Wei Zhang
• Li Zhang

Understanding road structures is crucial for autonomous driving. Intricate road structures are often depicted using lane graphs, which include centerline curves and connections forming a Directed Acyclic Graph (DAG). Accurate extraction of lane graphs relies on precisely estimating vertex and edge information within the DAG. Recent research highlights Transformer-based language models' impressive sequence prediction abilities, making them effective for learning graph representations when graph data are encoded as sequences. However, existing studies focus mainly on modeling vertices explicitly, leaving edge information simply embedded in the network. Consequently, these approaches fall short in the task of lane graph extraction. To address this, we introduce LaneGraph2Seq, a novel approach for lane graph extraction. It leverages a language model with vertex-edge encoding and connectivity enhancement. Our serialization strategy includes a vertex-centric depth-first traversal and a concise edge-based partition sequence. Additionally, we use classifier-free guidance combined with nucleus sampling to improve lane connectivity. We validate our method on prominent datasets, nuScenes and Argoverse 2, showcasing consistent and compelling results. Our LaneGraph2Seq approach demonstrates superior performance compared to state-of-the-art techniques in lane graph extraction.

NeurIPS Conference 2024 Conference Paper

Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement

• Zhi Wang
• Li Zhang
• Wenhao Wu
• Yuanheng Zhu
• Dongbin Zhao
• Chunlin Chen

A longstanding goal of artificial general intelligence is highly capable generalists that can learn from diverse experiences and generalize to unseen tasks. The language and vision communities have seen remarkable progress toward this trend by scaling up transformer-based models trained on massive datasets, while reinforcement learning (RL) agents still suffer from poor generalization capacity under such paradigms. To tackle this challenge, we propose Meta Decision Transformer (Meta-DT), which leverages the sequential modeling ability of the transformer architecture and robust task representation learning via world model disentanglement to achieve efficient generalization in offline meta-RL. We pretrain a context-aware world model to learn a compact task representation, and inject it as a contextual condition to the causal transformer to guide task-oriented sequence generation. Then, we subtly utilize history trajectories generated by the meta-policy as a self-guided prompt to exploit the architectural inductive bias. We select the trajectory segment that yields the largest prediction error on the pretrained world model to construct the prompt, aiming to encode task-specific information complementary to the world model maximally. Notably, the proposed framework eliminates the requirement of any expert demonstration or domain knowledge at test time. Experimental results on MuJoCo and Meta-World benchmarks across various dataset types show that Meta-DT exhibits superior few and zero-shot generalization capacity compared to strong baselines while being more practical with fewer prerequisites. Our code is available at https: //github. com/NJU-RL/Meta-DT.

NeurIPS Conference 2024 Conference Paper

Motion Forecasting in Continuous Driving

• Nan Song
• Bozhou Zhang
• Xiatian Zhu
• Li Zhang

Motion forecasting for agents in autonomous driving is highly challenging due to the numerous possibilities for each agent's next action and their complex interactions in space and time. In real applications, motion forecasting takes place repeatedly and continuously as the self-driving car moves. However, existing forecasting methods typically process each driving scene within a certain range independently, totally ignoring the situational and contextual relationships between successive driving scenes. This significantly simplifies the forecasting task, making the solutions suboptimal and inefficient to use in practice. To address this fundamental limitation, we propose a novel motion forecasting framework for continuous driving, named RealMotion. It comprises two integral streams both at the scene level: (1) The scene context stream progressively accumulates historical scene information until the present moment, capturing temporal interactive relationships among scene elements. (2) The agent trajectory stream optimizes current forecasting by sequentially relaying past predictions. Besides, a data reorganization strategy is introduced to narrow the gap between existing benchmarks and real-world applications, consistent with our network. These approaches enable exploiting more broadly the situational and progressive insights of dynamic motion across space and time. Extensive experiments on Argoverse series with different settings demonstrate that our RealMotion achieves state-of-the-art performance, along with the advantage of efficient real-world inference.

AAAI Conference 2024 Conference Paper

NeRF-LiDAR: Generating Realistic LiDAR Point Clouds with Neural Radiance Fields

• Junge Zhang
• Feihu Zhang
• Shaochen Kuang
• Li Zhang

Labelling LiDAR point clouds for training autonomous driving is extremely expensive and difficult. LiDAR simulation aims at generating realistic LiDAR data with labels for training and verifying self-driving algorithms more efficiently. Recently, Neural Radiance Fields (NeRF) have been proposed for novel view synthesis using implicit reconstruction of 3D scenes. Inspired by this, we present NeRF-LIDAR, a novel LiDAR simulation method that leverages real-world information to generate realistic LIDAR point clouds. Different from existing LiDAR simulators, we use real images and point cloud data collected by self-driving cars to learn the 3D scene representation, point cloud generation and label rendering. We verify the effectiveness of our NeRF-LiDAR by training different 3D segmentation models on the generated LiDAR point clouds. It reveals that the trained models are able to achieve similar accuracy when compared with the same model trained on the real LiDAR data. Besides, the generated data is capable of boosting the accuracy through pre-training which helps reduce the requirements of the real labeled data. Code is available at https://github.com/fudan-zvg/NeRF-LiDAR

EAAI Journal 2024 Journal Article

Parallel weight control based on policy gradient of relation refinement for cross-modal retrieval

• Li Zhang
• Yahu Yang
• Shuheng Ge
• Guanghui Sun
• Xiangqian Wu

Cross-modal retrieval has become one of the hot topics in the field of multi-modal research, which receives widespread attention. Existing approaches adopted object region (word) features to generate final image (text) global feature vector without weighting or with unsupervised attention mechanism, which lead to mistake attention or weighting of object region (word) features. In this paper, a parallel weight control method based on Policy Gradient of Relationship Refinement (PGRR) is proposed for cross-modal retrieval, which utilizes self-attention mechanism to model the relationship between any local features and all local features within the modality, thereby more accurately using discrete and continuous policy gradient to estimate the weight of this local feature in the final global feature. Furthermore, PGRR transforms the existing iterative prediction weight pattern into parallel weight control, which significantly improves the training and inference efficiency of the model. Extensive experiments on MS-COCO and Flicker30K datasets demonstrate PGRR consistently outperforms state-of-the-art methods for the image-text matching.

NeurIPS Conference 2024 Conference Paper

Rethinking 3D Convolution in $\ell_p$-norm Space

• Li Zhang
• Yan Zhong
• Jianan Wang
• Zhe Min
• Rujing Wang
• Liu Liu

Convolution is a fundamental operation in the 3D backbone. However, under certain conditions, the feature extraction ability of traditional convolution methods may be weakened. In this paper, we introduce a new convolution method based on $\ell_p$-norm. For theoretical support, we prove the universal approximation theorem for $\ell_p$-norm based convolution, and analyze the robustness and feasibility of $\ell_p$-norms in 3D point cloud tasks. Concretely, $\ell_{\infty}$-norm based convolution is prone to feature loss. $\ell_2$-norm based convolution is essentially a linear transformation of the traditional convolution. $\ell_1$-norm based convolution is an economical and effective feature extractor. We propose customized optimization strategies to accelerate the training process of $\ell_1$-norm based Nets and enhance the performance. Besides, a theoretical guarantee is given for the convergence by \textit{regret} argument. We apply our methods to classic networks and conduct related experiments. Experimental results indicate that our approach exhibits competitive performance with traditional CNNs, with lower energy consumption and instruction latency.

NeurIPS Conference 2024 Conference Paper

SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM

• Ming Nie
• Dan Ding
• Chunwei Wang
• Yuanfan Guo
• Jianhua Han
• Hang Xu
• Li Zhang

Large language models (LLMs) have demonstrated exceptional capabilities in text understanding, which has paved the way for their expansion into video LLMs (Vid-LLMs) to analyze video data. However, current Vid-LLMs struggle to simultaneously retain high-quality frame-level semantic information (i. e. , a sufficient number of tokens per frame) and comprehensive video-level temporal information (i. e. , an adequate number of sampled frames per video). This limitation hinders the advancement of Vid-LLMs towards fine-grained video understanding. To address this issue, we introduce the SlowFocus mechanism, which significantly enhances the equivalent sampling frequency without compromising the quality of frame-level visual tokens. SlowFocus begins by identifying the query-related temporal segment based on the posed question, then performs dense sampling on this segment to extract local high-frequency features. A multi-frequency mixing attention module is further leveraged to aggregate these local high-frequency details with global low-frequency contexts for enhanced temporal comprehension. Additionally, to tailor Vid-LLMs to this innovative mechanism, we introduce a set of training strategies aimed at bolstering both temporal grounding and detailed temporal reasoning capabilities. Furthermore, we establish FineAction-CGR, a benchmark specifically devised to assess the ability of Vid-LLMs to process fine-grained temporal understanding tasks. Comprehensive experiments demonstrate the superiority of our mechanism across both existing public video understanding benchmarks and our proposed FineAction-CGR.

NeurIPS Conference 2024 Conference Paper

Tetrahedron Splatting for 3D Generation

• Chun Gu
• Zeyu Yang
• Zijie Pan
• Xiatian Zhu
• Li Zhang

3D representation is essential to the significant advance of 3D generation with 2D diffusion priors. As a flexible representation, NeRF has been first adopted for 3D representation. With density-based volumetric rendering, it however suffers both intensive computational overhead and inaccurate mesh extraction. Using a signed distance field and Marching Tetrahedra, DMTet allows for precise mesh extraction and real-time rendering but is limited in handling large topological changes in meshes, leading to optimization challenges. Alternatively, 3D Gaussian Splatting (3DGS) is favored in both training and rendering efficiency while falling short in mesh extraction. In this work, we introduce a novel 3D representation, Tetrahedron Splatting (TeT-Splatting), that supports easy convergence during optimization, precise mesh extraction, and real-time rendering simultaneously. This is achieved by integrating surface-based volumetric rendering within a structured tetrahedral grid while preserving the desired ability of precise mesh extraction, and a tile-based differentiable tetrahedron rasterizer. Furthermore, we incorporate eikonal and normal consistency regularization terms for the signed distance field to improve generation quality and stability. Critically, our representation can be trained without mesh extraction, making the optimization process easier to converge. Our TeT-Splatting can be readily integrated in existing 3D generation pipelines, along with polygonal mesh for texture optimization. Extensive experiments show that our TeT-Splatting strikes a superior tradeoff among convergence speed, render efficiency, and mesh quality as compared to previous alternatives under varying 3D generation settings.

AAAI Conference 2024 Conference Paper

UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer

• Ji Liu
• Dehua Tang
• Yuanxian Huang
• Li Zhang
• Xiaocheng Zeng
• Dong Li
• Mingjie Lu
• Jinzhang Peng

Traditional channel-wise pruning methods by reducing network channels struggle to effectively prune efficient CNN models with depth-wise convolutional layers and certain efficient modules, such as popular inverted residual blocks. Prior depth pruning methods by reducing network depths are not suitable for pruning some efficient models due to the existence of some normalization layers. Moreover, finetuning subnet with directly removing activation layers would corrupt the original model weights, hindering the pruned model from achieving high performance. To address these issues, we propose a novel depth pruning method for efficient models. Our approach proposes a novel block pruning strategy and progressive training method for the subnet. Additionally, we extend our pruning method to vision transformer models. Experimental results demonstrate that our method consistently outperforms existing depth pruning methods across various pruning configurations. We obtained three pruned ConvNeXtV1 models with our method applying on ConvNeXtV1, which surpass most SOTA efficient models with comparable inference performance. Our method also achieves state-of-the-art pruning performance on the vision transformer model.

JBHI Journal 2024 Journal Article

Using Pupil Diameter for Psychological Resilience Assessment in Medical Students Based on SVM and SHAP Model

• Fayang Xiang
• Li Zhang
• Yidan Ye
• Chuyue Xiong
• Yanjie Zhang
• Yan Hu
• Jiang Du
• Yi Zhou

Effectively assessing psychological resilience for medical students is vital for identifying at-risk individuals and developing tailored interventions. At present, few studies have combined physiological indexes of the human body and machine learning for psychological resilience assessment. This study presents a novel approach that employs pupil diameter features and machine learning to predict psychological resilience risk objectively. Firstly, we designed a stimulus paradigm (via auditory and visual stimuli) and collected pupil diameter data from participants using eye-tracking technology. Secondly, the pupil data was preprocessed, including linear interpolation, blink detection, and subtractive baseline correction. Thirdly, statistical metrics were extracted and optimal feature subsets were obtained by Recursive Feature Elimination with Cross-Validation (RFECV). Subsequently, the classification models, including Logistic Regression (LR), Random Forest (RF), Support Vector Machine (SVM), and eXtreme Gradient Boosting (XGBoost), were trained. The experimental results show that the SVM model has the best performance, and its balance accuracy, recall, and AUC reach 0. 906, 0. 89, and 0. 932, respectively. Finally, we leveraged the Shapley additive explanation (SHAP) model for interpretability analysis. It revealed auditory stimuli have a more significant effect than visual stimuli in psychological resilience assessment. These findings suggested that pupil diameter could be a vital metric for assessing psychological resilience.

TMLR Journal 2023 Journal Article

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

• Aarohi Srivastava
• Abhinav Rastogi
• Abhishek Rao
• Abu Awal Md Shoeb
• Abubakar Abid
• Adam Fisch
• Adam R. Brown
• Adam Santoro

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG- bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood develop- ment, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google- internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.

ICML Conference 2023 Conference Paper

Fair and Accurate Decision Making through Group-Aware Learning

• Ramtin Hosseini
• Li Zhang
• Bhanu Garg
• Pengtao Xie

The integration of machine learning models in various real-world applications is becoming more prevalent to assist humans in their daily decision-making tasks as a result of recent advancements in this field. However, it has been discovered that there is a tradeoff between the accuracy and fairness of these decision-making tasks. In some cases, these AI systems can be unfair by exhibiting bias or discrimination against certain social groups, which can have severe consequences in real life. Inspired by one of the most well-known human learning skills called grouping, we address this issue by proposing a novel machine learning (ML) framework where the ML model learns to group a diverse set of problems into distinct subgroups to solve each subgroup using its specific sub-model. Our proposed framework involves three stages of learning, which are formulated as a three-level optimization problem: 1) grouping problems into subgroups, 2) learning group-specific sub-models for problem-solving, and 3) updating group assignments of training examples by minimizing validation loss. These three learning stages are performed end-to-end in a joint manner using gradient descent. To improve fairness and accuracy, we develop an efficient optimization algorithm to solve this three-level optimization problem. To further decrease the risk of overfitting in small datasets using our LBG method, we incorporate domain adaptation techniques in the second stage of training. We further apply our method to differentiable neural architecture search (NAS) methods.

YNIMG Journal 2023 Journal Article

Functional connectivity profiles of the default mode and visual networks reflect temporal accumulative effects of sustained naturalistic emotional experience

• Shuyue Xu
• Zhiguo Zhang
• Linling Li
• Yongjie Zhou
• Danyi Lin
• Min Zhang
• Li Zhang
• Gan Huang

Determining and decoding emotional brain processes under ecologically valid conditions remains a key challenge in affective neuroscience. The current functional Magnetic Resonance Imaging (fMRI) based emotion decoding studies are mainly based on brief and isolated episodes of emotion induction, while sustained emotional experience in naturalistic environments that mirror daily life experiences are scarce. Here we used 12 different 10-minute movie clips as ecologically valid emotion-evoking procedures in n = 52 individuals to explore emotion-specific fMRI functional connectivity (FC) profiles on the whole-brain level at high spatial resolution (432 parcellations including cortical and subcortical structures). Employing machine-learning based decoding and cross validation procedures allowed to investigate FC profiles contributing to classification that can accurately distinguish sustained happiness and sadness and that generalize across subjects, movie clips, and parcellations. Both functional brain network-based and subnetwork-based emotion classification results suggested that emotion manifests as distributed representation of multiple networks, rather than a single functional network or subnetwork. Further, the results showed that the Visual Network (VN) and Default Mode Network (DMN) associated functional networks, especially VN-DMN, exhibited a strong contribution to emotion classification. To further estimate the temporal accumulative effect of naturalistic long-term movie-based video-evoking emotions, we divided the 10-min episode into three stages: early stimulation (1∼200 s), middle stimulation (201∼400 s), and late stimulation (401∼600 s) and examined the emotion classification performance at different stimulation stages. We found that the late stimulation contributes most to the classification (accuracy=85.32%, F1-score=85.62%) compared to early and middle stimulation stages, implying that continuous exposure to emotional stimulation can lead to more intense emotions and further enhance emotion-specific distinguishable representations. The present work demonstrated that sustained happiness and sadness under naturalistic conditions are presented in emotion-specific network profiles and these expressions may play different roles in the generation and modulation of emotions. These findings elucidated the importance of network level adaptations for sustained emotional experiences during naturalistic contexts and open new venues for imaging network level contributions under naturalistic conditions.

AAAI Conference 2023 Conference Paper

Panoramic Video Salient Object Detection with Ambisonic Audio Guidance

• Xiang Li
• Haoyuan Cao
• Shijie Zhao
• Junlin Li
• Li Zhang
• Bhiksha Raj

Video salient object detection (VSOD), as a fundamental computer vision problem, has been extensively discussed in the last decade. However, all existing works focus on addressing the VSOD problem in 2D scenarios. With the rapid development of VR devices, panoramic videos have been a promising alternative to 2D videos to provide immersive feelings of the real world. In this paper, we aim to tackle the video salient object detection problem for panoramic videos, with their corresponding ambisonic audios. A multimodal fusion module equipped with two pseudo-siamese audio-visual context fusion (ACF) blocks is proposed to effectively conduct audio-visual interaction. The ACF block equipped with spherical positional encoding enables the fusion in the 3D context to capture the spatial correspondence between pixels and sound sources from the equirectangular frames and ambisonic audios. Experimental results verify the effectiveness of our proposed components and demonstrate that our method achieves state-of-the-art performance on the ASOD60K dataset.

AAAI Conference 2023 Conference Paper

PolarFormer: Multi-Camera 3D Object Detection with Polar Transformer

• Yanqin Jiang
• Li Zhang
• Zhenwei Miao
• Xiatian Zhu
• Jin Gao
• Weiming Hu
• Yu-Gang Jiang

3D object detection in autonomous driving aims to reason “what” and “where” the objects of interest present in a 3D world. Following the conventional wisdom of previous 2D object detection, existing methods often adopt the canonical Cartesian coordinate system with perpendicular axis. However, we conjugate that this does not fit the nature of the ego car’s perspective, as each onboard camera perceives the world in shape of wedge intrinsic to the imaging geometry with radical (non perpendicular) axis. Hence, in this paper we advocate the exploitation of the Polar coordinate system and propose a new Polar Transformer (PolarFormer) for more accurate 3D object detection in the bird’s-eye-view (BEV) taking as input only multi-camera 2D images. Specifically, we design a cross-attention based Polar detection head without restriction to the shape of input structure to deal with irregular Polar grids. For tackling the unconstrained object scale variations along Polar’s distance dimension, we further introduce a multi-scale Polar representation learning strategy. As a result, our model can make best use of the Polar representation rasterized via attending to the corresponding image observation in a sequence-to-sequence fashion subject to the geometric constraints. Thorough experiments on the nuScenes dataset demonstrate that our PolarFormer outperforms significantly state-of-the-art 3D object detection alternatives.

EAAI Journal 2023 Journal Article

Recognition of oil & gas pipelines operational states using graph network structural features

• Li Zhang
• Laurent Yeh
• Huai Su
• Karine Zeitouni
• Zhiheng Zuo
• Miao Li
• Luxin Jiang
• Lin Fan

The monitoring and recognition of operational states pattern is a crucial part for maintaining the safety, reliability and profitability of oil and gas pipeline systems. However, there are fewer methods to monitor the operational status of long-distance pipelines through operational data alone. In this paper, a purely data-driven approach is proposed for detecting and identifying the operational status of pipeline systems based on machine learning methods and the data log of pipelines. Firstly, a logic rule-based method is proposed to enrich the labels for each segment of operational data. Secondly, a change point-based detection model is used to detect the change of operational state in pipeline system or equipment. Then, a framework of oil pipeline operational pattern recognition methods based on graph structural features is proposed. Finally, the proposed model is applied to a real-world data from a pipeline system in China. Both the accuracy and the breadth of the recognition results can be improved by the use of real-time data validation and a human-machine interface. The results show that the precision of a change point-based detection model can reach more than 85% for different scenarios, and a reduction in missed rate of 17%–26%. Compared with the statistical feature-based method, the proposed method has improved the accuracy for all types of scenarios to a certain extent. The most significant improvement in recognition accuracy was achieved in the valve switch state and the combined state, with an increase of 30. 8% and 5% respectively.

AAAI Conference 2023 Conference Paper

Safety Verification of Nonlinear Systems with Bayesian Neural Network Controllers

• Xia Zeng
• Zhengfeng Yang
• Li Zhang
• Xiaochao Tang
• Zhenbing Zeng
• Zhiming Liu

Bayesian neural networks (BNNs) retain NN structures with a probability distribution placed over their weights. With the introduced uncertainties and redundancies, BNNs are proper choices of robust controllers for safety-critical control systems. This paper considers the problem of verifying the safety of nonlinear closed-loop systems with BNN controllers over unbounded-time horizon. In essence, we compute a safe weight set such that as long as the BNN controller is always applied with weights sampled from the safe weight set, the controlled system is guaranteed to be safe. We propose a novel two-phase method for the safe weight set computation. First, we construct a reference safe control set that constraints the control inputs, through polynomial approximation to the BNN controller followed by polynomial-optimization-based barrier certificate generation. Then, the computation of safe weight set is reduced to a range inclusion problem of the BNN on the system domain w.r.t. the safe control set, which can be solved incrementally and the set of safe weights can be extracted. Compared with the existing method based on invariant learning and mixed-integer linear programming, we could compute safe weight sets with larger radii on a series of linear benchmarks. Moreover, experiments on a series of widely used nonlinear control tasks show that our method can synthesize large safe weight sets with probability measure as high as 95% even for a large-scale system of dimension 7.

AAAI Conference 2023 Conference Paper

Self-Asymmetric Invertible Network for Compression-Aware Image Rescaling

• Jinhai Yang
• Mengxi Guo
• Shijie Zhao
• Junlin Li
• Li Zhang

High-resolution (HR) images are usually downscaled to low-resolution (LR) ones for better display and afterward upscaled back to the original size to recover details. Recent work in image rescaling formulates downscaling and upscaling as a unified task and learns a bijective mapping between HR and LR via invertible networks. However, in real-world applications (e.g., social media), most images are compressed for transmission. Lossy compression will lead to irreversible information loss on LR images, hence damaging the inverse upscaling procedure and degrading the reconstruction accuracy. In this paper, we propose the Self-Asymmetric Invertible Network (SAIN) for compression-aware image rescaling. To tackle the distribution shift, we first develop an end-to-end asymmetric framework with two separate bijective mappings for high-quality and compressed LR images, respectively. Then, based on empirical analysis of this framework, we model the distribution of the lost information (including downscaling and compression) using isotropic Gaussian mixtures and propose the Enhanced Invertible Block to derive high-quality/compressed LR images in one forward pass. Besides, we design a set of losses to regularize the learned LR images and enhance the invertibility. Extensive experiments demonstrate the consistent improvements of SAIN across various image rescaling datasets in terms of both quantitative and qualitative evaluation under standard image compression formats (i.e., JPEG and WebP). Code is available at https://github.com/yang-jin-hai/SAIN.

NeurIPS Conference 2023 Conference Paper

SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling

• Jiaxiang Dong
• Haixu Wu
• Haoran Zhang
• Li Zhang
• Jianmin Wang
• Mingsheng Long

Time series analysis is widely used in extensive areas. Recently, to reduce labeling expenses and benefit various tasks, self-supervised pre-training has attracted immense interest. One mainstream paradigm is masked modeling, which successfully pre-trains deep models by learning to reconstruct the masked content based on the unmasked part. However, since the semantic information of time series is mainly contained in temporal variations, the standard way of randomly masking a portion of time points will seriously ruin vital temporal variations of time series, making the reconstruction task too difficult to guide representation learning. We thus present SimMTM, a Simple pre-training framework for Masked Time-series Modeling. By relating masked modeling to manifold learning, SimMTM proposes to recover masked time points by the weighted aggregation of multiple neighbors outside the manifold, which eases the reconstruction task by assembling ruined but complementary temporal variations from multiple masked series. SimMTM further learns to uncover the local structure of the manifold, which is helpful for masked modeling. Experimentally, SimMTM achieves state-of-the-art fine-tuning performance compared to the most advanced time series pre-training methods in two canonical time series analysis tasks: forecasting and classification, covering both in- and cross-domain settings.

YNIMG Journal 2022 Journal Article

A new perspective on individual reliability beyond group effect for event-related potentials: A multisensory investigation and computational modeling

• Zhenxing Hu
• Zhiguo Zhang
• Zhen Liang
• Li Zhang
• Linling Li
• Gan Huang

The dominant approach in investigating the individual reliability for event-related potentials (ERPs) is to extract peak-related features at electrodes showing the strongest group effects. Such a peak-based approach implicitly assumes ERP components showing a stronger group effect are also more reliable, but this assumption has not been substantially validated and few studies have investigated the reliability of ERPs beyond peaks. In this study, we performed a rigorous evaluation of the test-retest reliability of ERPs collected in a multisensory and cognitive experiment from 82 healthy adolescents, each having two sessions. By comparing group effects and individual reliability, we found that a stronger group-level response in ERPs did not guarantee higher reliability. A perspective of neural oscillation should be adopted for the analysis of reliability. Further, by simulating ERPs with an oscillation-based computational model, we found that the consistency between group-level ERP responses and individual reliability was modulated by inter-subject latency jitter and inter-trial variability. The current findings suggest that the conventional peak-based approach may underestimate the individual reliability in ERPs and a neural oscillation perspective on ERP reliability should be considered. Hence, a comprehensive evaluation of the reliability of ERP measurements should be considered in individual-level neurophysiological trait evaluation and psychiatric disorder diagnosis.

NeurIPS Conference 2022 Conference Paper

DeepInteraction: 3D Object Detection via Modality Interaction

• Zeyu Yang
• Jiaqi Chen
• Zhenwei Miao
• Wei Li
• Xiatian Zhu
• Li Zhang

Existing top-performance 3D object detectors typically rely on the multi-modal fusion strategy. This design is however fundamentally restricted due to overlooking the modality-specific useful information and finally hampering the model performance. To address this limitation, in this work we introduce a novel modality interaction strategy where individual per-modality representations are learned and maintained throughout for enabling their unique characteristics to be exploited during object detection. To realize this proposed strategy, we design a DeepInteraction architecture characterized by a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder. Experiments on the large-scale nuScenes dataset show that our proposed method surpasses all prior arts often by a large margin. Crucially, our method is ranked at the first position at the highly competitive nuScenes object detection leaderboard.

IROS Conference 2022 Conference Paper

Hierarchical Road Topology Learning for Urban Mapless Driving

• Li Zhang
• Faezeh Tafazzoli
• Gunther Krehl
• Runsheng Xu
• Timo Rehfeld
• Manuel Schier
• Arunava Seal

The majority of current approaches in autonomous driving rely on High-Definition (HD) maps which detail the road geometry and surrounding area. Yet, this reliance is one of the obstacles to mass deployment of autonomous vehicles due to poor scalability of such prior maps. In this paper, we tackle the problem of online road map extraction via leveraging the sensory system aboard the vehicle itself. To this end, we design a structured model where a graph representation of the road network is generated in a hierarchical fashion within a fully convolutional network. The method is able to handle complex road topology and does not require a user in the loop.

AAAI Conference 2022 Conference Paper

Learning from Mistakes – a Framework for Neural Architecture Search

• Bhanu Garg
• Li Zhang
• Pradyumna Sridhara
• Ramtin Hosseini
• Eric Xing
• Pengtao Xie

Learning from one’s mistakes is an effective human learning technique where the learners focus more on the topics where mistakes were made, so as to deepen their understanding. In this paper, we investigate if this human learning strategy can be applied in machine learning. We propose a novel machine learning method called Learning From Mistakes (LFM), wherein the learner improves its ability to learn by focusing more on the mistakes during revision. We formulate LFM as a three-stage optimization problem: 1) learner learns; 2) learner re-learns focusing on the mistakes, and; 3) learner validates its learning. We develop an efficient algorithm to solve the LFM problem. We apply the LFM framework to neural architecture search on CIFAR-10, CIFAR-100, and Imagenet. Experimental results strongly demonstrate the effectiveness of our model.

YNIMG Journal 2022 Journal Article

M3CV: A multi-subject, multi-session, and multi-task database for EEG-based biometrics challenge

• Gan Huang
• Zhenxing Hu
• Weize Chen
• Shaorong Zhang
• Zhen Liang
• Linling Li
• Li Zhang
• Zhiguo Zhang

CV dataset and the EEG-based biometric competition aim to provide the opportunity to develop advanced machine learning algorithms for achieving an in-depth understanding of the commonality and variability of EEG signals across subjects, sessions, and tasks.

IS Journal 2022 Journal Article

Neural Graph for Personalized Tag Recommendation

• Yonghong Yu
• Xuewen Chen
• Li Zhang
• Rong Gao
• Haiyan Gao

Traditional personalized tag recommendation methods cannot guarantee that the coupling relationships hidden in the interactions among entities are effectively encoded in the process of learning the representations of entities, resulting in insufficient expressive capacity for characterizing the preferences or attributes of entities. In this article, we first propose a graph neural networks boosted personalized tag recommendation model, namely NGTR, which integrates the graph neural networks into the pairwise interaction tensor factorization model. Specifically, we exploit the graph neural networks to capture the coupling relationships and integrate the coupling relationships into the learning of representations of entities by transmitting and assembling the representations of neighbors along the interaction graphs. In addition, we also propose a light graph neural networks boosted personalized tag recommendation model, namely LNGTR. Different from NGTR, our proposed LNGTR model removes feature transformation and nonlinear activation components as well as adopts the weighted sum of the embeddings learned at all layers as the final embedding. Experimental results on real world datasets show that our proposed personalized tag recommendation models outperform the traditional tag recommendation methods.

IJCAI Conference 2022 Conference Paper

Region-Aware Metric Learning for Open World Semantic Segmentation via Meta-Channel Aggregation

• Hexin Dong
• Zifan Chen
• Mingze Yuan
• Yutong Xie
• Jie Zhao
• Fei Yu
• Bin Dong
• Li Zhang

As one of the most challenging and practical segmentation tasks, open-world semantic segmentation requires the model to segment the anomaly regions in the images and incrementally learn to segment out-of-distribution (OOD) objects, especially under a few-shot condition. The current state-of-the-art (SOTA) method, Deep Metric Learning Network (DMLNet), relies on pixel-level metric learning, with which the identification of similar regions having different semantics is difficult. Therefore, we propose a method called region-aware metric learning (RAML), which first separates the regions of the images and generates region-aware features for further metric learning. RAML improves the integrity of the segmented anomaly regions. Moreover, we propose a novel meta-channel aggregation (MCA) module to further separate anomaly regions, forming high-quality sub-region candidates and thereby improving the model performance for OOD objects. To evaluate the proposed RAML, we have conducted extensive experiments and ablation studies on Lost And Found and Road Anomaly datasets for anomaly segmentation and the CityScapes dataset for incremental few-shot learning. The results show that the proposed RAML achieves SOTA performance in both stages of open world segmentation. Our code and appendix are available at https: //github. com/czifan/RAML.

AAAI Conference 2021 Conference Paper

DAST: Unsupervised Domain Adaptation in Semantic Segmentation Based on Discriminator Attention and Self-Training

• Fei Yu
• Mo Zhang
• Hexin Dong
• Sheng Hu
• Bin Dong
• Li Zhang

Unsupervised domain adaption has recently been used to reduce the domain shift, which would ultimately improve the performance of semantic segmentation on unlabeled realworld data. In this paper, we follow the trend to propose a novel method to reduce the domain shift using strategies of discriminator attention and self-training. The discriminator attention strategy contains a two-stage adversarial learning process, which explicitly distinguishes the well-aligned (domain-invariant) and poorly-aligned (domain-specific) features, and then guides the model to focus on the latter. The self-training strategy adaptively improves the decision boundary of the model for target domain, which implicitly facilitates the extraction of domain-invariant features. By combining the two strategies, we find a more effective way to reduce the domain shift. Extensive experiments demonstrate the effectiveness of our proposed method on numerous benchmark datasets.

YNIMG Journal 2021 Journal Article

Designing individual-specific and trial-specific models to accurately predict the intensity of nociceptive pain from single-trial fMRI responses

• Qianqian Lin
• Gan Huang
• Linling Li
• Li Zhang
• Zhen Liang
• Ahmed M. Anter
• Zhiguo Zhang

Using machine learning to predict the intensity of pain from fMRI has attracted rapidly increasing interests. However, due to remarkable inter- and intra-individual variabilities in pain responses, the performance of existing fMRI-based pain prediction models is far from satisfactory. The present study proposed a new approach which can design a prediction model specific to each individual or each experimental trial so that the specific model can achieve more accurate prediction of the intensity of nociceptive pain from single-trial fMRI responses. More precisely, the new approach uses a supervised k-means method on nociceptive-evoked fMRI responses to cluster individuals or trials into a set of subgroups, each of which has similar and consistent fMRI activation patterns. Then, for a new test individual/trial, the proposed approach chooses one subgroup of individuals/trials, which has the closest fMRI patterns to the test individual/trial, as training samples to train an individual-specific or a trial-specific pain prediction model. The new approach was tested on a nociceptive-evoked fMRI dataset and achieved significantly higher prediction accuracy than conventional non-specific models, which used all available training samples to train a model. The generalizability of the proposed approach is further validated by training specific models on one dataset and testing these models on an independent new dataset. This proposed individual-specific and trial-specific pain prediction approach has the potential to be used for the development of individualized and precise pain assessment tools in clinical practice.

YNICL Journal 2021 Journal Article

Fixel-based evidence of microstructural damage in crossing pathways improves language mapping in Post-stroke aphasia

• Jie Zhang
• Weihao Zheng
• Desheng Shang
• Yating Chen
• Shuchang Zhong
• Jing Ye
• Lingling Li
• Yamei Yu

BACKGROUND: The complex crossing-fiber characteristics in the dual-stream system have been ignored by traditional diffusion tensor models regarding disconnections in post-stroke aphasia. It is valuable to identify microstructural damage of crossing-fiber pathways and reveal accurate fiber-specific language mapping in patients with aphasia. METHODS: This cross-sectional study collected magnetic resonance imaging data from 29 participants with post-stroke aphasia in the subacute stage and from 33 age- and sex-matched healthy controls. Fixel-based analysis was performed to examine microstructural fiber density (FD) and bundle cross-section alterations of specific fiber populations in crossing-fiber regions. Group comparisons were performed, and relationships with language scores were assessed. RESULTS: The aphasic group exhibited significant fixel-wise FD reductions in the dual-stream tracts, including the left inferior fronto-occipital fasciculus (IFOF), arcuate fasciculus, and superior longitudinal fasciculus (SLF) III (family-wise-error-corrected p < 0.05). Voxel- and fixel-wise comparisons revealed mismatched distributions in regions with crossing-fiber nexuses. Fixel-wise correlation analyses revealed significant associations between comprehension impairment and reduced FD in the temporal and frontal segments of the left IFOF, and also mapped naming ability to the IFOF. Average features along the whole course of dominant tracts assessed with tract-wise analyses attributed word-level comprehension to the IFOF (r = 0.723, p < 0.001) and revealed a trend-level correlation between sentence-level comprehension and FD of the SLF III (r = 0.451, p = 0.021). The mean FD of the uncinate fasciculus (UF) and IFOF correlated with total and picture naming scores, and the IFOF also correlated with responsive naming subdomains (Bonferroni corrected p < 0.05). CONCLUSIONS: FD reductions of dual streams suggest that intra-axonal volume reduction constitutes the microstructural damage of white matter integrity in post-stroke aphasia. Fixel-based analysis provides a complementary method of language mapping that identifies fiber-specific tracts in the left hemisphere language network with greater specificity than voxel-based analysis. It precisely locates the precise segments of the IFOF for comprehension, yields fiber-specific evidence for the debated UF-naming association, and reveals dissociative subdomain associations with distinct tracts.

AAAI Conference 2021 Conference Paper

Learning a Few-shot Embedding Model with Contrastive Learning

• Chen Liu
• Yanwei Fu
• Chengming Xu
• Siqian Yang
• Jilin Li
• Chengjie Wang
• Li Zhang

Few-shot learning (FSL) aims to recognize target classes by adapting the prior knowledge learned from source classes. Such knowledge usually resides in a deep embedding model for a general matching purpose of the support and query image pairs. The objective of this paper is to repurpose the contrastive learning for such matching to learn a few-shot embedding model. We make the following contributions: (i) We investigate the contrastive learning with Noise Contrastive Estimation (NCE) in a supervised manner for training a fewshot embedding model; (ii) We propose a novel contrastive training scheme dubbed infoPatch, exploiting the patch-wise relationship to substantially improve the popular infoNCE; (iii) We show that the embedding learned by the proposed infoPatch is more effective; (iv) Our model is thoroughly evaluated on few-shot recognition task; and demonstrates state-ofthe-art results on miniImageNet and appealing performance on tieredImageNet, Fewshot-CIFAR100 (FC-100).

JBHI Journal 2021 Journal Article

Learning-Based Computer-Aided Prescription Model for Parkinson's Disease: A Data-Driven Perspective

• Yinghuan Shi
• Wanqi Yang
• Kim-Han Thung
• Hao Wang
• Yang Gao
• Yang Pan
• Li Zhang
• Dinggang Shen

In this article, we study a novel problem: “automatic prescription recommendation for PD patients. ” To realize this goal, we first build a dataset by collecting 1) symptoms of PD patients, and 2) their prescription drug provided by neurologists. Then, we build a novel computer-aided prescription model by learning the relation between observed symptoms and prescription drug. Finally, for the new coming patients, we could recommend (predict) suitable prescription drug on their observed symptoms by our prescription model. From the methodology part, our proposed model, namely Prescription viA Learning lAtent Symptoms (PALAS), could recommend prescription using the multi-modality representation of the data. In PALAS, a latent symptom space is learned to better model the relationship between symptoms and prescription drug, as there is a large semantic gap between them. Moreover, we present an efficient alternating optimization method for PALAS. We evaluated our method using the data collected from 136 PD patients at Nanjing Brain Hospital, which can be regarded as a large dataset in PD research community. The experimental results demonstrate the effectiveness and clinical potential of our method in this recommendation task, if compared with other competing methods.

AAAI Conference 2021 Conference Paper

Partial-Label and Structure-constrained Deep Coupled Factorization Network

• Yan Zhang
• Zhao Zhang
• Yang Wang
• Zheng Zhang
• Li Zhang
• Shuicheng Yan
• Meng Wang

In this paper, we technically propose an enriched prior guided framework, called Dual-constrained Deep Semi-Supervised Coupled Factorization Network (DS2 CF-Net), for discovering hierarchical coupled data representation. To extract hidden deep features, DS2 CF-Net is formulated as a partial-label and geometrical structure-constrained framework. Specifically, DS2 CF-Net designs a deep factorization architecture using multilayers of linear transformations, which can coupled update both the basis vectors and new representations in each layer. To enable learned deep representations and coefficients to be discriminative, we also consider enriching the supervised prior by joint deep coefficients-based label prediction and then incorporate the enriched prior information as additional label and structure constraints. The label constraint can enable the intra-class samples to have same coordinate in feature space, and the structure constraint forces the coefficients in each layer to be block-diagonal so that the enriched prior using the self-expressive label propagation are more accurate. Our network also integrates the adaptive dualgraph learning to retain the local structures of both data and feature manifolds in each layer. Extensive experiments on image datasets demonstrate the effectiveness of DS2 CF-Net for representation learning and clustering.

EAAI Journal 2021 Journal Article

Pixel-level automatic annotation for forest fire image

• Xubing Yang
• Run Chen
• Fuquan Zhang
• Li Zhang
• Xijian Fan
• Qiaolin Ye
• Liyong Fu

We propose an automatic annotation method for forest fire images in the level of pixel, where supervise information is introduced by interactive convex hulls. Instead of usual rectangle-/regular-shaped regions, we propose a convex hull algorithm for visually selecting polygonal (irregular) fire and no-fire regions. Guided by the goals of forest fire monitoring systems: high fire detection rate (true-positive) and then low false alarm rate (false-positive), we construct a k-nearest neighbor (kNN) based KD-tree to speed annotation. Compared to state-of-the-art, the proposed method not only widens the view of fire detection from conventional two-class to multi-class classification problem to meet complex forest image background, but also relaxes the limit of i. i. d (independent and identical distribution) hypothesis on machine learning methods. Furthermore, it is simple to use, which just relies on pixel information and avoids considering additional auxiliary features from multiple color spaces. Experimental evaluations are carrying on forest fire images, MIVIA dead-directional videos, and more challenging omni-directional videos. The comparison demonstrates that the proposed pixel-level annotation method is able to achieve higher fire detection rate and lower false alarm rate at the same time.

NeurIPS Conference 2021 Conference Paper

Progressive Coordinate Transforms for Monocular 3D Object Detection

• Li Wang
• Li Zhang
• Yi Zhu
• Zhi Zhang
• Tong He
• Mu Li
• Xiangyang Xue

Recognizing and localizing objects in the 3D space is a crucial ability for an AI agent to perceive its surrounding environment. While significant progress has been achieved with expensive LiDAR point clouds, it poses a great challenge for 3D object detection given only a monocular image. While there exist different alternatives for tackling this problem, it is found that they are either equipped with heavy networks to fuse RGB and depth information or empirically ineffective to process millions of pseudo-LiDAR points. With in-depth examination, we realize that these limitations are rooted in inaccurate object localization. In this paper, we propose a novel and lightweight approach, dubbed {\em Progressive Coordinate Transforms} (PCT) to facilitate learning coordinate representations. Specifically, a localization boosting mechanism with confidence-aware loss is introduced to progressively refine the localization prediction. In addition, semantic image representation is also exploited to compensate for the usage of patch proposals. Despite being lightweight and simple, our strategy allows us to establish a new state-of-the-art among the monocular 3D detectors on the competitive KITTI benchmark. At the same time, our proposed PCT shows great generalization to most coordinate-based 3D detection frameworks.

NeurIPS Conference 2021 Conference Paper

SOFT: Softmax-free Transformer with Linear Complexity

• Jiachen Lu
• Jinghan Yao
• Junge Zhang
• Xiatian Zhu
• Hang Xu
• Weiguo Gao
• Chunjing Xu
• Tao Xiang

Vision transformers (ViTs) have pushed the state-of-the-art for various visual recognition tasks by patch-wise image tokenization followed by self-attention. However, the employment of self-attention modules results in a quadratic complexity in both computation and memory usage. Various attempts on approximating the self-attention computation with linear complexity have been made in Natural Language Processing. However, an in-depth analysis in this work shows that they are either theoretically flawed or empirically ineffective for visual recognition. We further identify that their limitations are rooted in keeping the softmax self-attention during approximations. Specifically, conventional self-attention is computed by normalizing the scaled dot-product between token feature vectors. Keeping this softmax operation challenges any subsequent linearization efforts. Based on this insight, for the first time, a softmax-free transformer or SOFT is proposed. To remove softmax in self-attention, Gaussian kernel function is used to replace the dot-product similarity without further normalization. This enables a full self-attention matrix to be approximated via a low-rank matrix decomposition. The robustness of the approximation is achieved by calculating its Moore-Penrose inverse using a Newton-Raphson method. Extensive experiments on ImageNet show that our SOFT significantly improves the computational efficiency of existing ViT variants. Crucially, with a linear complexity, much longer token sequences are permitted in SOFT, resulting in superior trade-off between accuracy and complexity.

JBHI Journal 2020 Journal Article

A Hybrid Global-Local Representation CNN Model for Automatic Cataract Grading

• Xi Xu
• Linglin Zhang
• Jianqiang Li
• Yu Guan
• Li Zhang

Cataract is one of the most serious eye diseases leading to blindness. Early detection and treatment can reduce the rate of blindness in cataract patients. However, the professional knowledge of ophthalmologists is necessary for the clinical cataract detection. Therefore, the potential costs may make it difficult for the widespread use of cataract detection to prevent blindness. Artificial intelligence assisted diagnosis based on medical images has attracted more and more attention of researchers. Many studies have focused on the use of pre-defined feature sets for cataract classification, but the predefined feature sets may be incomplete or redundant. On account of the aforementioned issues, some studies have proposed deep learning methods to automatically extract image features, but all based on global features and none has analyzed the layer-by-layer transformation process of the middle-tier features. This paper uses convolutional neural networks (CNN) to learn useful features directly from input data, and deconvolution network method is employed to investigate how CNN characterizes cataract layer-by-layer. We found that compared to the global feature set, the detail vascular information, which is lost after multi-layer convolution calculation also plays an important role in cataract grading task. And this finding fits with the morphological definition of fundus image. Through the finding, we gained insights into the design of hybrid global-local feature representation model to improve the recognition performance of automatic cataract grading.

YNICL Journal 2020 Journal Article

Altered resting-state functional connectivity in adolescents is associated with PTSD symptoms and trauma exposure

• Jony Sheynin
• Elizabeth R. Duval
• Yana Lokshina
• J. Cobb Scott
• Mike Angstadt
• Daniel Kessler
• Li Zhang
• Raquel E. Gur

Alterations in resting-state functional connectivity (rsFC) have been demonstrated in Posttraumatic Stress Disorder (PTSD). However, such reports have primarily focused on adult participants, whereas findings in adolescents with PTSD are mixed and not entirely consistent with the adult literature. Here, we examined rsFC in a non-treatment seeking adolescent sample with posttraumatic stress symptoms (PTSS; n = 59) relative to asymptomatic controls (n = 226). We also examined differences between trauma-exposed and non-exposed control subgroups (TEC n = 73 and Non-TEC n = 153) to examine alterations associated with more general trauma exposure. Finally, we compared the PTSS and TEC groups, to confirm that the reported alterations in PTSS were not driven by trauma exposure. Using a seed-based approach, we examined connectivity of default-mode (DMN) and salience (SN) networks, where alterations have been previously reported. Results suggest that PTSS are associated with less within-DMN connectivity and greater SN-DMN connectivity, as well as altered connectivity with attention regions. Trauma exposure is associated with greater within-SN connectivity. Additionally, we report findings from exploratory connectome-based analysis, which demonstrate a number of topological alterations within DMN in the PTSS group. Overall, our findings replicate prior reports of altered rsFC in PTSD and extend them to non-treatment seeking, trauma-exposed adolescents, who did or did not report PTSS. They specifically highlight SN-DMN desegregation, lower within-DMN and greater within-SN connectivity, as well as altered connectivity with attention regions, in trauma-exposed adolescents. Future research is required to confirm that adolescents with diagnosed PTSD have similar/exacerbated connectivity patterns.

AAAI Conference 2020 Conference Paper

Consistent Video Style Transfer via Compound Regularization

• Wenjing Wang
• Jizheng Xu
• Li Zhang
• Yue Wang
• Jiaying Liu

Recently, neural style transfer has drawn many attentions and significant progresses have been made, especially for image style transfer. However, flexible and consistent style transfer for videos remains a challenging problem. Existing training strategies, either using a significant amount of video data with optical flows or introducing single-frame regularizers, have limited performance on real videos. In this paper, we propose a novel interpretation of temporal consistency, based on which we analyze the drawbacks of existing training strategies; and then derive a new compound regularization. Experimental results show that the proposed regularization can better balance the spatial and temporal performance, which supports our modeling. Combining with the new cost formula, we design a zero-shot video style transfer framework. Moreover, for better feature migration, we introduce a new module to dynamically adjust inter-channel distributions. Quantitative and qualitative results demonstrate the superiority of our method over other state-of-the-art style transfer methods. Our project is publicly available at: https: //daooshee. github. io/CompoundVST/.

NeurIPS Conference 2020 Conference Paper

Rankmax: An Adaptive Projection Alternative to the Softmax Function

• Weiwei Kong
• Walid Krichene
• Nicolas Mayoraz
• Steffen Rendle
• Li Zhang

Several machine learning models involve mapping a score vector to a probability vector. Usually, this is done by projecting the score vector onto a probability simplex, and such projections are often characterized as Lipschitz continuous approximations of the argmax function, whose Lipschitz constant is controlled by a parameter that is similar to a softmax temperature. The aforementioned parameter has been observed to affect the quality of these models and is typically either treated as a constant or decayed over time. In this work, we propose a method that adapts this parameter to individual training examples. The resulting method exhibits desirable properties, such as sparsity of its support and numerically efficient implementation, and we find that it significantly outperforms competing non-adaptive projection methods. In our analysis, we also derive the general solution of (Bregman) projections onto the (n, k)-simplex, a result which may be of independent interest.

AAAI Conference 2020 Conference Paper

Universal Value Iteration Networks: When Spatially-Invariant Is Not Universal

• Li Zhang
• Xin Li
• Sen Chen
• Hongyu Zang
• Jie Huang
• Mingzhong Wang

In this paper, we first formally define the problem set of spatially invariant Markov Decision Processes (MDPs), and show that Value Iteration Networks (VIN) and its extensions are computationally bounded to it due to the use of the convolution kernel. To generalize VIN to spatially variant MDPs, we propose Universal Value Iteration Networks (UVIN). In comparison with VIN, UVIN automatically learns a flexible but compact network structure to encode the transition dynamics of the problems and support the differentiable planning module. We evaluate UVIN with both spatially invariant and spatially variant tasks, including navigation in regular maze, chessboard maze, and Mars, and Minecraft item syntheses. Results show that UVIN can achieve similar performance as VIN and its extensions on spatially invariant tasks, and significantly outperforms other models on more general problems.

YNIMG Journal 2019 Journal Article

A novel training-free externally-regulated neurofeedback (ER-NF) system using phase-guided visual stimulation for alpha modulation

• Gan Huang
• Jia Liu
• Linling Li
• Li Zhang
• Yixuan Zeng
• Lijie Ren
• Shiqing Ye
• Zhiguo Zhang

The efficacy of neurofeedback is a point of great controversy, because a certain proportion of users cannot properly regulate their brain activities and thereby fail to benefit from neurofeedback. To address the neurofeedback inefficacy problem, the present study is aimed to design and implement a new neurofeedback system that can more effectively and consistently regulate users’ brain activities than the conventional way of training users to voluntarily regulate brain activities. The new neurofeedback system delivers external visual stimuli continuously at a specific alpha phase, which is real-time decoded from ongoing alpha wave, to regulate the alpha wave. Experimental results show that the proposed training-free externally-regulated neurofeedback (ER-NF) system can achieve consistent (effective in almost all sessions for almost all users), flexible (either increasing or decreasing peak alpha frequency and alpha power), and immediate (taking or losing effect immediately after stimulation is on or off) modulation effects on alpha wave. Therefore, the ER-NF system holds great potential to be able to more reliably and flexibly modulate cognition and behavior.

YNIMG Journal 2019 Journal Article

Modeling glymphatic system of the brain using MRI

• Esmaeil Davoodi-Bojd
• Guangliang Ding
• Li Zhang
• Qingjiang Li
• Lian Li
• Michael Chopp
• Zhenggang Zhang
• Quan Jiang

The glymphatic system is functional waste clearance path from the brain parenchyma through dynamic exchange of cerebrospinal fluid (CSF) with interstitial fluid (ISF). Impairment of glymphatic waste clearance is involved in the development of neurodegenerative conditions. Despite many recent studies investigating the glymphatic system, few studies have tried to use a mathematical model to describe this system, quantitatively. In this study, we aim to model the glymphatic system from the kinetics of Gd-DTPA tracer measured using MRI in order to: 1) map the glymphatic system path, 2) derive kinetic parameters of the glymphatic system, and 3) provide quantitative maps of the structure and function of this system. In the proposed model, the brain is clustered to similar regions with respect to the profile of contrast agent (CA) density measured by MRI. Then, each region is described as a two-compartment kinetic model ‘derived from’ or ‘clears to’ its neighbors with local input function. We thus fit our model to the local cerebral regions rather than to the averaged time signal curve (TSC) of the whole brain. The estimated parameters showed distinctive differences between diabetes mellitus (DM) and control rats. The results suggest that in a typical DM brain the CSF bulk speed in the para-vasculature network is low. In addition, the resulting maps indicate that there may be increased binding and decreased absorbing of large molecules in a diabetic compared with a non-diabetic brain. The important contribution of this work was to fit the model to the local regions rather than to the averaged time signal curve (TSC) of the whole brain. This enabled us to derive quantitative maps of the glymphatic system from MRI.

AAMAS Conference 2017 Conference Paper

Firefly-based Facial Expression Recognition: Extended Abstract

• Kamlesh Mistry
• Li Zhang
• Yifeng Zeng
• Mengda He

AAMAS Conference 2016 Conference Paper

A Vision Enriched Intelligent Agent with Image Description Generation (Demonstration)

• Li Zhang
• Ben Fielding
• Philip Kinghorn
• Kamlesh Mistry

In this paper, we present an intelligent conversational agent enriched with automatic image understanding and facial expression recognition using state-of-the-art machine learning techniques for the advancement of autonomous interaction with the elderly or infirm. The agent is developed to conduct health and emotion well-being monitoring for the elderly. It is not only capable of conducting question-answering via speech-based interaction, but also able to provide analysis of the user’s surroundings, emotional states, hazards and fall actions via visual data. The agent is accessible from a web browser and can be communicated with via voice or text means, with a webcam required for the visual analysis functionality. The system has been evaluated with diverse real-life images to prove its efficiency.

NeurIPS Conference 2015 Conference Paper

Nearly Optimal Private LASSO

• Kunal Talwar
• Abhradeep Guha Thakurta
• Li Zhang

We present a nearly optimal differentially private version of the well known LASSO estimator. Our algorithm provides privacy protection with respect to each training data item. The excess risk of our algorithm, compared to the non-private version, is $\widetilde{O}(1/n^{2/3})$, assuming all the input data has bounded $\ell_\infty$ norm. This is the first differentially private algorithm that achieves such a bound without the polynomial dependence on $p$ under no addition assumption on the design matrix. In addition, we show that this error bound is nearly optimal amongst all differentially private algorithms.

AAAI Conference 2015 Conference Paper

Sampling Representative Users from Large Social Networks

• Jie Tang
• Chenhui Zhang
• Keke Cai
• Li Zhang
• Zhong Su

Finding a subset of users to statistically represent the original social network is a fundamental issue in Social Network Analysis (SNA). The problem has not been extensively studied in existing literature. In this paper, we present a formal definition of the problem of sampling representative users from social network. We propose two sampling models and theoretically prove their NP-hardness. To efficiently solve the two models, we present an efficient algorithm with provable approximation guarantees. Experimental results on two datasets show that the proposed models for sampling representative users significantly outperform (+6%-23% in terms of Precision@100) several alternative methods using authority or structure information only. The proposed algorithms are also effective in terms of time complexity. Only a few seconds are needed to sampling 300 representative users from a network of 100, 000 users. All data and codes are publicly available. 1

IS Journal 2014 Journal Article

Pervasive Service Bus: Smart SOA Infrastructure for Ambient Intelligence

• Gang Pan
• Li Zhang
• Zhaohui Wu
• Shijian Li
• Laurence Yang
• Man Lin
• Yuanchun Shi

Ambient intelligence (AmI) aims to make our everyday environments intelligent--that is, sensitive, adaptive, and responsive to the presence of people--in a transparent manner. Several challenges exist to building an efficient infrastructure for AmI, including interoperation of heterogeneous systems, intelligence for anticipatory user assistance, adaptability to dynamic environments for good user experience, and scalability to additional users and spaces. Here, the authors propose Pervasive Service Bus (PSB), a smart service-oriented architecture (SOA) framework for AmI spaces that models all computing activities as unified pervasive services. They present an online planning algorithm to adapt service flows to contexts and user tasks. PSB employs a sub-bus-based layout to maintain efficiency in large-scale service interactions. They also discuss their results in evaluating PSB's performance in a Smart Home testbed.

EAAI Journal 2013 Journal Article

A real-time dynamic optimal guidance scheme using a general regression neural network

• M.A. Hossain
• A.A.M. Madkour
• K.P. Dahal
• Li Zhang

This paper presents an investigation into the challenges in implementing a hard real-time optimal non-stationary system using general regression neural network (GRNN). This includes investigation into the dynamics of the problem domain, discretisation of the problem domain to reduce the computational complexity, parameters selection of the optimization algorithm, convergence guarantee for real-time solution and off-line optimization for real-time solution. In order to demonstrate these challenges, this investigation considers a real-time optimal missile guidance algorithm using GRNN to achieve an accurate interception of the maneuvering targets in three-dimension. Evolutionary Genetic Algorithms (GAs) are used to generate optimal guidance training data set for a large missile defense space to train the GRNN. The Navigation Constant of the Proportional Navigation Guidance and the target position at launching are considered for optimization using GAs. This is achieved by minimizing the miss distance and missile flight time. Finally, the merits of the proposed schemes for real-time accurate interception are presented and discussed through a set of experiments.

AAMAS Conference 2013 Conference Paper

Affect Detection from Semantic and Metaphorical Interpretation of Virtual Drama

• Li Zhang
• John Barnden
• Ming Jiang

We have developed an intelligent agent to engage with users in virtual drama improvisation previously. The agent was able to perform sentence-level affect detection especially from inputs with strong emotional indicators. In this research, we employ latent semantic analysis to interpret emotional expressions with vague affect indicators and ambiguous audiences. Latent semantic analysis is thus used to perform topic theme detection and target audience identification for such inputs. Then we also discuss how affect is detected for such inputs without strong emotional indicators with the consideration of emotions expressed by the intended audiences and relationships between speakers and audiences. This work also proves to be effective in recognizing metaphorical phenomena. Moreover, uncertainty-based active learning is also employed to deal with more open-ended and imbalanced affect detection tasks. Overall, this work enables the AI agent to deal with challenging issues in affect detection tasks.

AAMAS Conference 2013 Conference Paper

Multimodal Intelligent Affect Detection with Kinect

• Yang Zhang
• Li Zhang
• Alamgir Hossain

Communication between human beings involves complex and rich means. In the past decades, computers have successfully supported human in a variety of tasks such as calculating and memorizing. However, when confronted with the demand of multimodal interaction with users, can these indispensable partners make us satisfied? This research might answer this question.

IJCAI Conference 2011 Conference Paper

Affect Sensing in Metaphorical Phenomena and Dramatic Interaction Context

• Li Zhang

Metaphorical interpretation and affect detection using context profiles from open-ended text input are challenging in affective language processing field. In this paper, we explore recognition of a few typical affective metaphorical phenomena and context-based affect sensing using the modeling of speakers' improvisational mood and other participants' emotional influence to the speaking character under the improvisation of loose scenarios. The overall updated affect detection module is embedded in an AI agent. The new developments have enabled the AI agent to perform generally better in affect sensing tasks. The work emphasizes the conference themes on affective dialogue processing, human-agent interaction and intelligent user interfaces.

TCS Journal 2011 Journal Article

Proportional response dynamics in the Fisher market

• Li Zhang

We show that the proportional response dynamics, a utility based distributed dynamics, converges to the market equilibrium in the Fisher market with constant elasticity of substitution (CES) utility functions. By the proportional response dynamics, each buyer allocates his budget proportional to the utility he receives from each good in the previous time period. Unlike the tâtonnement process and its variants, the proportional response dynamics is a large step discrete dynamics, and the buyers do not solve any optimization problem at each step. In addition, the goods are always cleared and assigned to the buyers proportional to their bids at each step. Despite its simplicity, the dynamics converges fast for strictly concave CES utility functions, matching the best upper-bound of computing the market equilibrium via the solution of a global convex optimization problem.

AAMAS Conference 2008 Conference Paper

A �Companion' ECA with Planning and Activity Modelling

• Marc Cavazza
• Cameron Smith
• Daniel Charlton
• Li Zhang
• Jaakko Hakulinen
• Markku Turunen

In this paper, we describe the development of an Embodied Conversational Agent (ECA) implementing the concept of a companion, i. e. an agent supporting the persistent representation of user activities and dialogue-based communication with the user. This first experiment implements a Health and Fitness companion aimed at promoting a healthier lifestyle. The system operates by generating an ‘ideal’ plan of daily activities from background knowledge and dialogue interaction with the user. This plan then becomes an activity model, which will later be instantiated by reports from the user and analysed by the agent from the perspective of initial objectives. At various stages of the day, the plan can still be adapted through further dialogue. The agent is embodied using a wireless rabbit (Nabaztag™) device situated in the user’s home. After describing the planning component, based on Hierarchical Task Networks (HTN) and the spoken dialogue system, we present a working example from the system illustrating its behaviour through various phases of user activity generation, updating and re-planning.

AAMAS Conference 2008 Conference Paper

An Embodied Conversational Agent as a Lifestyle Advisor

• Cameron Smith
• Daniel Charlton
• Li Zhang
• Jaakko Hakulinen
• Markku Turunen
• Marc Cavazza

Persistent Embodied Conversational Agents (ECA) can be used to assist users in their daily activities. We introduce the Health and Fitness Companion (HFC), which is a conversational system aimed at promoting a healthier lifestyle. The system is embodied using the NabaztagTM device, a wireless plastic rabbit supporting multimodal input and output. The HFC integrates a spoken dialogue system previously developed by some of the authors, and a cognitive model, based on Hierarchical Task Network (HTN) planning, which enables the ECA to reason on the user activities and generate plans for recommended activities. A typical demonstration scenario consists in two short conversation sessions between the HFC and the user. During the first the HFC helps the user to plan her day ahead, while the second involves reporting the activities actually carried out. The system has undergone early tests with generic users, which have demonstrated its usability and stability (89% successful dialogue completions).

YNIMG Journal 2007 Journal Article

Neural basis of cultural influence on self-representation

• Ying Zhu
• Li Zhang
• Jin Fan
• Shihui Han

Culture affects the psychological structure of self and results in two distinct types of self-representation (Western independent self and East Asian interdependent self). However, the neural basis of culture–self interaction remains unknown. We used fMRI to measured brain activity from Western and Chinese subjects who judged personal trait adjectives regarding self, mother or a public person. We found that the medial prefrontal cortex (MPFC) and anterior cingulate cortex (ACC) showed stronger activation in self- than other-judgment conditions for both Chinese and Western subjects. However, relative to other-judgments, mother-judgments activated MPFC in Chinese but not in Western subjects. Our findings suggest that Chinese individuals use MPFC to represent both the self and the mother whereas Westerners use MPFC to represent exclusively the self, providing neuroimaging evidence that culture shapes the functional anatomy of self-representation.

AAAI Conference 2007 Conference Paper

Photometric and Geometric Restoration of Document Images Using Inpainting and Shape-from-Shading

• Li Zhang

The popularity of current hand-held digital imaging devices such as camera phones, PDAs, camcorders has promoted the use of digital cameras to capture document images for daily information recording purpose. However, the captured images often contain photometric and geometric distortions when the documents are of non-planar shapes, which cause significant problems to various document image analysis (DIA) tasks such as OCR. In this paper, we propose a restoration framework that removes both photometric and geometric distortions in smoothly warped document images to facilitate human perception and machine recognition. First, the photometric distortions are corrected by separating the shading image from the reflectance image using inpainting and surface fitting techniques. Next, a 2-pass Shape-from-Shading (SFS) method is exploited to recover the document’s surface shape based on the extracted shading image. Once the document’s shape is obtained, the geometric distortions are rectified through a physically-based flattening process. Experiments on real document images show the performance of each sub-task and demonstrate a complete solution to the restoration of physically-distorted document images.

YNIMG Journal 2006 Journal Article

MRI detects white matter reorganization after neural progenitor cell treatment of stroke

• Quan Jiang
• Zheng Gang Zhang
• Guang Liang Ding
• Brian Silver
• Li Zhang
• He Meng
• Mei Lu
• Siamak Pourabdillah-Nejed-D.

We evaluated the effects of neural progenitor cell treatment of stroke on white matter reorganization using MRI. Male Wistar rats (n = 26) were subjected to 3 h of middle cerebral artery occlusion and were treated with neural progenitor cells (n = 17) or without treatment (n = 9) and were sacrificed at 5–7 weeks thereafter. MRI measurements revealed that grafted neural progenitor cells selectively migrated towards the ischemic boundary regions. White matter reorganization, confirmed histologically, was coincident with increases of fractional anisotropy (FA, P < 0. 01) after stroke in the ischemic recovery regions compared to that in the ischemic core region in both treated and control groups. Immunoreactive staining showed axonal projections emanating from neurons and extruding from the corpus callosum into the ipsilateral striatum bounding the lesion areas after stroke. Fiber tracking (FT) maps derived from diffusion tensor imaging revealed similar orientation patterns to the immunohistological results. Complementary measurements in stroke patients indicated that FT maps exhibit an overall orientation parallel to the lesion boundary. Our data demonstrate that FA and FT identify and characterize cerebral tissue undergoing white matter reorganization after stroke and treatment with neural progenitor cells.

AIIM Journal 2005 Journal Article

A lexical metaschema for the UMLS semantic network

• Li Zhang
• Yehoshua Perl
• Michael Halper
• James Geller
• George Hripcsak

Objective: A metaschema is a high-level abstraction network of the UMLS’s semantic network (SN) obtained from a partition of the SN’s collection of semantic types. Every metaschema has nodes, called meta-semantic types, each of which denotes a group of semantic types constituting a subject area of the SN. A new kind of metaschema, called the lexical metaschema, is derived from a lexical partition of the SN. The lexical metaschema is compared to previously derived metaschemas, e. g. , the cohesive metaschema. Design: A new lexical partitioning methodology is presented based on identical word-usage among the names of semantic types and the definitions of their respective children. The lexical metaschema is derived from the application of the methodology. We compare the constituent meta-semantic types and their underlying semantic-type groups with the previously derived cohesive metaschema. A similar comparison of the lexical partition and a published partition of the SN is also carried out. Results: The lexical partition of the SN has 21 semantic-type groups, each of which represents a subject area. The lexical metaschema thus has 21 meta-semantic types, 19 meta-child-of hierarchical relationships, and 86 meta-relationships. Our comparison shows that 15 out of the 21 meta-semantic types in the lexical metaschema also appear in the cohesive metaschema, and 80 semantic types are covered by identical meta-semantic types or refinements between the two metaschemas. The comparison between the lexical partition and the semantic partition shows that they have very low similarity. Conclusion: The algorithmically derived lexical metaschema serves as an abstraction of the SN and provides views representing different subject areas. It compares favorably with the cohesive metaschema derived via the SN’s relationship configuration.

AIIM Journal 2005 Journal Article

An expert study evaluating the UMLS lexical metaschema

• Li Zhang
• George Hripcsak
• Yehoshua Perl
• Michael Halper
• James Geller

Objective: A metaschema is an abstraction network of the UMLS's semantic network (SN) obtained from a connected partition of its collection of semantic types. A lexical metaschema was previously derived based on a lexical partition which partitioned the SN into semantic-type groups using identical word-usage among the names of semantic types and the definitions of their respective children. In this paper, a statistical analysis methodology is presented to evaluate the lexical metaschema based on a study involving a group of established UMLS experts. Methods: In the study, each expert was asked to identify subject areas of the SN based on his or her understanding of the various semantic types. For this purpose, the expert scans the SN hierarchy top-down, identifying semantic types, which are important and different enough from their parent semantic types, as roots of their groups. From the response of each expert, an “expert metaschema” is constructed. The different experts’ metaschemas can vary widely. So, additional metaschemas are obtained from aggregations of the experts’ responses. Of special interest is the consensus metaschema which represents an aggregation of a simple majority of the experts’ responses. Statistical analysis comparing the lexical metaschema with the experts’ metaschemas and the consensus metaschema is presented. Results: The analysis results shows that 17 out of the 21 meta-semantic types in the lexical metaschema also appear in the consensus metaschema (about 81%). There are 107 semantic types (about 79%) covered by identical meta-semantic types and refinements. The results show the high similarity between the two metaschemas. Furthermore, the statistical analysis shows that the lexical metaschema did not grossly underperform compared to the experts. Conclusion: Our study shows that the lexical metaschema provides a good approximation for a partition of meaningful subject areas in the SN, when compared to the consensus metaschema capturing the aggregation of a simple majority of the human experts’ opinions.

YNIMG Journal 2005 Journal Article

Investigation of neural progenitor cell induced angiogenesis after embolic stroke in rat using MRI

• Quan Jiang
• Zheng Gang Zhang
• Guang Liang Ding
• Li Zhang
• James R. Ewing
• Lei Wang
• Ruilan Zhang
• Lian Li

Using MRI, we investigated dynamic changes of brain angiogenesis after neural progenitor cell transplantation in the living adult rat subjected to embolic stroke. Neural progenitor cells isolated from the subventricular zone (SVZ) of the adult rat were labeled by superparamagnetic particles and intracisternally transplanted into the adult rat 48 h after stroke (n = 8). Before and after the transplantation, an array of MRI parameters were measured, including high resolution 3D MRI and quantitative T 1, T 1sat (T 1 in the presence of an off-resonance irradiation of the macromolecules of brain), T 2, the inverse of the apparent forward transfer rate for magnetization transfer (k inv), cerebral blood flow (CBF), cerebral blood volume (CBV), and blood-to-brain transfer constant (K i) of Gd-DTPA. The von Willerbrand factor (vWF) immunoreactive images of coronal sections obtained at 6 weeks after cell transplantation were used to analyze vWF immunoreactive vessels. MRI measurements revealed that grafted neural progenitor cells selectively migrated towards the ischemic boundary regions. In the ischemic boundary regions, angiogenesis confirmed by an increase in vascular density and the appearance of large thin wall mother vessels was coincident with increases of CBF and CBV (CBF, P < 0. 01; CBV, P < 0. 01) at 6 weeks after treatment, and coincident with transient increases of K i with a peak at 2 to 3 weeks after cell therapy. Relative T 1, T 1sat, T 2, and k inv decreased in the ischemic boundary regions with angiogenesis compared to that in the non-angiogenic ischemic region (T 1, P < 0. 01 at 6 weeks; T 1sat, P < 0. 05 at 2 to 6 weeks; T 2, P < 0. 05 at 3 to 6 weeks; k inv P < 0. 05 at 6 weeks). Of these methods, K i appear to be the most useful MR measurements which identify and predict the location and area of angiogenesis. CBF, CBV, T 1sat, T 1, T 2, and k inv provide complementary information to characterize ischemic tissue with and without angiogenesis. Our data suggest that select MRI parameters can identify the cerebral tissue destined to undergo angiogenesis after treatment of embolic stroke with cell therapy.

AIIM Journal 2004 Journal Article

Auditing concept categorizations in the UMLS

• Huanying (Helen) Gu
• Yehoshua Perl
• Gai Elhanan
• Hua Min
• Li Zhang
• Yi Peng

The Unified Medical Language System (UMLS) integrates about 880, 000 concepts from 100 biomedical terminologies. Each concept is categorized to at least one semantic type of the Semantic Network. During the integration, it is unavoidable that some categorization errors and inconsistencies will be introduced. In this paper, we present an auditing technique to find such errors and inconsistencies. Our technique is based on an expert reviewing the pure intersections of meta-semantic types of a metaschema, a compact abstract view of the UMLS Semantic Network. We use a divide and conquer approach, handling differently small pure intersections and medium to large pure intersections. By using this approach, we limit the number of concepts reviewed, for which we expect a high percentage of errors. We reviewed all concepts in 657 pure intersections containing one to 10 concepts. Various kinds of errors are identified and the analysis of the results are presented in the paper. Also, we checked the pure intersections containing more than 10 concepts for their semantic soundness, where the semantically suspicious pure intersections are presented in the paper and their concepts are reviewed.

YNIMG Journal 2004 Journal Article

In vivo magnetic resonance imaging tracks adult neural progenitor cell targeting of brain tumor

• Zhenggang Zhang
• Quan Jiang
• Feng Jiang
• Gaungliang Ding
• Ruilan Zhang
• Lei Wang
• Li Zhang
• Adam M. Robin

Using magnetic resonance imaging (MRI), we described a method for noninvasively tracking grafted neural progenitor cells and bone marrow stromal cells (MSCs) in brain tumor of the rat. Neural progenitor cells and MSCs were labeled with lipophilic dye-coated superparamagnetic particles. The labeled neural progenitor cells and MSCs were transplanted to rats via the cisterna magna and a tail vein, respectively, 1 week after 9L-gliosarcoma cell implantation. Three-dimensional (3D) gradient echo and contrast agent images revealed dynamic migration of adult neural progenitor cells and MSCs detected by loss of MRI signals towards tumor mass and infiltrated tumor cells. Prussian blue staining and fluorescent microscope analysis showed that grafted cells targeted tumor cells and areas with grafted cells corresponded to areas with loss of MRI signals. These results demonstrate that the MRI technique provides a sensitive method for in vivo assessment of grafted cells targeting tumor mass and infiltrated tumor cells and that adult neural progenitor cells and MSCs can target tumor aggregates in the brain.

ICRA Conference 2004 Conference Paper

Multi-scale Simulation for Microsurgery Trainer

• Kian Meng Lim
• Fei Wang 0010
• Tim Poston
• Li Zhang
• Chee Leong Teo
• Etienne Burdet

For use in a Virtual Reality based training system for surgical micromanipulation, we have developed a fast multi-scale FEM algorithm that concentrates detail where needed while still handling global deformations. The resulting 6-to-7-fold speed up is promising for the development of real-time simulation of the mechanical response of a virtual organ or tissue. FEM algorithm uses elements from multiple levels in a hierarchy of mesh similar to the progressive mesh. This algorithm has been integrated with a visual/haptic feedback workstation.