Arrow Research search

Author name cluster

Lei Zhu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

74 papers
2 author rows

Possible papers

74

AAAI Conference 2026 Conference Paper

De-biased Natural Language Egocentric Task Verification via Prototypical Evidence Learning

  • Chong Liu
  • Xun Jiang
  • Fumin Shen
  • Lei Zhu
  • Jingkuan Song
  • Heng Tao Shen
  • Xing Xu

Natural Language-based Egocentric Task Verification (NLETV) aims to verify the alignment between action sequences in egocentric videos and their corresponding textual descriptions. However, existing NLETV approaches are still facing two critical challenges: (1) These methods are designed for simulating environments, ignoring the domain gap between synthetic and realistic data. (2) The matching processes are regarded as a simple binary classification problem, which undermines model reliability due to evaluation bias and uncalibrated decision settings. To address these challenges, we propose a novel method termed Prototypical Evidential Learning (PEL), which can be adapted to existing NLETV approaches and boost the model generalization and mitigate prediction bias. Our method leverages prototypes to guide cross-domain alignment and evidence collection. Specifically, PEL consists of two key components: (1) Prototypical Domain Adaptation module enabling cross-domain feature alignment and intra-domain prototype preservation between synthetic and realistic domains; (2) Matching Evidence Collector module, which quantifies prediction uncertainty on the prototypical representations through evidential deep learning. It enforces the model to collect the vision-text consistency and discrepancy evidence, thus addressing the issues of biased decisions in binary classification. Extensive experiments on two public datasets demonstrate that our PEL method outperforms existing state-of-the-art NLETV methods and shows remarkable generalizability.

PDF Details DOI

JBHI Journal 2026 Journal Article

Multi-level Asymmetric Contrastive Learning for Medical Image Segmentation Pre-training

  • Shuang Zeng
  • Lei Zhu
  • Xinliang Zhang
  • Qian Chen
  • Hangzhou He
  • Lujia Jin
  • Zifeng Tian
  • Zhaoheng Xie

Medical image segmentation is a fundamental yet challenging task due to the arduous process of acquiring large volumes of high-quality labeled data from experts. Contrastive learning offers a promising but still problematic solution to this dilemma. Firstly existing medical contrastive learning strategies focus on extracting image-level representation, which ignores abundant multi-level representations. Furthermore they underutilize the decoder either by random initialization or separate pre-training from the encoder, thereby neglecting the potential collaboration between the encoder and decoder. To address these issues, we propose a novel multi-level asymmetric contrastive learning framework named MACL for enhancing medical image segmentation. Specifically, we design an asymmetric contrastive learning structure to pre-train encoder and decoder simultaneously to provide better initialization for segmentation models. Moreover, we develop a multi-level contrastive learning strategy that integrates correspondences across feature-level, image-level, and pixel-level representations to ensure the encoder and decoder capture comprehensive details from representations of varying scales and granularities during the pre-training phase. Finally, experiments on 8 medical image datasets indicate our MACL framework outperforms existing 11 contrastive learning strategies. i. e. Our MACL achieves a superior performance with more precise predictions from visualization figures and 1. 72%, 7. 87%, 2. 49% and 1. 48% Dice higher than previous best results on ACDC, MMWHS, HVSMR and CHAOS with 10% labeled data, respectively. And our MACL also has a strong generalization ability among 5 variant U-Net backbones.

Details DOI

AAAI Conference 2026 Conference Paper

S2-UniSeg: Fast Universal Agglomerative Pooling for Scalable Segment Anything Without Supervision

  • Huihui Xu
  • Jin Ye
  • Hongqiu Wang
  • Changkai Ji
  • Jiashi Lin
  • Ming Hu
  • Ziyan Huang
  • Ying Chen

Recent self-supervised image segmentation models have achieved promising performance on semantic segmentation and class-agnostic instance segmentation. However, their pretraining schedule is multi-stage, requiring a time-consuming pseudo-masks generation process between each training epoch. This time-consuming offline process not only makes it difficult to scale with training dataset size, but also leads to sub-optimal solutions due to its discontinuous optimization routine. To solve these, we first present a novel pseudo-mask algorithm, Fast Universal Agglomerative Pooling (UniAP). Each layer of UniAP can identify groups of similar nodes in parallel, allowing to generate both semantic-level and instance-level and multi-granular pseudo-masks within ens of milliseconds for one image. Based on the fast UniAP, we propose the Scalable Self-Supervised Universal Segmentation (S2-UniSeg), which employs a student and a momentum teacher for continuous pretraining. A novel segmentation-oriented pretext task, Query-wise Self-Distillation (QuerySD), is proposed to pretrain S2-UniSeg to learn the local-to-global correspondences. Under the same setting, S2-UniSeg outperforms the SOTA UnSAM model, achieving notable improvements of AP+6.9 on COCO, AR+11.1 on UVO, PixelAcc+4.5 on COCOStuff-27, RQ+8.0 on Cityscapes. After scaling up to a larger 2M-image subset of SA-1B, S2-UniSeg further achieves performance gains on all four benchmarks.

PDF Details DOI

AAAI Conference 2026 Conference Paper

SynerDetect: Hierarchical Synergistic Learning for Generalizable AI-Generated Image Detection

  • Shuaibo Li
  • Yijun Yang
  • Zhaohu Xing
  • Hongqiu Wang
  • Pengfei Hao
  • Xingyu Li
  • Zekai Liu
  • Qing Zhang

The rapid advancement of generative models, which produce increasingly realistic synthetic images, urgently demands robust and generalizable detection methods. Consequently, research has largely pivoted to leveraging large-scale Vision Foundation Models (VFMs) for enhanced generalization. However, existing VFM-based approaches primarily adhere to either perceptual or generative paradigms, each with limitations: perceptual models capture high-level semantics but often miss subtle artifacts, whereas generative models emphasize fine-grained flaws yet overlook semantic inconsistency. To resolve this inherent trade-off, we introduce SynerDetect, a novel hierarchical synergistic framework that fundamentally unifies the two paradigms. SynerDetect achieves deep integration of heterogeneous forensic representations through two levels of synergy: Cross-Model Interactive Distillation (CMID) distills generative forensic signals into perceptual encoders via prompt-guided reconstruction; and Optimal Transport-Guided Discriminative Contrastive Learning (OT-DCL) structurally aligns and integrates these heterogeneous representations, consolidating them into a robust, unified detection space. SynerDetect achieves superior performance on standard benchmarks (AIGCDetectBenchmark and GenImage) and attains a notable 5.20% accuracy gain on the challenging Chameleon benchmark, whose synthetic images consistently pass the Visual Turing Test. These results unequivocally validate the robust, real-world generalization of our unified cross-paradigm framework.

PDF Details DOI

TMLR Journal 2026 Journal Article

The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMs

  • Jierun Chen
  • Tiezheng Yu
  • Haoli Bai
  • Lewei Yao
  • Jiannan Wu
  • Kaican Li
  • Fei Mi
  • Chaofan Tao

Large vision-language models (VLMs) increasingly adopt post-training techniques such as long chain-of-thought (CoT) supervised fine-tuning (SFT) and reinforcement learning (RL) to elicit sophisticated reasoning. While these methods exhibit synergy in language-only models, their joint effectiveness in VLMs remains uncertain. We present a systematic investigation into the distinct roles and interplay of long-CoT SFT and RL across multiple multimodal reasoning benchmarks. We find that SFT improves performance on difficult questions by in-depth, structured reasoning, but introduces verbosity and degrades performance on simpler ones. In contrast, RL promotes generalization and brevity, yielding consistent improvements across all difficulty levels, though the improvements on the hardest questions are less prominent compared to SFT. Surprisingly, combining them through two-staged, interleaved, or progressive training strategies, as well as data mixing and model merging, all fails to produce additive benefits, instead leading to trade-offs in accuracy, reasoning style, and response length. This "synergy dilemma" highlights the need for more seamless and adaptive approaches to unlock the full potential of combined post-training techniques for reasoning VLMs. Code, dataset, and fine-tuned models are available at https://github.com/JierunChen/SFT-RL-SynergyDilemma.

PDF Details

AAAI Conference 2026 Conference Paper

Toward Real-World High-Precision Image Matting and Segmentation

  • Haipeng Zhou
  • Zhaohu Xing
  • Hongqiu Wang
  • Jun Ma
  • Ping Li
  • Lei Zhu

High-precision scene parsing tasks, including image matting and dichotomous segmentation, aim to accurately predict masks with extremely fine details (such as hair). Most existing methods focus on salient, single foreground objects. While interactive methods allow for target adjustment, their class-agnostic design restricts generalization across different categories. Furthermore, the scarcity of high-quality annotation has led to a reliance on inharmonious synthetic data, resulting in poor generalization to real-world scenarios. To this end, we propose a Foreground Consistent Learning model, dubbed as FCLM, to address the aforementioned issues. Specifically, we first introduce a Depth-Aware Distillation strategy where we transfer the depth-related knowledge for better foreground representation. Considering the data dilemma, we term the processing of synthetic data as domain adaptation problem where we propose a domain-invariant learning strategy to focus on foreground learning. To support interactive prediction, we contribute an Object-Oriented Decoder that can receive both visual and language prompts to predict the referring target. Experimental results show that our method quantitatively and qualitatively outperforms state-of-the-art methods.

PDF Details DOI

AAAI Conference 2025 Conference Paper

AGLLDiff: Guiding Diffusion Models Towards Unsupervised Training-free Real-world Low-light Image Enhancement

  • Yunlong Lin
  • Tian Ye
  • Sixiang Chen
  • Zhenqi Fu
  • Yingying Wang
  • Wenhao Chai
  • Zhaohu Xing
  • Wenxue Li

Existing low-light image enhancement (LIE) methods have achieved noteworthy success in solving synthetic distortions, yet they often fall short in practical applications. The limitations arise from two inherent challenges in real-world LIE: 1) the collection of distorted/clean image pairs is often impractical and sometimes even unavailable, and 2) accurately modeling complex degradations presents a non-trivial problem. To overcome them, we propose the Attribute Guidance Diffusion framework (AGLLDiff), a training-free method for effective real-world LIE. Instead of specifically defining the degradation process, AGLLDiff shifts the paradigm and models the desired attributes, such as image exposure, structure and color of normal-light images. These attributes are readily available and impose no assumptions about the degradation process, which guides the diffusion sampling process to a reliable high-quality solution space. Extensive experiments demonstrate that our approach outperforms the current leading unsupervised LIE methods across benchmarks in terms of distortion-based and perceptual-based metrics, and it performs well even in sophisticated wild degradation.

PDF Details DOI

EAAI Journal 2025 Journal Article

Attention Guided Class Activation Maps for Boosting Weakly Supervised Semantic Segmentation

  • Junhui Li
  • Lei Zhu
  • Wenwu Wang
  • Yin Gong

Weakly Supervised Semantic Segmentation (WSSS) has garnered significant attention for its ability to utilize weaker labels in place of expensive pixel-level annotations while maintaining commendable performance. Class Activation Maps (CAM) can still possess target localization capabilities without pixel-level annotations, and are thus widely used as pseudo labels to supervise subsequent segmentation tasks. With the continuous advancement of artificial intelligence, generating high-quality CAM by combining the strengths of Convolutional Neural Networks (CNN) and Transformer architectures has received widespread attention. As a classic architecture, Conformer adopts a parallel structure and has been applied in multiple WSSS models. We observed that the attention matrices of different levels of Transformer blocks in Conformer exhibit significant characteristic differences, and these matrices effectively capture the correlations between different regions. Based on this observation, we propose a method called Attention-Guided Class Activation Map (AG-CAM), which selectively utilizes attention matrices at different levels to enhance features for various purposes. Detailed experiments on common datasets have shown that the proposed AG-CAM method significantly improves the quality of class activation maps. Our work provides a more precise solution for WSSS, thereby demonstrating immense potential and value in real-world applications where data annotation is scarce.

Details DOI

EAAI Journal 2025 Journal Article

BrainSeek: A neural-driven deep semantic reasoning framework from functional magnetic resonance imaging signals

  • Yuxiao Zhao
  • Guohua Dong
  • Runtao Duan
  • Lei Zhu
  • Xiaomin Ying

While semantic decoding of functional Magnetic Resonance Imaging (fMRI) signals advances human cognition research, current methods struggle to capture high-level inferred semantics – especially reasoning-dependent constructs – limiting their utility for embodied agents’ environmental interaction and cognitive decision-making. To address this, we propose BrainSeek, a deep semantic reasoning framework that transcends conventional brain-signal translation paradigms via neurally grounded semantic inference. BrainSeek first establishes cross-modal neural-semantic mapping, projecting fMRI voxel responses into the Sigmoid Loss for Language-Image Pre-training (SigLIP) multimodal semantic space to align neural patterns with semantic representations, enabling embodied environmental perception. Neural features are then integrated into a Large Language Model (LLM) semantic space via the Janus-Pro multimodal reasoning decoder, performing prompt-driven inference to extract high-level cognitive constructs like intentional reasoning and contextual associations for human–robot collaboration. Evaluation results demonstrate that BrainSeek significantly improves semantic decoding accuracy: in stimulus caption decoding, it achieves Metric for Evaluation of Translation with Explicit ORdering (METEOR) 1. 191, Recall-Oriented Understudy for Gisting Evaluation (ROUGE-1 and ROUGE-L) scores of 0. 242 and 0. 223, all surpassing the baseline; in high-order cognitive inference tasks, it attains Bidirectional Encoder Representations from Transformers (BERT)-based Precision, Recall, and F1 Score values all above 0. 87. This framework establishes a novel paradigm for semantic reasoning–based decoding and shows promising potential for modeling human cognition in embodied robotics as well as for applications in industrial and healthcare domains. The code is available at https: //github. com/zhao0609/BrainSeek

Details DOI

JBHI Journal 2025 Journal Article

Cascaded Inner-Outer Clip Retformer for Ultrasound Video Object Segmentation

  • Jialu Li
  • Lei Zhu
  • Zhaohu Xing
  • Baoliang Zhao
  • Ying Hu
  • Faqin Lv
  • Qiong Wang

Computer-aided ultrasound (US) imaging is an important prerequisite for early clinical diagnosis and treatment. Due to the harsh ultrasound (US) image quality and the blurry tumor area, recent memory-based video object segmentation models (VOS) achieve frame-level segmentation by performing intensive similarity matching among the past frames which could inevitably result in computational redundancy. In this paper, we first build a larger annotated benchmark dataset for breast lesion segmentation in ultrasound videos, then we propose a lightweight clip-level VOS framework for achieving higher segmentation accuracy while maintaining the speed. Then an Inner-Outer Clip Retformer is proposed to extract spatial-temporal tumor features in parallel. Specifically, the proposed Outer Clip Retformer extracts the tumor movement feature from past video clips to locate the current clip tumor position, while the Inner Clip Retformer detailedly extracts current tumor features that can produce more accurate segmentation results. Then a Clip Contrastive loss function is further proposed to align the extracted tumor features along both the spatial-temporal dimensions to improve the segmentation accuracy. In addition, the Global Retentive Memory is proposed to maintain the complementary tumor features with lower computing resources which can generate coherent temporal movement features. In this way, our model can significantly improve the spatial-temporal perception ability without increasing a large number of parameters, achieving more accurate segmentation results while maintaining a faster segmentation speed. Finally, we conduct extensive experiments to evaluate our proposed model on several video object segmentation datasets, the results show that our framework outperforms state-of-the-art segmentation methods.

Details DOI

AAAI Conference 2025 Conference Paper

Dynamic Target Distribution Estimation for Source-Free Open-Set Domain Adaptation

  • Zhiqi Yu
  • Zhichao Liao
  • Jingjing Li
  • Zhi Chen
  • Lei Zhu

Unsupervised domain adaptation (UDA) has emerged as a promising technique for transferring knowledge from a labeled domain to an unlabeled domain. However, existing UDA methods are severely constrained by data privacy and semantic inconsistencies. To alleviate these limitations, this work challenges the Source-Free Open-Set Domain Adaptation (SF-OSDA), where the pre-trained source model is directly leveraged on the open target domain for adaptation. For this purpose, we introduce the novel Dynamic Target Distribution Estimation (DTDE) method, which effectively performs known classification and unknown separation through self-supervised learning with prototypes. To construct known prototypes, a self-adaptive sampling strategy is employed to consider the category disparity. For unknown prototypes, we utilize a self-splitting and excluding principle to bypass the unknown semantics problem. Specifically, self-splitting is to evaluate the overall clustering distribution of the target domain. By excluding clusters resembling known prototypes, the remaining cluster centroids can serve as unknown prototypes. The superiority of our approach is validated across multiple benchmarks. Remarkably, DTDE outperforms the best competitor by 7.6% on the VisDA dataset.

PDF Details DOI

JBHI Journal 2025 Journal Article

Enhancing Visual Reasoning With LLM-Powered Knowledge Graphs for Visual Question Localized-Answering in Robotic Surgery

  • Pengfei Hao
  • Hongqiu Wang
  • Guang Yang
  • Lei Zhu

Expert surgeons often have heavy workloads and cannot promptly respond to queries from medical students and junior doctors about surgical procedures. Thus, research on Visual Question Localized-Answering in Surgery (Surgical-VQLA) is essential to assist medical students and junior doctors in understanding surgical scenarios. Surgical-VQLA aims to generate accurate answers and locate relevant areas in the surgical scene, requiring models to identify and understand surgical instruments, operative organs, and procedures. A key issue is the model's ability to accurately distinguish surgical instruments. Current Surgical-VQLA models rely primarily on sparse textual information, limiting their visual reasoning capabilities. To address this issue, we propose a framework called Enhancing Visual Reasoning with LLM-Powered Knowledge Graphs (EnVR-LPKG) for the Surgical-VQLA task. This framework enhances the model's understanding of the surgical scenario by utilizing knowledge graphs of surgical instruments constructed by the Large Language Model (LLM). Specifically, we design a Fine-grained Knowledge Extractor (FKE) to extract the most relevant information from knowledge graphs and perform contrastive learning with the extracted knowledge graphs and local image. Furthermore, we design a Multi-attention-based Surgical Instrument Enhancer (MSIE) module, which employs knowledge graphs to obtain an enhanced representation of the corresponding surgical instrument in the global scene. Through the MSIE module, the model can learn how to fuse visual features with knowledge graph text features, thereby strengthening the understanding of surgical instruments and further improving visual reasoning capabilities. Extensive experimental results on the EndoVis-17-VQLA and EndoVis-18-VQLA datasets demonstrate that our proposed method outperforms other state-of-the-art methods. We will release our code for future research.

Details DOI

JBHI Journal 2025 Journal Article

Federated Pseudo Modality Generation for Incomplete Multi-Modal MRI Reconstruction

  • Yunlu Yan
  • Chun-Mei Feng
  • Yuexiang Li
  • Ping Li
  • Rick Siow Mong Goh
  • Baiying Lei
  • Weiming Wang
  • David Dagan Feng

While multi-modal learning has been widely used for MRI reconstruction, it relies on paired multi-modal data, which is difficult to acquire in real clinical scenarios. Especially in the federated setting, there is a common issue that several medical institutions suffer from missing modalities or even only have single-modal data. Therefore, it is infeasible to deploy a standard federated learning framework in such conditions. In this paper, we propose a novel communication-efficient federated learning framework (namely Fed-PMG) to address the missing modality challenge in federated multi-modal MRI reconstruction. Specifically, we utilize a pseudo modality generation mechanism to recover the missing modality for each single-modal client by sharing the distribution information of the amplitude spectrum in frequency space. However, the step of sharing the original amplitude spectrum leads to heavy communication costs. To reduce the communication cost, we introduce a clustering scheme to project the set of amplitude spectrum into a finite number of cluster centroids and share them among the clients. With such an elaborate design, our approach can effectively complete the missing modality within an acceptable communication cost. Extensive experimental results demonstrate that our proposed method can outperform state-of-the-art methods and reach a performance similar to the ideal scenario (i. e. , all clients have the full set of modalities).

Details DOI

IROS Conference 2025 Conference Paper

Joint Pedestrian and Vehicle Traffic Optimization in Urban Environments using Reinforcement Learning

  • Bibek Poudel
  • Xuan Wang
  • Weizi Li
  • Lei Zhu
  • Kevin Heaslip

Reinforcement learning (RL) holds significant promise for adaptive traffic signal control. While existing RL-based methods demonstrate effectiveness in reducing vehicular congestion, their predominant focus on vehicle-centric optimization leaves pedestrian mobility needs and safety challenges unaddressed. In this paper, we present a deep RL framework for adaptive control of eight traffic signals along a real-world urban corridor, jointly optimizing both pedestrian and vehicular efficiency. Our single-agent policy is trained using real-world pedestrian and vehicle demand data derived from Wi-Fi logs and video analysis. The results demonstrate significant performance improvements over traditional fixed-time signals, reducing average wait times per pedestrian and per vehicle by up to 67% and 52% respectively, while simultaneously decreasing total wait times for both groups by up to 67% and 53%. Additionally, our results demonstrate generalization capabilities across varying traffic demands, including conditions entirely unseen during training, validating RL’s potential for developing transportation systems that serve all road users.

Details

AAAI Conference 2025 Conference Paper

Learning Together Securely: Prototype-Based Federated Multi-Modal Hashing for Safe and Efficient Multi-Modal Retrieval

  • Ruifan Zuo
  • Chaoqun Zheng
  • Lei Zhu
  • Wenpeng Lu
  • Yuanyuan Xiang
  • Zhao Li
  • Xiaofeng Qu

With the proliferation of multi-modal data, safe and efficient multi-modal hashing retrieval has become a pressing research challenge, particularly due to concerns over data privacy during centralized processing. To address this, we propose Prototype-based Federated Multi-modal Hashing (PFMH), an innovative framework that seamlessly integrates federated learning with multi-modal hashing techniques. PFMH achieves fine-grained fusion of heterogeneous multi-modal data, enhancing retrieval accuracy while ensuring data privacy through prototype-based communication, thereby reducing communication costs and mitigating risks of data leakage. Furthermore, using a prototype completion strategy, PFMH tackles class imbalance and statistical heterogeneity in multi-modal data, improving model generalization and performance across diverse data distributions. Extensive experiments demonstrate the efficiency and effectiveness of PFMH within the federated learning framework, enabling distributed training for secure and precise multi-modal retrieval in real-world scenarios.

PDF Details DOI

AAAI Conference 2025 Conference Paper

MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction

  • Zixuan Gong
  • Qi Zhang
  • Guangyin Bao
  • Lei Zhu
  • Rongtao Xu
  • Ke Liu
  • Liang Hu
  • Duoqian Miao

Decoding natural visual scenes from brain activity has flourished, with extensive research in single-subject tasks and, however, less in cross-subject tasks. Reconstructing high-quality images in cross-subject tasks is a challenging problem due to profound individual differences between subjects and the scarcity of data annotation. In this work, we proposed MindTuner for cross-subject visual decoding, which achieves high-quality and rich semantic reconstructions using only 1 hour of fMRI training data benefiting from the phenomena of visual fingerprint in the human visual system and a novel fMRI-to-text alignment paradigm. Firstly, we pre-train a multi-subject model among 7 subjects and fine-tune it with scarce data on new subjects, where LoRAs with Skip-LoRAs are utilized to learn the visual fingerprint. Then, we take the image modality as the intermediate pivot modality to achieve fMRI-to-text alignment, which achieves impressive fMRI-to-text retrieval performance and corrects fMRI-to-image reconstruction with fine-tuned semantics. The results of both qualitative and quantitative analyses demonstrate that MindTuner surpasses state-of-the-art cross-subject visual decoding models on the Natural Scenes Dataset (NSD), whether using training data of 1 hour or 40 hours.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE

  • Zongle Huang
  • Lei Zhu
  • ZongYuan Zhan
  • Ting Hu
  • Weikai Mao
  • Xianzhi Yu
  • Yongpan Liu
  • Tianyu Zhang

Large Language Models (LLMs) have achieved remarkable success across many applications, with Mixture of Experts (MoE) models demonstrating great potential. Compared to traditional dense models, MoEs achieve better performance with less computation. Speculative decoding (SD) is a widely used technique to accelerate LLM inference without accuracy loss, but it has been considered efficient only for dense models. In this work, we first demonstrate that, under medium batch sizes, MoE surprisingly benefits more from SD than dense models. Furthermore, as MoE becomes sparser -- the prevailing trend in MoE designs -- the batch size range where SD acceleration is expected to be effective becomes broader. To quantitatively understand tradeoffs involved in SD, we develop a reliable modeling based on theoretical analyses. While current SD research primarily focuses on improving acceptance rates of algorithms, changes in workload and model architecture can still lead to degraded SD acceleration even with high acceptance rates. To address this limitation, we introduce a new metric 'target efficiency' that characterizes these effects, thus helping researchers identify system bottlenecks and understand SD acceleration more comprehensively. For scenarios like private serving, this work unveils a new perspective to speed up MoE inference, where existing solutions struggle. Experiments on different GPUs show up to 2. 29x speedup for Qwen2-57B-A14B at medium batch sizes and validate our theoretical predictions.

PDF Details

NeurIPS Conference 2025 Conference Paper

OpenGU: A Comprehensive Benchmark for Graph Unlearning

  • Bowen Fan
  • Yuming Ai
  • Xunkai Li
  • Zhilin Guo
  • Lei Zhu
  • Guang Zeng
  • Rong-Hua Li
  • Guoren Wang

Graph Machine Learning is essential for understanding and analyzing relational data. However, privacy-sensitive applications demand the ability to efficiently remove sensitive information from trained graph neural networks (GNNs), avoiding the unnecessary time and space overhead caused by retraining models from scratch. To address this issue, Graph Unlearning (GU) has emerged as a critical solution to support dynamic graph updates while ensuring privacy compliance. Unlike machine unlearning in computer vision or other fields, GU faces unique difficulties due to the non-Euclidean nature of graph data and the recursive message-passing mechanism of GNNs. Additionally, the diversity of downstream tasks and the complexity of unlearning requests further amplify these challenges. Despite the proliferation of diverse GU strategies, the absence of a benchmark providing fair comparisons for GU, and the limited flexibility in combining downstream tasks and unlearning requests, have yielded inconsistencies in evaluations, hindering the development of this domain. To fill this gap, we present OpenGU, the first GU benchmark, where 16 SOTA GU algorithms and 37 multi-domain datasets are integrated, enabling various downstream tasks with 13 GNN backbones when responding to flexible unlearning requests. Through extensive experimentation, we have drawn $10$ crucial conclusions about existing GU methods, while also gaining valuable insights into their limitations, shedding light on potential avenues for future research. Our code is available at \href{https: //github. com/bwfan-bit/OpenGU}{https: //github. com/bwfan-bit/OpenGU}.

PDF Details

AAAI Conference 2025 Conference Paper

PromptHaze: Prompting Real-world Dehazing via Depth Anything Model

  • Tian Ye
  • Sixiang Chen
  • Haoyu Chen
  • Wenhao Chai
  • Jingjing Ren
  • Zhaohu Xing
  • Wenxue Li
  • Lei Zhu

Real-world image dehazing remains a challenging task due to the diverse nature of haze degradation and the lack of large-scale paired datasets. Existing methods based on hand-crafted priors or generative priors struggle to recover accurate backgrounds and fine details from dense haze regions. In this work, we propose a novel paradigm, PromptHaze, for real-world image dehazing via the depth prompt from the Depth Anything model. By employing a prompt-by-prompt strategy, our method iteratively updates the depth prompt and progressively restores the background through a dehazing network with controllable dehazing strength. Extensive experiments on widely-used real-world dehazing benchmarks demonstrate the superiority of PromptHaze in recovering authentic backgrounds and fine details from various haze scenes, outperforming state-of-the-art methods across multiple quality metrics.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

Provable Ordering and Continuity in Vision-Language Pretraining for Generalizable Embodied Agents

  • Zhizhen Zhang
  • Lei Zhu
  • Zhen Fang
  • Zi Huang
  • Yadan Luo

Pre-training vision-language representations on human action videos has emerged as a promising approach to reduce reliance on large-scale expert demonstrations for training embodied agents. However, prior methods often employ time con- trastive learning based on goal-reaching heuristics, progressively aligning language instructions from the initial to the final frame. This overemphasis on future frames can result in erroneous vision-language associations, as actions may terminate early or include irrelevant moments in the end. To address this issue, we propose Action Temporal Coherence Learning (AcTOL) to learn ordered and continuous vision-language representations without rigid goal-based constraint. AcTOL treats a video as a continuous trajectory where it (1) contrasts semantic differences be- tween frames to reflect their natural ordering, and (2) imposes a local Brownian bridge constraint to ensure smooth transitions across intermediate frames. Exten- sive imitation learning experiments on both simulated and real robots show that the pretrained features significantly enhance downstream manipulation tasks with high robustness to different linguistic styles of instructions, offering a viable pathway toward generalized embodied agents. Our project page is at https: //actol-pretrain. github. io/.

PDF Details

IROS Conference 2025 Conference Paper

Reducing Redundancy in VSLAM: VLMs-driven Keyframe Selection using Multi-dimensional Semantic Information

  • Xiang Huo
  • Shilang Chen
  • Lei Zhu
  • Haifei Zhu
  • Yisheng Guan
  • Hong Zhang 0013
  • Weinan Chen

Keyframe selection plays a crucial role in balancing computational efficiency and localization accuracy in Visual Simultaneous Localization and Mapping (VSLAM) systems. Existing keyframe selection methods often struggle to capture high-level semantic information in environments where multiple semantic dimensions interact. In this paper, we propose the Multi-dimensional Semantic Analysis (MSA) module based on Visual-Language Models (VLMs). By leveraging the capability of VLMs to extract rich semantic features, we compute the similarity between each image frame and a set of textual descriptions, generating a scene descriptor that quantifies the semantic distance between frames across multiple dimensions (e. g. , object count, texture, and lighting). We then introduce the Scene Change Assessment (SCA) module based on Bayesian On-line Changepoint Detection (BOCD), which identifies keyframes with significant semantic information gain, thereby reducing the total number of keyframes. Extensive experiments on an open dataset demonstrate that our method not only significantly reduces the number of keyframes but also maintains high localization accuracy. Furthermore, the inference speed of the MSA module satisfies the real-time requirements of VSLAM. These results underscore the potential of our approach to enhance the efficiency of keyframe selection.

Details

AAAI Conference 2025 Conference Paper

Residual Diffusion Deblurring Model for Single Image Defocus Deblurring

  • Haoxuan Feng
  • Haohui Zhou
  • Tian Ye
  • Sixiang Chen
  • Lei Zhu

Defocus deblurring is a challenging task due to the spatially varying nature of defocus blur with multiple plausible solutions of a single given image. However, most existing methods falter when faced with extensive and variable defocus blur, either ignoring it or relying on additional loss functions to enhance perceptual quality. This often results in unrealistic reconstructions and compromised generalizability. In this paper, we propose a novel Residual Diffusion Deblurring Model framework for single image defocus deblurring. Our approach integrates a pre-trained defocus map estimator and a lightweight pre-deblur module with a learnable receptive field, providing crucial posterior information to effectively address large-scale and varying shaped defocus blur. In addition, a carefully-design denoising network enables the generation of diverse reconstructions from a single input. This approach not only significantly improves the perceptual quality of defocus deblurring outputs through multi-step residual learning, but also offers a more efficient inference strategy. Experimental results demonstrate that our method achieves competitive performance on real-world defocus deblurring image datasets across both perceptual and distortion evaluation metrics.

PDF Details DOI

ICLR Conference 2025 Conference Paper

TEASER: Token Enhanced Spatial Modeling for Expressions Reconstruction

  • Yun Fei Liu
  • Lei Zhu
  • Lijian Lin
  • Ye Zhu
  • Ailing Zhang
  • Yu Li

3D facial reconstruction from a single in-the-wild image is a crucial task in human-centered computer vision tasks. While existing methods can recover accurate facial shapes, there remains significant space for improvement in fine-grained expression capture. Current approaches struggle with irregular mouth shapes, exaggerated expressions, and asymmetrical facial movements. We present TEASER (Token EnhAnced Spatial modeling for Expressions Reconstruction), which addresses these challenges and enhances 3D facial geometry performance⁠⁠. TEASER tackles two main limitations of existing methods: insufficient photometric loss for self-reconstruction and inaccurate localization of subtle expressions. We introduce a multi-scale tokenizer to extract facial appearance information. Combined with a neural renderer, these tokens provide precise geometric guidance for expression reconstruction. Furthermore, TEASER incorporates a pose-dependent landmark loss to further improve geometric performance⁠. Our approach not only significantly enhances expression reconstruction quality but also offers interpretable tokens suitable for various downstream applications, such as photorealistic facial video driving, expression transfer, and identity swapping. Quantitative and qualitative experimental results across multiple datasets demonstrate that TEASER achieves state-of-the-art performance in precise expression reconstruction.

Details

AAAI Conference 2025 Conference Paper

Towards Realistic Semi-supervised Medical Image Classification

  • Wenxue Li
  • Lie Ju
  • Feilong Tang
  • Peng Xia
  • Xinyu Xiong
  • Ming Hu
  • Lei Zhu
  • Zongyuan Ge

Existing semi-supervised learning (SSL) approaches follow the idealized closed-world assumption, neglecting the challenges present in realistic medical scenarios, such as open-set distribution and imbalanced class distribution. Although some methods in natural domains attempt to address the open-set problem, they are insufficient for medical domains, where intertwined challenges like class imbalance and small inter-class lesion discrepancies persist. Thus, this paper presents a novel self-recalibrated semantic training framework, which is tailored for SSL in medical imaging by ingeniously harvesting realistic unlabeled samples. Inspired by the observation that certain open-set samples share some similar disease-related representations with in-distribution samples, we first propose an informative sample selection strategy that identifies high-value samples to serve as augmentations, thereby effectively enriching the semantics of known categories. Furthermore, we adopt a compact semantic clustering strategy to address the semantic confusion raised by the above newly introduced open-set semantics. Moreover, to mitigate the interference of class imbalance in open-set SSL, we introduce a less biased dual-balanced classifier with similarity pseudo-label regularization and category-customized regularization. Extensive experiments on a variety of medical image datasets demonstrate the superior performance of our proposed method over state-of-the-art Closed-set and Open-set SSL methods.

PDF Details DOI

AAAI Conference 2025 Conference Paper

V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer

  • Hangzhou He
  • Lei Zhu
  • Xinliang Zhang
  • Shuang Zeng
  • Qian Chen
  • Yanye Lu

Concept Bottleneck Models (CBMs) offer inherent interpretability by initially translating images into human-comprehensible concepts, followed by a linear combination of these concepts for classification. However, the annotation of concepts for visual recognition tasks requires extensive expert knowledge and labor, constraining the broad adoption of CBMs. Recent approaches have leveraged the knowledge of large language models to construct concept bottlenecks, with multimodal models like CLIP subsequently mapping image features into the concept feature space for classification. Despite this, the concepts produced by language models can be verbose and may introduce non-visual attributes, which hurts accuracy and interpretability. In this study, we investigate to avoid these issues by constructing CBMs directly from multimodal models. To this end, we adopt common words as base concept vocabulary and leverage auxiliary unlabeled images to construct a Vision-to-Concept (V2C) tokenizer that can explicitly quantize images into their most relevant visual concepts, thus creating a vision-oriented concept bottleneck tightly coupled with the multimodal model. This leads to our V2C-CBM which is training efficient and interpretable with high accuracy. Our V2C-CBM has matched or outperformed LLM-supervised CBMs on various visual classification benchmarks, validating the efficacy of our approach.

PDF Details DOI

AIIM Journal 2025 Journal Article

VAE-GANMDA: A microbe-drug association prediction model integrating variational autoencoders and generative adversarial networks

  • Bo Wang
  • Yang He
  • Xiaoxin Du
  • Lei Zhu
  • Junqi Wang
  • Tongxuan Wang

Traditional biological experimental methods typically require weeks or even months of experimentation, and the cost of each experiment can reach hundreds or even thousands of dollars, which is quite expensive and time-consuming. To address this, a model called VAE-GANMDA, which integrates variational autoencoders (VAE) and generative adversarial networks (GAN) for predicting microbe-drug associations, has been proposed. Firstly, a heterogeneous network of microbes and drugs is established to enrich the association information. Secondly, by fusing VAE and GAN, the model learns the manifold distribution of data through association features, obtaining nonlinear manifold features. Furthermore, the VAE generation module is improved by integrating the Convolutional Block Attention Module (CBAM) and Gaussian kernel function, enhancing the smooth perception of manifold features, thus endowing VAE with stronger feature extraction capabilities. Then, singular value decomposition (SVD) technique is employed to extract linear features of the data. Finally, by combining linear and nonlinear features, the k-means++ algorithm is used to select balanced and high-quality negative samples for training the MLP classifier. Through performance evaluation, the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC) of VAE-GANMDA reach 0. 9724 and 0. 9635 respectively, outperforming classical machine learning methods and the majority of deep learning methods. Case studies demonstrate that VAE-GANMDA accurately predicts candidate drugs related to SARS-CoV-2 and candidate microbes related to ciprofloxacin.

Details DOI

NeurIPS Conference 2025 Conference Paper

VQ-Seg: Vector-Quantized Token Perturbation for Semi-Supervised Medical Image Segmentation

  • Sicheng Yang
  • Zhaohu Xing
  • Lei Zhu

Consistency learning with feature perturbation is a widely used strategy in semi-supervised medical image segmentation. However, many existing perturbation methods rely on dropout, and thus require a careful manual tuning of the dropout rate, which is a sensitive hyperparameter and often difficult to optimize and may lead to suboptimal regularization. To overcome this limitation, we propose VQ-Seg, the first approach to employ vector quantization (VQ) to discretize the feature space and introduce a novel and controllable Quantized Perturbation Module (QPM) that replaces dropout. Our QPM perturbs discrete representations by shuffling the spatial locations of codebook indices, enabling effective and controllable regularization. To mitigate potential information loss caused by quantization, we design a dual-branch architecture where the post-quantization feature space is shared by both image reconstruction and segmentation tasks. Moreover, we introduce a Post-VQ Feature Adapter (PFA) to incorporate guidance from a foundation model (FM), supplementing the high-level semantic information lost during quantization. Furthermore, we collect a large-scale Lung Cancer (LC) dataset comprising 828 CT scans annotated for central-type lung carcinoma. Extensive experiments on the LC dataset and other public benchmarks demonstrate the effectiveness of our method, which outperforms state-of-the-art approaches. Codes will be released.

PDF Details

TMLR Journal 2025 Journal Article

Where Do We Stand with Implicit Neural Representations? A Technical and Performance Survey

  • Amer Essakine
  • Yanqi Cheng
  • Chun-Wun Cheng
  • Lipei Zhang
  • Zhongying Deng
  • Lei Zhu
  • Carola-Bibiane Schönlieb
  • Angelica I Aviles-Rivero

Implicit Neural Representations (INRs) have emerged as a paradigm in knowledge representation, offering exceptional flexibility and performance across a diverse range of applications. INRs leverage multilayer perceptrons (MLPs) to model data as continuous implicit functions, providing critical advantages such as resolution independence, memory efficiency, and generalisation beyond discretised data structures. Their ability to solve complex inverse problems makes them particularly effective for tasks including audio reconstruction, image representation, 3D object reconstruction, and high-dimensional data synthesis. This survey provides a comprehensive review of state-of-the-art INR methods, introducing a clear taxonomy that categorises them into four key areas: activation functions, position encoding, combined strategies, and network structure optimisation. We rigorously analyse their critical properties—such as full differentiability, smoothness, compactness, and adaptability to varying resolutions—while also examining their strengths and limitations in addressing locality biases and capturing fine details. Our experimental comparison offers new insights into the trade-offs between different approaches, showcasing the capabilities and challenges of the latest INR techniques across various tasks. In addition to identifying areas where current methods excel, we highlight key limitations and potential avenues for improvement, such as developing more expressive activation functions, enhancing positional encoding mechanisms, and improving scalability for complex, high-dimensional data. This survey serves as a roadmap for researchers, offering practical guidance for future exploration in the field of INRs. We aim to foster new methodologies by outlining promising research directions for INRs and applications.

PDF Details

AAAI Conference 2024 Conference Paper

Agile Multi-Source-Free Domain Adaptation

  • Xinyao Li
  • Jingjing Li
  • Fengling Li
  • Lei Zhu
  • Ke Lu

Efficiently utilizing rich knowledge in pretrained models has become a critical topic in the era of large models. This work focuses on adaptively utilize knowledge from multiple source-pretrained models to an unlabeled target domain without accessing the source data. Despite being a practically useful setting, existing methods require extensive parameter tuning over each source model, which is computationally expensive when facing abundant source domains or larger source models. To address this challenge, we propose a novel approach which is free of the parameter tuning over source backbones. Our technical contribution lies in the Bi-level ATtention ENsemble (Bi-ATEN) module, which learns both intra-domain weights and inter-domain ensemble weights to achieve a fine balance between instance specificity and domain consistency. By slightly tuning source bottlenecks, we achieve comparable or even superior performance on a challenging benchmark DomainNet with less than 3% trained parameters and 8 times of throughput compared with SOTA method. Furthermore, with minor modifications, the proposed module can be easily equipped to existing methods and gain more than 4% performance boost. Code is available at https://github.com/TL-UESTC/Bi-ATEN.

PDF Details DOI

JBHI Journal 2024 Journal Article

Cross-Modal Vertical Federated Learning for MRI Reconstruction

  • Yunlu Yan
  • Hong Wang
  • Yawen Huang
  • Nanjun He
  • Lei Zhu
  • Yong Xu
  • Yuexiang Li
  • Yefeng Zheng

Federated learning enables multiple hospitals to cooperatively learn a shared model without privacy disclosure. Existing methods often take a common assumption that the data from different hospitals have the same modalities. However, such a setting is difficult to fully satisfy in practical applications, since the imaging guidelines may be different between hospitals, which makes the number of individuals with the same set of modalities limited. To this end, we formulate this practical-yet-challenging cross-modal vertical federated learning task, in which data from multiple hospitals have different modalities with a small amount of multi-modality data collected from the same individuals. To tackle such a situation, we develop a novel framework, namely Federated Consistent Regularization constrained Feature Disentanglement (Fed-CRFD), for boosting MRI reconstruction by effectively exploring the overlapping samples (i. e. , same patients with different modalities at different hospitals) and solving the domain shift problem caused by different modalities. Particularly, our Fed-CRFD involves an intra-client feature disentangle scheme to decouple data into modality-invariant and modality-specific features, where the modality-invariant features are leveraged to mitigate the domain shift problem. In addition, a cross-client latent representation consistency constraint is proposed specifically for the overlapping samples to further align the modality-invariant features extracted from different modalities. Hence, our method can fully exploit the multi-source data from hospitals while alleviating the domain shift problem. Extensive experiments on two typical MRI datasets demonstrate that our network clearly outperforms state-of-the-art MRI reconstruction methods.

Details DOI

AAAI Conference 2024 Conference Paper

Effective Comparative Prototype Hashing for Unsupervised Domain Adaptation

  • Hui Cui
  • Lihai Zhao
  • Fengling Li
  • Lei Zhu
  • Xiaohui Han
  • Jingjing Li

Unsupervised domain adaptive hashing is a highly promising research direction within the field of retrieval. It aims to transfer valuable insights from the source domain to the target domain while maintaining high storage and retrieval efficiency. Despite its potential, this field remains relatively unexplored. Previous methods usually lead to unsatisfactory retrieval performance, as they frequently directly apply slightly modified domain adaptation algorithms to hash learning framework, or pursue domain alignment within the Hamming space characterized by limited semantic information. In this paper, we propose a simple yet effective approach named Comparative Prototype Hashing (CPH) for unsupervised domain adaptive image retrieval. We establish a domain-shared unit hypersphere space through prototype contrastive learning and then obtain the Hamming hypersphere space via mapping from the shared hypersphere. This strategy achieves a cohesive synergy between learning uniformly distributed and category conflict-averse feature representations, eliminating domain discrepancies, and facilitating hash code learning. Moreover, by leveraging dual-domain information to supervise the entire hashing model training process, we can generate hash codes that retain inter-sample similarity relationships within both domains. Experimental results validate that our CPH significantly outperforms the state-of-the-art counterparts across multiple cross-domain and single-domain retrieval tasks. Notably, on Office-Home and Office-31 datasets, CPH achieves an average performance improvement of 19.29% and 13.85% on cross-domain retrieval tasks compared to the second-best results, respectively. The source codes of our method are available at: https://github.com/christinecui/CPH.

PDF Details DOI

JBHI Journal 2024 Journal Article

Hybrid Masked Image Modeling for 3D Medical Image Segmentation

  • Zhaohu Xing
  • Lei Zhu
  • Lequan Yu
  • Zhiheng Xing
  • Liang Wan

Masked image modeling (MIM) with transformer backbones has recently been exploited as a powerful self-supervised pre-training technique. The existing MIM methods adopt the strategy to mask random patches of the image and reconstruct the missing pixels, which only considers semantic information at a lower level, and causes a long pre-training time. This paper presents HybridMIM, a novel hybrid self-supervised learning method based on masked image modeling for 3D medical image segmentation. Specifically, we design a two-level masking hierarchy to specify which and how patches in sub-volumes are masked, effectively providing the constraints of higher level semantic information. Then we learn the semantic information of medical images at three levels, including: 1) partial region prediction to reconstruct key contents of the 3D image, which largely reduces the pre-training time burden (pixel-level); 2) patch-masking perception to learn the spatial relationship between the patches in each sub-volume (region-level); and 3) drop-out-based contrastive learning between samples within a mini-batch, which further improves the generalization ability of the framework (sample-level). The proposed framework is versatile to support both CNN and transformer as encoder backbones, and also enables to pre-train decoders for image segmentation. We conduct comprehensive experiments on five widely-used public medical image segmentation datasets, including BraTS2020, BTCV, MSD Liver, MSD Spleen, and BraTS2023. The experimental results show the clear superiority of HybridMIM against competing supervised methods, masked pre-training approaches, and other self-supervised methods, in terms of quantitative metrics, speed performance and qualitative observations.

Details DOI

JBHI Journal 2024 Journal Article

Improving Cancer Survival Prediction via Graph Convolutional Neural Network Learning on Protein-Protein Interaction Networks

  • Hongmin Cai
  • Yi Liao
  • Lei Zhu
  • Zhikang Wang
  • Jiangning Song

Cancer is one of the most challenging health problems worldwide. Accurate cancer survival prediction is vital for clinical decision making. Many deep learning methods have been proposed to understand the association between patients’ genomic features and survival time. In most cases, the gene expression matrix is fed directly to the deep learning model. However, this approach completely ignores the interactions between biomolecules, and the resulting models can only learn the expression levels of genes to predict patient survival. In essence, the interaction between biomolecules is the key to determining the direction and function of biological processes. Proteins are the building blocks and principal undertakings of life activities, and as such, their complex interaction network is potentially informative for deep learning methods. Therefore, a more reliable approach is to have the neural network learn both gene expression data and protein interaction networks. We propose a new computational approach, termed CRESCENT, which is a protein-protein interaction (PPI) prior knowledge graph-based convolutional neural network (GCN) to improve cancer survival prediction. CRESCENT relies on the gene expression networks rather than gene expression levels to predict patient survival. The performance of CRESCENT is evaluated on a large-scale pan-cancer dataset consisting of 5991 patients from 16 different types of cancers. Extensive benchmarking experiments demonstrate that our proposed method is competitive in terms of the evaluation metric of the time-dependent concordance index( ${C}^{td}$ ) when compared with several existing state-of-the-art approaches. Experiments also show that incorporating the network structure between genomic features effectively improves cancer survival prediction.

Details DOI

NeurIPS Conference 2024 Conference Paper

NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction

  • Zixuan Gong
  • Guangyin Bao
  • Qi Zhang
  • Zhongwei Wan
  • Duoqian Miao
  • Shoujin Wang
  • Lei Zhu
  • Changwei Wang

Reconstruction of static visual stimuli from non-invasion brain activity fMRI achieves great success, owning to advanced deep learning models such as CLIP and Stable Diffusion. However, the research on fMRI-to-video reconstruction remains limited since decoding the spatiotemporal perception of continuous visual experiences is formidably challenging. We contend that the key to addressing these challenges lies in accurately decoding both high-level semantics and low-level perception flows, as perceived by the brain in response to video stimuli. To the end, we propose NeuroClips, an innovative framework to decode high-fidelity and smooth video from fMRI. NeuroClips utilizes a semantics reconstructor to reconstruct video keyframes, guiding semantic accuracy and consistency, and employs a perception reconstructor to capture low-level perceptual details, ensuring video smoothness. During inference, it adopts a pre-trained T2V diffusion model injected with both keyframes and low-level perception flows for video reconstruction. Evaluated on a publicly available fMRI-video dataset, NeuroClips achieves smooth high-fidelity video reconstruction of up to 6s at 8FPS, gaining significant improvements over state-of-the-art models in various metrics, e. g. , a 128% improvement in SSIM and an 81% improvement in spatiotemporal metrics. Our project is available at https: //github. com/gongzix/NeuroClips.

PDF Details DOI

EAAI Journal 2024 Journal Article

Prediction of the transient emission characteristics from diesel engine using temporal convolutional networks

  • Jianxiong Liao
  • Jie Hu
  • Peng Chen
  • Lei Zhu
  • Yan Wu
  • Zhizhou Cai
  • Hanming Wu
  • Maoxuan Wang

In order to predict the transient emission characteristics from diesel engine accurately and quickly, a novel prediction model, based on temporal convolutional networks (TCN) that incorporates the dilated convolutions and residual connections, was presented in the paper. Firstly, 1800 samples from the World Harmonized Transient Cycle (WHTC) were employed to train and validate the model. A Random Forest algorithm was used to select six top important variables as inputs to reduce the data dimensionality. Then the effect of model hyperparameters on the prediction performance was discussed and the optimal hyperparameter combination was obtained by a particle swarm optimization (PSO) algorithm. The optimized TCN model showed a coefficient of determination value (R2) above 0. 972 for training dataset and 0. 941 for validation dataset, respectively. The root mean squared error (RMSE) and the mean absolute error (MAE) were relatively low. Finally, the measured data from World Harmonized Steady Cycle (WHSC) was used to test model, and the average R2 value of 0. 936 demonstrated that TCN model has excellent robustness and generalization. Moreover, a comparative investigation between TCN model and other advanced algorithms, including BP, GBRT, XGBoost, RNN, LSTM and Transformer, was also conducted. The result showed that TCN model has not only higher accuracy, but also has less computing time. This demonstrates that it is a promising method to predict the emission characteristics of diesel engine.

Details DOI

NeurIPS Conference 2024 Conference Paper

RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models

  • Haoyu Chen
  • Wenbo Li
  • Jinjin Gu
  • Jingjing Ren
  • Sixiang Chen
  • Tian Ye
  • Renjing Pei
  • Kaiwen Zhou

Natural images captured by mobile devices often suffer from multiple types of degradation, such as noise, blur, and low light. Traditional image restoration methods require manual selection of specific tasks, algorithms, and execution sequences, which is time-consuming and may yield suboptimal results. All-in-one models, though capable of handling multiple tasks, typically support only a limited range and often produce overly smooth, low-fidelity outcomes due to their broad data distribution fitting. To address these challenges, we first define a new pipeline for restoring images with multiple degradations, and then introduce RestoreAgent, an intelligent image restoration system leveraging multimodal large language models. RestoreAgent autonomously assesses the type and extent of degradation in input images and performs restoration through (1) determining the appropriate restoration tasks, (2) optimizing the task sequence, (3) selecting the most suitable models, and (4) executing the restoration. Experimental results demonstrate the superior performance of RestoreAgent in handling complex degradation, surpassing human experts. Furthermore, the system’s modular design facilitates the fast integration of new tasks and models.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Revisiting the Integration of Convolution and Attention for Vision Backbone

  • Lei Zhu
  • Xinjiang Wang
  • Wayne Zhang
  • Rynson Lau

Convolutions (Convs) and multi-head self-attentions (MHSAs) are typically considered alternatives to each other for building vision backbones. Although some works try to integrate both, they apply the two operators simultaneously at the finest pixel granularity. With Convs responsible for per-pixel feature extraction already, the question is whether we still need to include the heavy MHSAs at such a fine-grained level. In fact, this is the root cause of the scalability issue w. r. t. the input resolution for vision transformers. To address this important problem, we propose in this work to use MSHAs and Convs in parallel \textbf{at different granularity levels} instead. Specifically, in each layer, we use two different ways to represent an image: a fine-grained regular grid and a coarse-grained set of semantic slots. We apply different operations to these two representations: Convs to the grid for local features, and MHSAs to the slots for global features. A pair of fully differentiable soft clustering and dispatching modules is introduced to bridge the grid and set representations, thus enabling local-global fusion. Through extensive experiments on various vision tasks, we empirically verify the potential of the proposed integration scheme, named \textit{GLMix}: by offloading the burden of fine-grained features to light-weight Convs, it is sufficient to use MHSAs in a few (e. g. , 64) semantic slots to match the performance of recent state-of-the-art backbones, while being more efficient. Our visualization results also demonstrate that the soft clustering module produces a meaningful semantic grouping effect with only IN1k classification supervision, which may induce better interpretability and inspire new weakly-supervised semantic segmentation approaches. Code will be available at \url{https: //github. com/rayleizhu/GLMix}.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Scaling the Codebook Size of VQ-GAN to 100,000 with a Utilization Rate of 99%

  • Lei Zhu
  • Fangyun Wei
  • Yanye Lu
  • Dong Chen

In the realm of image quantization exemplified by VQGAN, the process encodes images into discrete tokens drawn from a codebook with a predefined size. Recent advancements, particularly with LLAMA 3, reveal that enlarging the codebook significantly enhances model performance. However, VQGAN and its derivatives, such as VQGAN-FC (Factorized Codes) and VQGAN-EMA, continue to grapple with challenges related to expanding the codebook size and enhancing codebook utilization. For instance, VQGAN-FC is restricted to learning a codebook with a maximum size of 16, 384, maintaining a typically low utilization rate of less than 12% on ImageNet. In this work, we propose a novel image quantization model named VQGAN-LC (Large Codebook), which extends the codebook size to 100, 000, achieving an utilization rate exceeding 99%. Unlike previous methods that optimize each codebook entry, our approach begins with a codebook initialized with 100, 000 features extracted by a pre-trained vision encoder. Optimization then focuses on training a projector that aligns the entire codebook with the feature distributions of the encoder in VQGAN-LC. We demonstrate the superior performance of our model over its counterparts across a variety of tasks, including image reconstruction, image classification, auto-regressive image generation using GPT, and image creation with diffusion- and flow-based generative models.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Scribble Hides Class: Promoting Scribble-Based Weakly-Supervised Semantic Segmentation with Its Class Label

  • Xinliang Zhang
  • Lei Zhu
  • Hangzhou He
  • Lujia Jin
  • Yanye Lu

Scribble-based weakly-supervised semantic segmentation using sparse scribble supervision is gaining traction as it reduces annotation costs when compared to fully annotated alternatives. Existing methods primarily generate pseudo-labels by diffusing labeled pixels to unlabeled ones with local cues for supervision. However, this diffusion process fails to exploit global semantics and class-specific cues, which are important for semantic segmentation. In this study, we propose a class-driven scribble promotion network, which utilizes both scribble annotations and pseudo-labels informed by image-level classes and global semantics for supervision. Directly adopting pseudo-labels might misguide the segmentation model, thus we design a localization rectification module to correct foreground representations in the feature space. To further combine the advantages of both supervisions, we also introduce a distance entropy loss for uncertainty reduction, which adapts per-pixel confidence weights according to the reliable region determined by the scribble and pseudo-label's boundary. Experiments on the ScribbleSup dataset with different qualities of scribble annotations outperform all the previous methods, demonstrating the superiority and robustness of our method. The code is available at https://github.com/Zxl19990529/Class-driven-Scribble-Promotion-Network.

PDF Details DOI

AIIM Journal 2024 Journal Article

SSM-Net: Semi-supervised multi-task network for joint lesion segmentation and classification from pancreatic EUS images

  • Jiajia Li
  • Pingping Zhang
  • Xia Yang
  • Lei Zhu
  • Teng Wang
  • Ping Zhang
  • Ruhan Liu
  • Bin Sheng

Pancreatic cancer does not show specific symptoms, which makes the diagnosis of early stages difficult with established image-based screening methods and therefore has the worst prognosis among all cancers. Although endoscopic ultrasonography (EUS) has a key role in diagnostic algorithms for pancreatic diseases, B-mode imaging of the pancreas can be affected by confounders such as chronic pancreatitis, which can make both pancreatic lesion segmentation and classification laborious and highly specialized. To address these challenges, this work proposes a semi-supervised multi-task network (SSM-Net) to leverage unlabeled and labeled EUS images for joint pancreatic lesion classification and segmentation. Specifically, we first devise a saliency-aware representation learning module (SRLM) on a large number of unlabeled images to train a feature extraction encoder network for labeled images by computing a contrastive loss with a semantic saliency map, which is obtained by our spectral residual module (SRM). Moreover, for labeled EUS images, we devise channel attention blocks (CABs) to refine the features extracted from the pre-trained encoder on unlabeled images for segmenting lesions, and then devise a merged global attention module (MGAM) and a feature similarity loss (FSL) for obtaining a lesion classification result. We collect a large-scale EUS-based pancreas image dataset (LS-EUSPI) consisting of 9, 555 pathologically proven labeled EUS images (499 patients from four categories) and 15, 500 unlabeled EUS images. Experimental results on the LS-EUSPI dataset and a public thyroid gland lesion dataset show that our SSM-Net clearly outperforms state-of-the-art methods.

Details DOI

NeurIPS Conference 2024 Conference Paper

Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?

  • Pedro R. Bassi
  • Wenxuan Li
  • Yucheng Tang
  • Fabian Isensee
  • Zifu Wang
  • Jieneng Chen
  • Yu-Cheng Chou
  • Saikat Roy

How can we test AI performance? This question seems trivial, but it isn't. Standard benchmarks often have problems such as in-distribution and small-size test sets, oversimplified metrics, unfair comparisons, and short-term outcome pressure. As a consequence, good performance on standard benchmarks does not guarantee success in real-world scenarios. To address these problems, we present Touchstone, a large-scale collaborative segmentation benchmark of 9 types of abdominal organs. This benchmark is based on 5, 195 training CT scans from 76 hospitals around the world and 5, 903 testing CT scans from 11 additional hospitals. This diverse test set enhances the statistical significance of benchmark results and rigorously evaluates AI algorithms across various out-of-distribution scenarios. We invited 14 inventors of 19 AI algorithms to train their algorithms, while our team, as a third party, independently evaluated these algorithms on three test sets. In addition, we also evaluated pre-existing AI frameworks---which, differing from algorithms, are more flexible and can support different algorithms—including MONAI from NVIDIA, nnU-Net from DKFZ, and numerous other open-source frameworks. We are committed to expanding this benchmark to encourage more innovation of AI algorithms for the medical domain.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

UltraPixel: Advancing Ultra High-Resolution Image Synthesis to New Peaks

  • Jingjing Ren
  • Wenbo Li
  • Haoyu Chen
  • Renjing Pei
  • Bin Shao
  • Yong Guo
  • Long Peng
  • Fenglong Song

Ultra-high-resolution image generation poses great challenges, such as increased semantic planning complexity and detail synthesis difficulties, alongside substantial training resource demands. We present UltraPixel, a novel architecture utilizing cascade diffusion models to generate high-quality images at multiple resolutions (\textit{e. g. }, 1K, 2K, and 4K) within a single model, while maintaining computational efficiency. UltraPixel leverages semantics-rich representations of lower-resolution images in a later denoising stage to guide the whole generation of highly detailed high-resolution images, significantly reducing complexity. Specifically, we introduce implicit neural representations for continuous upsampling and scale-aware normalization layers adaptable to various resolutions. Notably, both low- and high-resolution processes are performed in the most compact space, sharing the majority of parameters with less than 3$\%$ additional parameters for high-resolution outputs, largely enhancing training and inference efficiency. Our model achieves fast training with reduced data requirements, producing photo-realistic high-resolution images and demonstrating state-of-the-art performance in extensive experiments.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

Benchmarking Large Language Models on CMExam - A comprehensive Chinese Medical Exam Dataset

  • Junling Liu
  • Peilin Zhou
  • Yining Hua
  • Dading Chong
  • Zhongyu Tian
  • Andrew Liu
  • Helin Wang
  • Chenyu You

Recent advancements in large language models (LLMs) have transformed the field of question answering (QA). However, evaluating LLMs in the medical field is challenging due to the lack of standardized and comprehensive datasets. To address this gap, we introduce CMExam, sourced from the Chinese National Medical Licensing Examination. CMExam consists of 60K+ multiple-choice questions for standardized and objective evaluations, as well as solution explanations for model reasoning evaluation in an open-ended manner. For in-depth analyses of LLMs, we invited medical professionals to label five additional question-wise annotations, including disease groups, clinical departments, medical disciplines, areas of competency, and question difficulty levels. Alongside the dataset, we further conducted thorough experiments with representative LLMs and QA algorithms on CMExam. The results show that GPT-4 had the best accuracy of 61. 6% and a weighted F1 score of 0. 617. These results highlight a great disparity when compared to human accuracy, which stood at 71. 6%. For explanation tasks, while LLMs could generate relevant reasoning and demonstrate improved performance after finetuning, they fall short of a desired standard, indicating ample room for improvement. To the best of our knowledge, CMExam is the first Chinese medical exam dataset to provide comprehensive medical annotations. The experiments and findings of LLM evaluation also provide valuable insights into the challenges and potential solutions in developing Chinese medical QA systems and LLM evaluation pipelines.

PDF Details

AAAI Conference 2023 Conference Paper

Cross-Domain Adaptative Learning for Online Advertisement Customer Lifetime Value Prediction

  • Hongzu Su
  • Zhekai Du
  • Jingjing Li
  • Lei Zhu
  • Ke Lu

Accurate estimation of customer lifetime value (LTV), which reflects the potential consumption of a user over a period of time, is crucial for the revenue management of online advertising platforms. However, predicting LTV in real-world applications is not an easy task since the user consumption data is usually insufficient within a specific domain. To tackle this problem, we propose a novel cross-domain adaptative framework (CDAF) to leverage consumption data from different domains. The proposed method is able to simultaneously mitigate the data scarce problem and the distribution gap problem caused by data from different domains. To be specific, our method firstly learns a LTV prediction model from a different but related platform with sufficient data provision. Subsequently, we exploit domain-invariant information to mitigate data scarce problem by minimizing the Wasserstein discrepancy between the encoded user representations of two domains. In addition, we design a dual-predictor schema which not only enhances domain-invariant information in the semantic space but also preserves domain-specific information for accurate target prediction. The proposed framework is evaluated on five datasets collected from real historical data on the advertising platform of Tencent Games. Experimental results verify that the proposed framework is able to significantly improve the LTV prediction performance on this platform. For instance, our method can boost DCNv2 with the improvement of 13.7% in terms of AUC on dataset G2. Code: https://github.com/TL-UESTC/CDAF.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Mutual-Enhanced Incongruity Learning Network for Multi-Modal Sarcasm Detection

  • Yang Qiao
  • Liqiang Jing
  • Xuemeng Song
  • Xiaolin Chen
  • Lei Zhu
  • Liqiang Nie

Sarcasm is a sophisticated linguistic phenomenon that is prevalent on today's social media platforms. Multi-modal sarcasm detection aims to identify whether a given sample with multi-modal information (i.e., text and image) is sarcastic. This task's key lies in capturing both inter- and intra-modal incongruities within the same context. Although existing methods have achieved compelling success, they are disturbed by irrelevant information extracted from the whole image and text, or overlooking some important information due to the incomplete input. To address these limitations, we propose a Mutual-enhanced Incongruity Learning Network for multi-modal sarcasm detection, named MILNet. In particular, we design a local semantic-guided incongruity learning module and a global incongruity learning module. Moreover, we introduce a mutual enhancement module to take advantage of the underlying consistency between the two modules to boost the performance. Extensive experiments on a widely-used dataset demonstrate the superiority of our model over cutting-edge methods.

PDF Details DOI

JBHI Journal 2023 Journal Article

Uncertainty-Aware Multi-Dimensional Mutual Learning for Brain and Brain Tumor Segmentation

  • Junting Zhao
  • Zhaohu Xing
  • Zhihao Chen
  • Liang Wan
  • Tong Han
  • Huazhu Fu
  • Lei Zhu

Existing segmentation methods for brain MRI data usually leverage 3D CNNs on 3D volumes or employ 2D CNNs on 2D image slices. We discovered that while volume-based approaches well respect spatial relationships across slices, slice-based methods typically excel at capturing fine local features. Furthermore, there is a wealth of complementary information between their segmentation predictions. Inspired by this observation, we develop an Uncertainty-aware Multi-dimensional Mutual learning framework to learn different dimensional networks simultaneously, each of which provides useful soft labels as supervision to the others, thus effectively improving the generalization ability. Specifically, our framework builds upon a 2D-CNN, a 2. 5D-CNN, and a 3D-CNN, while an uncertainty gating mechanism is leveraged to facilitate the selection of qualified soft labels, so as to ensure the reliability of shared information. The proposed method is a general framework and can be applied to varying backbones. The experimental results on three datasets demonstrate that our method can significantly enhance the performance of the backbone network by notable margins, achieving a Dice metric improvement of 2. 8% on MeniSeg, 1. 4% on IBSR, and 1. 3% on BraTS2020.

Details DOI

IJCAI Conference 2022 Conference Paper

Copy Motion From One to Another: Fake Motion Video Generation

  • Zhenguang Liu
  • Sifan Wu
  • Chejian Xu
  • Xiang Wang
  • Lei Zhu
  • Shuang Wu
  • Fuli Feng

One compelling application of artificial intelligence is to generate a video of a target person performing arbitrary desired motion (from a source person). While the state-of-the-art methods are able to synthesize a video demonstrating similar broad stroke motion details, they are generally lacking in texture details. A pertinent manifestation appears as distorted face, feet, and hands, and such flaws are very sensitively perceived by human observers. Furthermore, current methods typically employ GANs with a L2 loss to assess the authenticity of the generated videos, inherently requiring a large amount of training samples to learn the texture details for adequate video generation. In this work, we tackle these challenges from three aspects: 1) We disentangle each video frame into foreground (the person) and background, focusing on generating the foreground to reduce the underlying dimension of the network output. 2) We propose a theoretically motivated Gromov-Wasserstein loss that facilitates learning the mapping from a pose to a foreground image. 3) To enhance texture details, we encode facial features with geometric guidance and employ local GANs to refine the face, feet, and hands. Extensive experiments show that our method is able to generate realistic target person videos, faithfully copying complex motions from a source person. Our code and datasets are released at https: //github. com/Sifann/FakeMotion.

PDF Details DOI

JBHI Journal 2022 Journal Article

Identify Multiple Gene-Drug Common Modules via Constrained Graph Matching

  • Jiazhou Chen
  • Jie Huang
  • Yi Liao
  • Lei Zhu
  • Hongmin Cai

Identifying gene-drug interactions is vital to understanding biological mechanisms and achieving precise drug repurposing. High-throughput technologies produce a large amount of pharmacological and genomic data, providing an opportunity to explore the associations between oncogenic genes and therapeutic drugs. However, most studies only focus on “one-to-one” or “one-to-many” interactions, ignoring the multivariate patterns between genes and drugs. In this article, a high-order graph matching model with hypergraph constraints is proposed to discover the gene-drug common regulatory modules. Moreover, the prior knowledge is formulated into hypergraph constraints to reveal their multiple correspondences, penalizing the tensor matching process. The experimental results on the synthetic data demonstrate the proposed model is robust to noise contamination and outlier corruption, achieving a better performance than four state-of-the-art methods. We then evaluate the statistical power of our proposed method on the pharmacogenomics data. Our identified gene-drug common modules not only show significantly enriched pathways associated with cancer but also manifest the highly close gene-drug interactions.

Details DOI

IROS Conference 2022 Conference Paper

InterFusion: Interaction-based 4D Radar and LiDAR Fusion for 3D Object Detection

  • Li Wang 0092
  • Xinyu Zhang 0001
  • Baowei Xv
  • Jinzhao Zhang
  • Rong Fu
  • Xiaoyu Wang
  • Lei Zhu
  • Haibing Ren

Many recent works detect 3D objects by several sensor modalities for autonomous driving, where high-resolution cameras and high-line LiDARs are mostly used but relatively expensive. To achieve a balance between overall cost and detection accuracy, many multi-modal fusion techniques have been suggested. In recent years, the fusion of LiDAR and Radar has gained ever-increasing attention, especially 4D Radar, which can adapt to bad weather conditions due to its penetrability. Although features have been fused from multiple sensing modalities, most methods cannot learn interactions from different modalities, which does not make for their best use. Inspired by the self-attention mechanism, we present InterFusion, an interaction-based fusion framework, to fuse 16-line LiDAR with 4D Radar. It aggregates features from two modalities and identifies cross-modal relations between Radar and LiDAR features. In experimental evaluations on the Astyx HiRes 2019 dataset, our method outperformed the baseline by 4. 20% mAP in 3D and 10. 76% BEV mAP for the car class at the moderate level.

Details

IROS Conference 2022 Conference Paper

Keyframe Selection with Information Occupancy Grid Model for Long-term Data Association

  • Weinan Chen
  • Hanjing Ye
  • Lei Zhu
  • Chao Tang 0001
  • Changfei Fu
  • Yonggang Chen
  • Hong Zhang 0013

As the basics of Visual Simultaneous Localization And Mapping (VSLAM), keyframes play an essential role. In previous works, keyframes are selected according to a series of view change-based strategies for short-term data association (STDA). However, the texture enrichment of frames is always ignored, resulting in the failure of long-term data association (LTDA). In this paper, we propose an information enrichment selection strategy with an information occupancy grid model and a deep descriptor. Frame is expressed by a deep global descriptor for a statistical explainable abstraction, in which the texture enrichment is indicated. Based on the abstraction, an information occupancy grid model is established to measure the information enrichment and the potential LTDA ability. Evaluations on variant datasets are conducted, showing the advantage of our proposed method in terms of keyframe selection and tracking precision. Also, the statistical explainability of the deep descriptor is provided. The proposed keyframe selection strategy can improve LTDA and tracking precision, especially in situations with repeated observations and loop-closures.

Details

IJCAI Conference 2022 Conference Paper

VMAgent: A Practical Virtual Machine Scheduling Platform

  • Junjie Sheng
  • Shengliang Cai
  • Haochuan Cui
  • Wenhao Li
  • Yun Hua
  • Bo Jin
  • Wenli Zhou
  • Yiqiu Hu

Virtual machine (VM) scheduling is one of the critical tasks in cloud computing. Many works have attempted to incorporate machine learning, especially reinforcement learning, to empower VM scheduling procedures. Although improved results are shown in several demo simulators, the performances in real-world scenarios are still underexploited. In this paper, we design a practical VM scheduling platform, i. e. , VMAgent, to assist researchers in developing their methods on the VM scheduling problem. VMAgent consists of three components: simulator, scheduler, and visualizer. The simulator abstracts three general realistic scheduling scenarios (fading, recovering, and expansion) based on Huawei Cloud’s scheduling data, which is the core of our platform. Flexible configurations are further provided to make the simulator compatible with practical cloud computing architecture (i. e. , Multi Non-Uniform Memory Access) and scenarios. Researchers then need to instantiate the scheduler to interact with the simulator, which is also pre-built in various types (e. g. , heuristic, machine learning, and operations research) of scheduling algorithms to speed up the algorithm design. The visualizer, as an auxiliary component of the simulator and scheduler, facilitates researchers to conduct an in-depth analysis of the scheduling procedure and comprehensively compare different scheduling algorithms. We believe that VMAgent would shed light on the AI for the VM scheduling community, and the demo video is presented in https: //bit. ly/vmagent-demo-video.

PDF Details DOI

AAAI Conference 2021 Conference Paper

Balanced Open Set Domain Adaptation via Centroid Alignment

  • Mengmeng Jing
  • Jingjing Li
  • Lei Zhu
  • Zhengming Ding
  • Ke Lu
  • Yang Yang

Open Set Domain Adaptation (OSDA) is a challenging domain adaptation setting which allows the existence of unknown classes on the target domain. Although existing OSDA methods are good at classifying samples of known classes, they ignore the classification ability for the unknown samples, making them unbalanced OSDA methods. To alleviate this problem, we propose a balanced OSDA methods which could recognize the unknown samples while maintain high classification performance for the known samples. Specifically, to reduce the domain gaps, we first project the features to a hyperspherical latent space. In this space, we propose to bound the centroid deviation angles to not only increase the intraclass compactness but also enlarge the inter-class margins. With the bounded centroid deviation angles, we employ the statistical Extreme Value Theory to recognize the unknown samples that are misclassified into known classes. In addition, to learn better centroids, we propose an improved centroid update strategy based on sample reweighting and adaptive update rate to cooperate with centroid alignment. Experimental results on three OSDA benchmarks verify that our method can significantly outperform the compared methods and reduce the proportion of the unknown samples being misclassified into known classes.

PDF Details

IJCAI Conference 2021 Conference Paper

DA-GCN: A Domain-aware Attentive Graph Convolution Network for Shared-account Cross-domain Sequential Recommendation

  • Lei Guo
  • Li Tang
  • Tong Chen
  • Lei Zhu
  • Quoc Viet Hung Nguyen
  • Hongzhi Yin

Shared-account Cross-domain Sequential Recommendation (SCSR) is the task of recommending the next item based on a sequence of recorded user behaviors, where multiple users share a single account, and their behaviours are available in multiple domains. Existing work on solving SCSR mainly relies on mining sequential patterns via RNN-based models, which are not expressive enough to capture the relationships among multiple entities. Moreover, all existing algorithms try to bridge two domains via knowledge transfer in the latent space, and the explicit cross-domain graph structure is unexploited. In this work, we propose a novel graph-based solution, namely DA-GCN, to address the above challenges. Specifically, we first link users and items in each domain as a graph. Then, we devise a domain-aware graph convolution network to learn user-specific node representations. To fully account for users' domain-specific preferences on items, two novel attention mechanisms are further developed to selectively guide the message passing process. Extensive experiments on two real-world datasets are conducted to demonstrate the superiority of our DA-GCN method.

PDF Details DOI

AAAI Conference 2021 Conference Paper

Savable but Lost Lives when ICU Is Overloaded: a Model from 733 Patients in Epicenter Wuhan, China

  • Tingting Dan
  • Yang Li
  • Ziwei Zhu
  • Xijie Chen
  • Wuxiu Quan
  • Yu Hu
  • Guihua Tao
  • Lei Zhu

Coronavirus Disease 2019 (COVID-19) causes a sudden turnover to bad at some checkpoints and thus needs the intervention of intensive care unit (ICU). This resulted in urgent and large needs of ICUs posed great risks to the medical system. Estimating the mortality of critical in-patients who were not admitted into the ICU will be valuable to optimize the management and assignment of ICU. Retrospective, 733 in-patients diagnosed with COVID-19 at a local hospital (Wuhan, China), as of March 18, 2020. Demographic, clinical and laboratory results were collected and analyzed using machine learning to build a predictive model. Considering the shortage of ICU beds at the beginning of disease emergence, we defined the mortality for those patients who were predicted to be in needing ICU care yet they did not as Missing-ICU (MI)-mortality. To estimate MI-mortality, a prognostic classification model was built to identify the in-patients who may need ICU care. Its predictive accuracy was 0. 8288, with an AUC of 0. 9119. On our cohort of 733 patients, 25 in-patients who have been predicted by our model that they should need ICU, yet they did not enter ICU due to lack of shorting ICU wards. Our analysis had shown that the MI-mortality is 41%, yet the mortality of ICU is 32%, implying that enough bed of ICU in treating patients in critical conditions.

PDF Details

IJCAI Conference 2021 Conference Paper

Smart Contract Vulnerability Detection: From Pure Neural Network to Interpretable Graph Feature and Expert Pattern Fusion

  • Zhenguang Liu
  • Peng Qian
  • Xiang Wang
  • Lei Zhu
  • Qinming He
  • Shouling Ji

Smart contracts hold digital coins worth billions of dollars, their security issues have drawn extensive attention in the past years. Towards smart contract vulnerability detection, conventional methods heavily rely on fixed expert rules, leading to low accuracy and poor scalability. Recent deep learning approaches alleviate this issue but fail to encode useful expert knowledge. In this paper, we explore combining deep learning with expert patterns in an explainable fashion. Specifically, we develop automatic tools to extract expert patterns from the source code. We then cast the code into a semantic graph to extract deep graph features. Thereafter, the global graph feature and local expert patterns are fused to cooperate and approach the final prediction, while yielding their interpretable weights. Experiments are conducted on all available smart contracts with source code in two platforms, Ethereum and VNT Chain. Empirically, our system significantly outperforms state-of-the-art methods. Our code is released.

PDF Details DOI

AAAI Conference 2021 Conference Paper

Unified Tensor Framework for Incomplete Multi-view Clustering and Missing-view Inferring

  • Jie Wen
  • Zheng Zhang
  • Zhao Zhang
  • Lei Zhu
  • Lunke Fei
  • Bob Zhang
  • Yong Xu

In this paper, we propose a novel method, referred to as incomplete multi-view tensor spectral clustering with missingview inferring (IMVTSC-MVI) to address the challenging multi-view clustering problem with missing views. Different from the existing methods which commonly focus on exploring the certain information of the available views while ignoring both of the hidden information of the missing views and the intra-view information of data, IMVTSC-MVI seeks to recover the missing views and explore the full information of such recovered views and available views for data clustering. In particular, IMVTSC-MVI incorporates the feature space based missing-view inferring and manifold space based similarity graph learning into a unified framework. In such a way, IMVTSC-MVI allows these two learning tasks to facilitate each other and can well explore the hidden information of the missing views. Moreover, IMVTSC-MVI introduces the low-rank tensor constraint to capture the high-order correlations of multiple views. Experimental results on several datasets demonstrate the effectiveness of IMVTSC-MVI for incomplete multi-view clustering.

PDF Details

TIST Journal 2020 Journal Article

Flexible Multi-modal Hashing for Scalable Multimedia Retrieval

  • Lei Zhu
  • Xu Lu
  • Zhiyong Cheng
  • Jingjing Li
  • Huaxiang Zhang

Multi-modal hashing methods could support efficient multimedia retrieval by combining multi-modal features for binary hash learning at the both offline training and online query stages. However, existing multi-modal methods cannot binarize the queries, when only one or part of modalities are provided. In this article, we propose a novel Flexible Multi-modal Hashing (FMH) method to address this problem. FMH learns multiple modality-specific hash codes and multi-modal collaborative hash codes simultaneously within a single model. The hash codes are flexibly generated according to the newly coming queries, which provide any one or combination of modality features. Besides, the hashing learning procedure is efficiently supervised by the pair-wise semantic matrix to enhance the discriminative capability. It could successfully avoid the challenging symmetric semantic matrix factorization and O ( n 2 ) storage cost of semantic matrix. Finally, we design a fast discrete optimization to learn hash codes directly with simple operations. Experiments validate the superiority of the proposed approach.

Details DOI

YNICL Journal 2020 Journal Article

Is monetary reward processing altered in drug-naïve youth with a behavioral addiction? Findings from internet gaming disorder

  • Yuan-Wei Yao
  • Lu Liu
  • Patrick D. Worhunsky
  • Sarah Lichenstein
  • Shan-Shan Ma
  • Lei Zhu
  • Xin-Hui Shi
  • Songshan Yang

Current models of addiction biology highlight altered neural responses to non-drug rewards as a central feature of addiction. However, given that drugs of abuse can directly impact reward-related dopamine circuitry, it is difficult to determine the extent to which reward processing alterations are a trait feature of individuals with addictions, or primarily a consequence of exogenous drug exposure. Examining individuals with behavioral addictions is one promising approach for disentangling neural features of addiction from the direct effects of substance exposure. The current fMRI study compared neural responses during monetary reward processing between drug naïve young adults with a behavioral addiction, internet gaming disorder (IGD; n = 22), and healthy controls (n = 27) using a monetary incentive delay task. Relative to controls, individuals with IGD exhibited blunted caudate activity associated with loss magnitude at the outcome stage, but did not differ from controls in neural activity at other stages. These findings suggest that decreased loss sensitivity might be a critical feature of IGD, whereas alterations in gain processing may be less characteristic of individuals with IGD, relative to those with substance use disorders. Therefore, classic theories of altered reward processing in substance use disorders should be translated to behavioral addictions with caution.

Details DOI

AAAI Conference 2020 Conference Paper

Multi-Feature Discrete Collaborative Filtering for Fast Cold-Start Recommendation

  • Yang Xu
  • Lei Zhu
  • Zhiyong Cheng
  • Jingjing Li
  • Jiande Sun

Hashing is an effective technique to address the largescale recommendation problem, due to its high computation and storage efficiency on calculating the user preferences on items. However, existing hashing-based recommendation methods still suffer from two important problems: 1) Their recommendation process mainly relies on the user-item interactions and single specific content feature. When the interaction history or the content feature is unavailable (the cold-start problem), their performance will be seriously deteriorated. 2) Existing methods learn the hash codes with relaxed optimization or adopt discrete coordinate descent to directly solve binary hash codes, which results in significant quantization loss or consumes considerable computation time. In this paper, we propose a fast cold-start recommendation method, called Multi-Feature Discrete Collaborative Filtering (MFDCF), to solve these problems. Specifically, a lowrank self-weighted multi-feature fusion module is designed to adaptively project the multiple content features into binary yet informative hash codes by fully exploiting their complementarity. Additionally, we develop a fast discrete optimization algorithm to directly compute the binary hash codes with simple operations. Experiments on two public recommendation datasets demonstrate that MFDCF outperforms the stateof-the-arts on various aspects.

PDF Details

AAAI Conference 2019 Conference Paper

Explicit Interaction Model towards Text Classification

  • Cunxiao Du
  • Zhaozheng Chen
  • Fuli Feng
  • Lei Zhu
  • Tian Gan
  • Liqiang Nie

Text classification is one of the fundamental tasks in natural language processing. Recently, deep neural networks have achieved promising performance in the text classification task compared to shallow models. Despite of the significance of deep models, they ignore the fine-grained (matching signals between words and classes) classification clues since their classifications mainly rely on the text-level representations. To address this problem, we introduce the interaction mechanism to incorporate word-level matching signals into the text classification task. In particular, we design a novel framework, EXplicit interAction Model (dubbed as EXAM), equipped with the interaction mechanism. We justified the proposed approach on several benchmark datasets including both multilabel and multi-class text classification tasks. Extensive experimental results demonstrate the superiority of the proposed method. As a byproduct, we have released the codes and parameter settings to facilitate other researches.

PDF Details

AAAI Conference 2019 Conference Paper

From Zero-Shot Learning to Cold-Start Recommendation

  • Jingjing Li
  • Mengmeng Jing
  • Ke Lu
  • Lei Zhu
  • Yang Yang
  • Zi Huang

Zero-shot learning (ZSL) and cold-start recommendation (CSR) are two challenging problems in computer vision and recommender system, respectively. In general, they are independently investigated in different communities. This paper, however, reveals that ZSL and CSR are two extensions of the same intension. Both of them, for instance, attempt to predict unseen classes and involve two spaces, one for direct feature representation and the other for supplementary description. Yet there is no existing approach which addresses CSR from the ZSL perspective. This work, for the first time, formulates CSR as a ZSL problem, and a tailor-made ZSL method is proposed to handle CSR. Specifically, we propose a Lowrank Linear Auto-Encoder (LLAE), which challenges three cruxes, i. e. , domain shift, spurious correlations and computing efficiency, in this paper. LLAE consists of two parts, a low-rank encoder maps user behavior into user attributes and a symmetric decoder reconstructs user behavior from user attributes. Extensive experiments on both ZSL and CSR tasks verify that the proposed method is a win-win formulation, i. e. , not only can CSR be handled by ZSL models with a significant performance improvement compared with several conventional state-of-the-art methods, but the consideration of CSR can benefit ZSL as well.

PDF Details

IJCAI Conference 2019 Conference Paper

Supervised Short-Length Hashing

  • Xingbo Liu
  • Xiushan Nie
  • Quan Zhou
  • Xiaoming Xi
  • Lei Zhu
  • Yilong Yin

Hashing can compress high-dimensional data into compact binary codes, while preserving the similarity, to facilitate efficient retrieval and storage. However, when retrieving using an extremely short length hash code learned by the existing methods, the performance cannot be guaranteed because of severe information loss. To address this issue, in this study, we propose a novel supervised short-length hashing (SSLH). In this proposed SSLH, mutual reconstruction between the short-length hash codes and original features are performed to reduce semantic loss. Furthermore, to enhance the robustness and accuracy of the hash representation, a robust estimator term is added to fully utilize the label information. Extensive experiments conducted on four image benchmarks demonstrate the superior performance of the proposed SSLH with short-length hash codes. In addition, the proposed SSLH outperforms the existing methods, with long-length hash codes. To the best of our knowledge, this is the first linear-based hashing method that focuses on both short and long-length hash codes for maintaining high precision.

PDF Details

IJCAI Conference 2018 Conference Paper

A^3NCF: An Adaptive Aspect Attention Model for Rating Prediction

  • Zhiyong Cheng
  • Ying Ding
  • Xiangnan He
  • Lei Zhu
  • Xuemeng Song
  • Mohan Kankanhalli

Current recommender systems consider the various aspects of items for making accurate recommendations. Different users place different importance to these aspects which can be thought of as a preference/attention weight vector. Most existing recommender systems assume that for an individual, this vector is the same for all items. However, this assumption is often invalid, especially when considering a user's interactions with items of diverse characteristics. To tackle this problem, in this paper, we develop a novel aspect-aware recommender model named A$^3$NCF, which can capture the varying aspect attentions that a user pays to different items. Specifically, we design a new topic model to extract user preferences and item characteristics from review texts. They are then used to 1) guide the representation learning of users and items, and 2) capture a user's special attention on each aspect of the targeted item with an attention network. Through extensive experiments on several large-scale datasets, we demonstrate that our model outperforms the state-of-the-art review-aware recommender systems in the rating prediction task.

PDF Details

IJCAI Conference 2018 Conference Paper

Adaptive Collaborative Similarity Learning for Unsupervised Multi-view Feature Selection

  • Xiao Dong
  • Lei Zhu
  • Xuemeng Song
  • Jingjing Li
  • Zhiyong Cheng

In this paper, we investigate the research problem of unsupervised multi-view feature selection. Conventional solutions first simply combine multiple pre-constructed view-specific similarity structures into a collaborative similarity structure, and then perform the subsequent feature selection. These two processes are separate and independent. The collaborative similarity structure remains fixed during feature selection. Further, the simple undirected view combination may adversely reduce the reliability of the ultimate similarity structure for feature selection, as the view-specific similarity structures generally involve noises and outlying entries. To alleviate these problems, we propose an adaptive collaborative similarity learning (ACSL) for multi-view feature selection. We propose to dynamically learn the collaborative similarity structure, and further integrate it with the ultimate feature selection into a unified framework. Moreover, a reasonable rank constraint is devised to adaptively learn an ideal collaborative similarity structure with proper similarity combination weights and desirable neighbor assignment, both of which could positively facilitate the feature selection. An effective solution guaranteed with the proved convergence is derived to iteratively tackle the formulated optimization problem. Experiments demonstrate the superiority of the proposed approach.

PDF Details

AAAI Conference 2018 Conference Paper

Recurrently Aggregating Deep Features for Salient Object Detection

  • Xiaowei Hu
  • Lei Zhu
  • Jing Qin
  • Chi-Wing Fu
  • Pheng-Ann Heng

Salient object detection is a fundamental yet challenging problem in computer vision, aiming to highlight the most visually distinctive objects or regions in an image. Recent works benefit from the development of fully convolutional neural networks (FCNs) and achieve great success by integrating features from multiple layers of FCNs. However, the integrated features tend to include non-salient regions (due to low level features of the FCN) or lost details of salient objects (due to high level features of the FCN) when producing the saliency maps. In this paper, we develop a novel deep saliency network equipped with recurrently aggregated deep features (RADF) to more accurately detect salient objects from an image by fully exploiting the complementary saliency information captured in different layers. The RADF utilizes the multi-level features integrated from different layers of a FCN to recurrently refine the features at each layer, suppressing the non-salient noise at low-level of the FCN and increasing more salient details into features at high layers. We perform experiments to evaluate the effectiveness of the proposed network on 5 famous saliency detection benchmarks and compare it with 15 state-of-the-art methods. Our method ranks first in 4 of the 5 datasets and second in the left dataset.

PDF Details

IJCAI Conference 2018 Conference Paper

R³Net: Recurrent Residual Refinement Network for Saliency Detection

  • Zijun Deng
  • Xiaowei Hu
  • Lei Zhu
  • Xuemiao Xu
  • Jing Qin
  • Guoqiang Han
  • Pheng-Ann Heng

Saliency detection is a fundamental yet challenging task in computer vision, aiming at highlighting the most visually distinctive objects in an image. We propose a novel recurrent residual refinement network (R^3Net) equipped with residual refinement blocks (RRBs) to more accurately detect salient regions of an input image. Our RRBs learn the residual between the intermediate saliency prediction and the ground truth by alternatively leveraging the low-level integrated features and the high-level integrated features of a fully convolutional network (FCN). While the low-level integrated features are capable of capturing more saliency details, the high-level integrated features can reduce non-salient regions in the intermediate prediction. Furthermore, the RRBs can obtain complementary saliency information of the intermediate prediction, and add the residual into the intermediate prediction to refine the saliency maps. We evaluate the proposed R^3Net on five widely-used saliency detection benchmarks by comparing it with 16 state-of-the-art saliency detectors. Experimental results show that our network outperforms our competitors in all the benchmark datasets.

PDF Details

IROS Conference 2018 Conference Paper

Submap-Based Pose-Graph Visual SLAM: A Robust Visual Exploration and Localization System* The work in this paper is supported by the National Natural Science Foundation of China (61603103, 61673125), the Natural Science Foundation of Guangdong of China (2016A030310293), and the Major Scientific and Technological Special Project of Guangdong of China (2016B090910003)

  • Weinan Chen
  • Lei Zhu
  • Yisheng Guan
  • C. Ronald Kube
  • Hong Zhang 0013

For VSLAM (Visual Simultaneous Localization and Mapping), localization is a challenging task, especially for some challenging situations: textureless frames, motion blur, etc. To build a robust exploration and localization system in a given space, a submap-based VSLAM system is proposed in this paper. Our system uses a submap back-end and a visual front-end. The main advantage of our system is its robustness with respect to tracking failure, a common problem in current VSLAM algorithms. The robustness of our system is compared with the state-of-the-art in terms of average tracking percentage. The precision of our system is also evaluated in terms of ATE (absolute trajectory error) RMSE (root mean square error) comparing the state-of-the-art. The ability of our system in solving the “kidnapped” problem is demonstrated. Our system can improve the robustness of visual localization in challenging situations.

Details

IJCAI Conference 2017 Conference Paper

Dynamic Multi-View Hashing for Online Image Retrieval

  • Liang Xie
  • Jialie Shen
  • Jungong Han
  • Lei Zhu
  • Ling Shao

Advanced hashing technique is essential to facilitate effective large scale online image organization and retrieval, where image contents could be frequently changed. Traditional multi-view hashing methods are developed based on batch-based learning, which leads to very expensive updating cost. Meanwhile, existing online hashing methods mainly focus on single-view data and thus can not achieve promising performance when searching real online images, which are multiple view based data. Further, both types of hashing methods can only produce hash code with fixed length. Consequently they suffer from limited capability to comprehensive characterization of streaming image data in the real world. In this paper, we propose dynamic multi-view hashing (DMVH), which can adaptively augment hash codes according to dynamic changes of image. Meanwhile, DMVH leverages online learning to generate hash codes. It can increase the code length when current code is not able to represent new images effectively. Moreover, to gain further improvement on overall performance, each view is assigned with a weight, which can be efficiently updated during the online learning process. In order to avoid the frequent updating of code length and view weights, an intelligent buffering scheme is also specifically designed to preserve significant data to maintain good effectiveness of DMVH. Experimental results on two real-world image datasets demonstrate superior performance of DWVH over several state-of-the-art hashing methods.

PDF Details

IJCAI Conference 2017 Conference Paper

Exploiting Music Play Sequence for Music Recommendation

  • Zhiyong Cheng
  • Jialie Shen
  • Lei Zhu
  • Mohan Kankanhalli
  • Liqiang Nie

Users leave digital footprints when interacting with various music streaming services. Music play sequence, which contains rich information about personal music preference and song similarity, has been largely ignored in previous music recommender systems. In this paper, we explore the effects of music play sequence on developing effective personalized music recommender systems. Towards the goal, we propose to use word embedding techniques in music play sequences to estimate the similarity between songs. The learned similarity is then embedded into matrix factorization to boost the latent feature learning and discovery. Furthermore, the proposed method only considers the k-nearest songs (e. g. , k = 5) in the learning process and thus avoids the increase of time complexity. Experimental results on two public datasets demonstrate that our methods could significantly improve the performance of both rating prediction and top-n recommendation tasks.

PDF Details

AAAI Conference 2017 Conference Paper

Web-Based Semantic Fragment Discovery for On-Line Lingual-Visual Similarity

  • Xiaoshuai Sun
  • Jiewei Cao
  • Chao Li
  • Lei Zhu
  • Heng Tao Shen

In this paper, we present an automatic approach for on-line discovery of visual-lingual semantic fragments from weakly labeled Internet images. Instead of learning region-entity correspondences from well-labeled image-sentence pairs, our approach directly collects and enhances the weakly labeled visual contents from the Web and constructs an adaptive visual representation which automatically links generic lingual phrases to their related visual contents. To ensure reliable and efficient semantic discovery, we adopt non-parametric density estimation to re-rank the related visual instances and proposed a fast self-similarity-based quality assessment method to identify the high-quality semantic fragments. The discovered semantic fragments provide an adaptive joint representation for texts and images, based on which lingual-visual similarity can be defined for further co-analysis of heterogeneous multimedia data. Experimental results on semantic fragment quality assessment, sentence-based image retrieval, automatic multimedia insertion and ordering demonstrated the effectiveness of the proposed framework. The experiments show that the proposed methods can make effective use of the Web knowledge, and are able to generate competitive results compared to state-of-the-art approaches in various tasks.

PDF Details

IJCAI Conference 2016 Conference Paper

Learning Compact Visual Representation with Canonical Views for Robust Mobile Landmark Search

  • Lei Zhu
  • Jialie Shen
  • Xiaobai Liu
  • Liang Xie
  • Liqiang Nie

Mobile Landmark Search (MLS) recently receives increasing attention. However, it still remains unsolved due to two important issues. One is high bandwidth consumption of query transmission, and the other is the huge visual variations of query images. This paper proposes a Canonical View based Compact Visual Representation (2CVR) to handle these problems via novel three stage learning. First, a submodular function is designed to measure visual representativeness and redundancy of a view set. With it, canonical views, which capture key visual appearances of landmark with limited redundancy, are efficiently discovered with an iterative mining strategy. Second, multimodal sparse coding is applied to transform multiple visual features into an intermediate representation which can robustly characterize visual contents of varied landmark images with only fixed canonical views. Finally, compact binary codes are learned on intermediate representation within a tailored binary embedding model which preserves visual relations of images measured with canonical views and removes noises. With 2CVR, robust visual query processing, low cost of query transmission, and fast search process are simultaneously supported. Experiments demonstrate the superior performance of 2CVR over several state-of-the-art methods.

PDF Details

AAAI Conference 2016 Conference Paper

Online Cross-Modal Hashing for Web Image Retrieval

  • Liang Xie
  • Jialie Shen
  • Lei Zhu

Cross-modal hashing (CMH) is an efficient technique for the fast retrieval of web image data, and it has gained a lot of attentions recently. However, traditional CMH methods usually apply batch learning for generating hash functions and codes. They are inefficient for the retrieval of web images which usually have streaming fashion. Online learning can be exploited for CMH. But existing online hashing methods still cannot solve two essential problems: efficient updating of hash codes and analysis of cross-modal correlation. In this paper, we propose Online Cross-modal Hashing (OCMH) which can effectively address the above two problems by learning the shared latent codes (SLC). In OCMH, hash codes can be represented by the permanent SLC and dynamic transfer matrix. Therefore, inefficient updating of hash codes is transformed to the efficient updating of SLC and transfer matrix, and the time complexity is irrelevant to the database size. Moreover, SLC is shared by all the modalities, and thus it can encode the latent cross-modal correlation, which further improves the overall cross-modal correlation between heterogeneous data. Experimental results on two real-world multi-modal web image datasets: MIR Flickr and NUS-WIDE, demonstrate the effectiveness and efficiency of OCMH for online cross-modal web image retrieval.

PDF Details

YNIMG Journal 2013 Journal Article

Increased neural responses to unfairness in a loss context

  • Xiuyan Guo
  • Li Zheng
  • Lei Zhu
  • Jianqi Li
  • Qianfeng Wang
  • Zoltan Dienes
  • Zhiliang Yang

Unfairness plays an important role in economic decision making. This fMRI study sought to investigate how the loss and the gain contexts could modulate behavioral and brain responses to unfairness by focusing on participants' rejection behaviors during an Ultimatum Game paradigm. Participants were scanned while they were playing the Ultimatum Game as responders in both loss and gain contexts, i. e. receiving ¥50 as gains and paying for ¥50 as losses. At the behavioral level, lower fairness ratings and higher rejection rates were revealed for unfair losses than unfair gains. At the neural level, left dorsolateral prefrontal cortex, bilateral anterior insula, anterior cingulate cortex/anterior middle cingulate cortex and bilateral dorsal striatum were associated with rejection (vs. acceptance) in the loss context, but not in the gain context. Together, our data indicated that participants may experience more unfairness in UG and stronger desire to sanction social norm violations in the loss context than in the gain context, inducing more fairness-related neutral activities when rejecting (vs. accepting) unfair losses than unfair gains. These findings shed light on the significance of context (i. e. loss or gain) in fairness-related social decision-making processes.

Details DOI

ICRA Conference 2011 Conference Paper

The analysis on period doubling gait and chaotic gait of the compass-gait biped model

  • Jie Zhao 0003
  • Xiaoguang Wu
  • Xizhe Zang
  • Yanhe Zhu
  • Lei Zhu

The passive dynamic walking model, which can only depend on the gravity and its own inertia, presents stable, high-efficient, natural periodic gait on a slight slope. The stable periodic gait of the robot has a delicate balance of energy conversion, which makes the gait adjust itself as the parameters of the model change. In our work, the cell mapping method is combined with Newton-Raphson iteration to obtain the limit cycle of the periodic gait in the model, the track stability of the limit cycle is analyzed, and the eigenvalues change rule of Poincare Jacobi matrix is deduced. The influence of changing parameters on the gait is analyzed and discussed by simulations on the model with different sets of parameters. The result suggests that, the location of the center of leg mass too high or too low, foot radius increase or decrease, the slope or moment of inertia increase, will lead to the occurrence of bifurcation of the gait period and chaos; while the way the gait enters chaos from period doubling bifurcation, which results from different parameters change, obeys the law all the period doubling bifurcation share, that is, it has the same Feigenbaum constant. Furthermore, the dynamic features of the robot at the entrance of the chaos are obtained by the rule of the period doubling bifurcation of the gait; meanwhile, it can be found by the analysis of the gait features in the chaos area that there is also certain periodic law in the chaotic gait.

Details
v2026.09.13