Arrow Research search

Author name cluster

Liang Zhang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

59 papers
2 author rows

Possible papers

59

AAAI Conference 2026 Conference Paper

ChartEditor: A Reinforcement Learning Framework for Robust Chart Editing

  • Liangyu Chen
  • Yichen Xu
  • Jianzhe Ma
  • Yuqi Liu
  • Donglu Yang
  • Liang Zhang
  • Zihao Yue
  • Wenxuan Wang

Chart editing reduces manual effort in visualization design. Typical benchmarks assume access to complete chart code, which is unrealistic for real-world applications. In this paper, we present ChartEditVista, a comprehensive benchmark consisting of 7,964 samples spanning 31 chart categories. It encompasses diverse editing instruction types and covers nearly all editable chart elements. The inputs in ChartEditVista include only the original chart image and natural language editing instructions, without the original chart codes. ChartEditVista is generated through a fully automated pipeline that produces, edits, and verifies charts, ensuring high-quality data. Besides, we introduce two novel fine-grained, rule-based evaluation metrics: the layout metric, which evaluates the position, size; and color of graphical components, and the text metric, which jointly assesses textual content and font styling. Building on top of ChartEditVista, we present ChartEditor, a model trained using a reinforcement learning framework that incorporates a novel rendering reward to simultaneously enforce code executability and visual fidelity. Through extensive experiments and human evaluations, we demonstrate that ChartEditVista provides a robust evaluation, while ChartEditor consistently outperforms models with similar-scale and larger-scale on chart editing tasks.

PDF Details DOI

AAAI Conference 2026 Conference Paper

FDC-Ground: Improving GRPO for GUI Grounding via Exponential Rewards and Fact-Aligned Pruning

  • Xiangjian Zeng
  • Wenjing Li
  • Qingqiang Wu
  • Liang Zhang

This paper presents FDC-Ground, a reinforcement learning framework that addresses the high-cost, low-signal challenge of GUI grounding training. The framework introduces two core contributions: (1) the Exponentially Decayed Distance Reward (EDDR), which provides resolution-robust and continuous feedback for position predictions, and (2) the Fact-Aligned Dynamic Completions Pruning (FDC-Pruning) strategy, which selectively retains completions whose advantage signs align with factual correctness, thereby reducing computational overhead while enhancing gradient quality and training stability. Using only 3.2K training samples and a single epoch, our 7B-parameter model achieves 88.3% and 91.0% accuracy on ScreenSpot and ScreenSpot-v2, outperforming several RL-based models such as UIShift and SE-GUI. Our 3B-parameter model based on Qwen2.5-VL-3B surpasses its original performance by +26.6%, demonstrating the effectiveness of our reward design and pruning strategy under low-resource conditions. Furthermore, the proposed FDC-Pruning strategy achieves a 1.18× training speedup and a +5.9% accuracy improvement over standard GRPO, and expanding the exploration space to 4× yields an additional +10.5% gain, confirming both the scalability and the training efficiency of our approach. These findings highlight that combining EDDR with FDC-Pruning offers a practical path toward scalable and efficient RL-based GUI grounding, even in low-resource settings.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Promptus: Can Prompt Streaming Replace Video Streaming

  • Jiangkai Wu
  • Liming Liu
  • Yunpeng Tan
  • Junlin Hao
  • Liang Zhang
  • Xinggong Zhang

With the exponential growth of video traffic, traditional video streaming systems are approaching their limits in communication capacity. To further reduce bitrate while maintaining quality, we propose Promptus, a disruptive semantic communication system that streams prompts instead of videos. Promptus represents the real-world video with a series of "prompts" for delivery and employs Stable Diffusion to generate the same video at the receiver. To ensure that the generated video is pixel-aligned with the original video, a gradient descent-based prompt fitting framework is proposed. Further, a low-rank decomposition-based bitrate control algorithm is introduced to achieve adaptive bitrate. For inter-frame compression, an interpolation-aware fitting algorithm is proposed. Evaluations across various video genres demonstrate that, compared to H.265, Promptus can achieve more than a 4x bandwidth reduction while preserving the same perceptual quality. On the other hand, at extremely low bitrates, Promptus can enhance the perceptual quality by 0.139 and 0.118 (in LPIPS) compared to VAE and H.265, respectively, and decreases the ratio of severely distorted frames by 89.3% and 91.7%. Our work opens up a new paradigm for efficient video communication.

PDF Details DOI

AAMAS Conference 2026 Conference Paper

Rethinking Priority Scheduling for Sequential Multi-Agent Decision Making in Stackelberg Games

  • Xiangyu Liu
  • Liang Zhang
  • Bo Jin
  • Ziqi Wei

Current research applying N-level Stackelberg Game to multi-agent systems often uses the default decision order of agents provided by the environment. However, this raises the question: Does the order of agents necessarily affect the final equilibrium point of the game? To address this, we formally analyze the N-level Stackelberg Game in which changing the order where the agents make decisions typically leads to an overdetermined system. As a result, the equilibrium point is shifted unless special structural conditions are met. Based on this, we propose the Hierarchical Priority Adjustment (HPA) method, which adjusts and selects the agents’ decision order. For the upper level, an upper policy dynamically selects the optimal decision order of agents based on the current game state; for the lower level, agents execute the strategy in the Spatio-Temporal Sequential Markov Game (STMG) based on the selectedorder. Tocoordinatelearningacrosstimescales, weemploy a slow-fast update scheme with shared intrinsic rewards derived from the upper policy advantage function. Experimental results on high-precision control tasks such as multi-agent MuJoCo show that HPA outperforms the benchmark algorithms and robustly adapts to changing environments. These results highlight the crucial role of optimizing the decision order of agents in N-level Stackelberg Game.

PDF

JBHI Journal 2026 Journal Article

RLAD: A Reliable Hippo-Guided Multi-Task Model for Alzheimer's Disease Diagnosis

  • Zhenxin Lei
  • Wenjing Zhu
  • Jiale Liu
  • Cong Hua
  • Johann Li
  • Syed Afaq Ali Shah
  • Liang Zhang
  • Mohammed Bennamoun

Early diagnosis of Alzheimer's disease (AD) is crucial for its prevention, and hippocampal atrophy is a significant lesion for early diagnosis. The current DL-based AD diagnosis methods only focus on either AD classification or hippocampus segmentation independently, neglecting the correlation between the two tasks and lacking pathological interpretability. To address this issue, we propose a Reliable Hippo-guided Learning model for Alzheimer's Disease diagnosis (RLAD), which employs multi-task learning for AD classification as a main task supplemented by hippocampus segmentation. More specifically, our model consists of 1) a hybrid shared features encoder that encodes local and global information in MRI to enhance the model's ability to learn discriminative features; 2) Task Specific Decoders to accomplish AD classification and hippocampus segmentation; and 3) Task Coordination module to correlate the two tasks and guide the classification task to focus on the hippocampus area. Our proposed RLAD model is evaluated on MRI scans of 1631 subjects from three independent datasets, including ADNI-1, ADNI-2, and HarP. Our extensive experimental results demonstrate that the proposed model significantly improves the performance of AD classification and hippocampus segmentation with strong generalization capabilities.

Details DOI

AAAI Conference 2026 Conference Paper

SubGCache: Accelerating Graph-based RAG with Subgraph-level KV Cache

  • Qiuyu Zhu
  • Liang Zhang
  • Qianxiong Xu
  • Cheng Long
  • Jie Zhang

Graph-based retrieval-augmented generation (RAG) enables large language models (LLMs) to incorporate structured knowledge via graph retrieval as contextual input, enhancing more accurate and context-aware reasoning. We observe that for different queries, it could retrieve similar subgraphs as prompts, and thus we propose SubGCache, which aims to reduce inference latency by reusing computation across queries with similar structural prompts (i.e., subgraphs). Specifically, SubGCache clusters queries based on subgraph embeddings, constructs a representative subgraph for each cluster, and pre-computes the key-value (KV) cache of the representative subgraph. For each query with its retrieved subgraph within a cluster, it reuses the pre-computed KV cache of the representative subgraph of the cluster without computing the KV tensors again for saving computation. Extensive experiments on three datasets across multiple LLM backbones and graph-based RAG frameworks demonstrate that SubGCache consistently reduces inference latency with comparable and even improved generation quality, achieving up to 6.68x reduction in time-to-first-token (TTFT).

PDF Details DOI

AAAI Conference 2025 Conference Paper

Adversarial Contrastive Graph Augmentation with Counterfactual Regularization

  • Tao Long
  • Lei Zhang
  • Liang Zhang
  • Laizhong Cui

With the advancement of graph representation learning, self-supervised graph contrastive learning (GCL) has emerged as a key technique in the field. In GCL, positive and negative samples are generated through data augmentation. While recent works have introduced model-based methods to enhance positive graph augmentations, they often overlook the importance of negative samples, relying instead on rule-based methods that can fail to capture meaningful graph patterns. To address this issue, we propose a novel model-based adversarial contrastive graph augmentation (ACGA) method that automatically generates both positive graph samples with minimal sufficient information and hard negative graph samples. Additionally, we provide a theoretical framework to analyze the process of positive and negative graph augmentation in self-supervised GCL. We evaluate our ACGA method through extensive experiments on representative benchmark datasets, and the results demonstrate that ACGA outperforms state-of-the-art baselines.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

Beyond Last-Click: An Optimal Mechanism for Ad Attribution

  • Nan An
  • Weian Li
  • Qi Qi
  • Changyuan Yu
  • Liang Zhang

Accurate attribution for multiple platforms is critical for evaluating performance-based advertising. However, existing attribution methods rely heavily on the heuristic methods, e. g. , Last-Click Mechanism (LCM) which always allocates the attribution to the platform with the latest report, lacking theoretical guarantees for attribution accuracy. In this work, we propose a novel theoretical model for the advertising attribution problem, in which we aim to design the optimal dominant strategy incentive compatible (DSIC) mechanisms and evaluate their performance. We first show that LCM is not DSIC and performs poorly in terms of accuracy and fairness. To address this limitation, we introduce the Peer-Validated Mechanism (PVM), a DSIC mechanism in which a platform's attribution depends solely on the reports of other platforms. We then examine the accuracy of PVM across both homogeneous and heterogeneous settings, and provide provable accuracy bounds for each case. Notably, we show that PVM is the optimal DSIC mechanism in the homogeneous setting. Finally, numerical experiments are conducted to show that PVM consistently outperforms LCM in terms of attribution accuracy and fairness.

PDF Details

JBHI Journal 2025 Journal Article

CrossConvPyramid: Deep Multimodal Fusion for Epileptic Magnetoencephalography Spike Detection

  • Liang Zhang
  • Shurong Sheng
  • Xiongfei Wang
  • Jia-Hong Gao
  • Yi Sun
  • Kuntao Xiao
  • Wanli Yang
  • Pengfei Teng

Magnetoencephalography (MEG) is a vital non-invasive tool for epilepsy analysis, as it captures high-resolution signals that reflect changes in brain activity over time. The automated detection of epileptic spikes within these signals can significantly reduce the labor and time required for manual annotation of MEG recording data, thereby aiding clinicians in identifying epileptogenic foci and evaluating treatment prognosis. Research in this domain often utilizes the raw, multi-channel signals from MEG scans for spike detection, commonly neglecting the multi-channel spiking patterns from spatially adjacent channels. Moreover, epileptic spikes share considerable morphological similarities with artifact signals within the recordings, posing a challenge for models to differentiate between the two. In this paper, we introduce a multimodal fusion framework that addresses these two challenges collectively. Instead of relying solely on the signal recordings, our framework also mines knowledge from their corresponding topography-map images, which encapsulate the spatial context and amplitude distribution of the input signals. To facilitate more effective data fusion, we present a novel multimodal feature fusion technique called CrossConvPyramid, built upon a convolutional pyramid architecture augmented by an attention mechanism. It initially employs cross-attention and a convolutional pyramid to encode inter-modal correlations within the intermediate features extracted by individual unimodal networks. Subsequently, it utilizes a self-attention mechanism to refine and select the most salient features from both inter-modal and unimodal features, specifically tailored for the spike classification task. Our method achieved the average F1 scores of 92. 88% and 95. 23% across two distinct real-world MEG datasets from separate centers, respectively outperforming the current state-of-the-art by 2. 31% and 0. 88%. We plan to release the code on GitHub later.

Details DOI

NeurIPS Conference 2025 Conference Paper

Enhancing Interpretability in Deep Reinforcement Learning through Semantic Clustering

  • Liang Zhang
  • Justin Lieffers
  • Adarsh Pyarelal

In this paper, we explore semantic clustering properties of deep reinforcement learning (DRL) to improve its interpretability and deepen our understanding of its internal semantic organization. In this context, semantic clustering refers to the ability of neural networks to cluster inputs based on their semantic similarity in the feature space. We propose a DRL architecture that incorporates a novel semantic clustering module that combines feature dimensionality reduction with online clustering. This module integrates seamlessly into the DRL training pipeline, addressing the instability of t-SNE and eliminating the need for extensive manual annotation inherent to prior semantic analysis methods. We experimentally validate the effectiveness of the proposed module and demonstrate its ability to reveal semantic clustering properties within DRL. Furthermore, we introduce new analytical methods based on these properties to provide insights into the hierarchical structure of policies and semantic organization within the feature space. Our code is available at https: //github. com/ualiangzhang/semantic_rl.

PDF Details

JBHI Journal 2025 Journal Article

FDDSeg: Unleashing the Power of Scribble Annotation for Cardiac MRI Images Through Feature Decomposition Distillation

  • Lin Zhang
  • Wenzong Li
  • Kaiyue Bi
  • Pei Li
  • Liang Zhang
  • Hui Liu

Cardiovascular diseases can be diagnosed with computer assistance when using the magnetic resonance imaging (MRI) image that is produced by the MRI sensor. Deep learning-based scribbling MRI image segmentation has demonstrated impressive results recently. However, the majority of current approaches possess an excessive number of model parameters and do not completely utilize scribbling annotations. To develop a feature decomposition distillation deep learning method, named FDDSeg, for scribble-supervised cardiac MRI image segmentation. Public ACDC and MSCMR cardiac MRI datasets were used to evaluate the segmentation performance of FDDSeg. FDDSeg adopts a scribble annotation reuse policy to help provide accurate boundaries, and the intermediate features are split class region and class-free region by using the pseudo labels to further improve feature learning. Effective distillation knowledge is then captured by feature decomposition. FDDSeg was compared with 7 state-of-the-art methods, MAAG, ShapePU, CycleMix, Dual-Branch, ZscribbleSeg, Perturbation Dual-Branch as well as ScribbleVC on both ACDC and MSCMR datasets. FDDSeg is shown to perform the best in DSC(89. 05% and 88. 75%), JC(80. 30% and 79. 78%) as well as HD95(5. 76% and 4. 44%) metrics with only 2. 01 M of parameters. FDDSeg methods can segment cardiac MRI images more precise with only scribble annotations at lower computation cost, which may help increase the efficiency of quantitative analysis of cardiac.

Details DOI

ICML Conference 2025 Conference Paper

Generative Human Trajectory Recovery via Embedding-Space Conditional Diffusion

  • Kaijun Liu
  • Sijie Ruan
  • Liang Zhang
  • Cheng Long 0001
  • Shuliang Wang 0001
  • Liang Yu 0005

Recovering human trajectories from incomplete or missing data is crucial for many mobility-based urban applications, e. g. , urban planning, transportation, and location-based services. Existing methods mainly rely on recurrent neural networks or attention mechanisms. Though promising, they encounter limitations in capturing complex spatial-temporal dependencies in low-sampling trajectories. Recently, diffusion models show potential in content generation. However, most of proposed methods are used to generate contents in continuous numerical representations, which cannot be directly adapted to the human location trajectory recovery. In this paper, we introduce a conditional diffusion-based trajectory recovery method, namely, DiffMove. It first transforms locations in trajectories into the embedding space, in which the embedding denoising is performed, and then missing locations are recovered by an embedding decoder. DiffMove not only improves accuracy by introducing high-quality generative methods in the trajectory recovery, but also carefully models the transition, periodicity, and temporal patterns in human mobility. Extensive experiments based on two representative real-world mobility datasets are conducted, and the results show significant improvements (an average of 11% in recall) over the best baselines.

Details

EAAI Journal 2025 Journal Article

Heterogeneous ensemble short-term integrated load forecasting considering differentiated cycle-trend properties

  • Nantian Huang
  • Yaoyao Wang
  • Bingling Li
  • Guowei Cai
  • Liang Zhang

Aiming at the Integrated Energy System (IES), there are key problems such as the potential change pattern of seasonal differentiation of multiple loads, the lack of mining complex coupling characteristics, the limited generalization ability of forecasting models and the weak interpretability. A heterogeneous integrated short-term multiple load forecasting method considering differentiated cycle-trend characteristics is proposed. Firstly, Multiple Seasonal Trend decomposition using Loess (MSTL) is used to decompose the multivariate loads into multi-timescale cycle and trend series, and to construct multi-timescale cycle-trend feature sets, which breaks through the limitations of the traditional single-cycle decomposition in characterizing inter-seasonal coupling characteristics. Secondly, the evolutionary multi-objective optimization mechanism based on the dual indexes of diversity and accuracy is introduced into the Stacking integration framework for the first time, and the seasonal adaptive optimal model combination selection of electric/cooling/heating loads is realized. Finally, a two-layer interpretable analysis paradigm based on Shapley Additive Explanations (SHAP) values is established to improve the interpretability of the proposed multivariate load forecasting method by performing input feature attribution analysis from both global and individual dimensions. The experimental analysis is carried out on the example of Tempe Campus IES of Arizona State University in the U. S. A. The results show that the average percentage errors of electricity, cooling, and heating forecasts of the proposed method reach 1. 783 %, 1. 012 %, and 2. 225 %, respectively, demonstrating superior forecasting accuracy compared to existing methods.

Details DOI

JBHI Journal 2025 Journal Article

MADRNet: Morphology-Aware Dual-Path Reversible Network for Sperm Classification

  • Fan Yang
  • Jingzhang Sun
  • Honglan Huang
  • Liang Zhang
  • Jiheng Zhang

Sperm morphology analysis plays a crucial role in the clinical diagnosis of male infertility. However, manual evaluation is inherently subjective, and inconsistencies in diagnostic criteria may compromise accuracy. Some existing sperm image classification models are introduced but requiring manual intervention. Most models lack of consideration of alignment between computational classification and WHO sperm morphology standards. To address these challenges, we propose an innovative morphology-aware dual-path reversible network ( MADRNet ) in designing our model. We integrate key biomarkers, such as head aspect ratio and acrosomal integrity, both of which are crucial for clinical sperm assessment, into the network. Particularly, the network utilizes a dual-path attention mechanism, incorporating both parallel spatial and channel attention, while embedding the acrosome anatomical constraint within the channel attention. To further enhance the alignment of our model with the WHO standards, we develop a dynamic loss function considering head aspect ratio constraint. Further, we employ a reversible architecture to enable the model to preserve more microscopic details while reducing GPU memory consumption. Experiments on the HuSHeM dataset demonstrate that the model achieves an accuracy of 96. 3% and an F1 score of 96. 8%. Meanwhile, the model maintains a real-time processing speed of 32 ms per image, providing a precise and efficient solution for clinical sperm screening. The implementation source code and the underlying dataset are available at https://github.com/fanyangZK/MADRNet.

Details DOI

AAAI Conference 2025 Conference Paper

Merging Mechanisms for Ads and Organic Items in E-commerce Platforms

  • Nan An
  • Weian Li
  • Qi Qi
  • Liang Zhang

In contemporary e-commerce platforms, search result pages display two types of items: ad items and organic items. Ad items are determined through an advertising auction system, while organic items are selected by a recommendation system. These systems have distinct optimization objectives, creating the challenge of effectively merging these two components. Recent research has explored merging mechanisms for e-commerce platforms, but none have simultaneously achieved all desirable properties: incentive compatibility, individual rationality, adaptability to multiple slots, integration of inseparable candidates, and avoidance of repeated exposure for ads and organic items. This paper addresses the design of a merging mechanism that satisfies all these properties. We first provide the necessary conditions for the optimal merging mechanisms. Next, we introduce two simple and effective mechanisms, termed the generalized fix mechanism and the generalized change mechanism. Finally, we theoretically prove that both mechanisms offer guaranteed approximation ratios compared to the optimal mechanism in both simplest and general settings.

PDF Details DOI

IROS Conference 2025 Conference Paper

Natural Humanoid Robot Locomotion with Generative Motion Prior

  • Haodong Zhang
  • Liang Zhang
  • Zhenghan Chen
  • Lu Chen
  • Yue Wang 0020
  • Rong Xiong

Natural and lifelike locomotion remains a fundamental challenge for humanoid robots to interact with human society. However, previous methods either neglect motion naturalness or rely on unstable and ambiguous style rewards. In this paper, we propose a novel Generative Motion Prior (GMP) that provides fine-grained motion-level supervision for the task of natural humanoid robot locomotion. To leverage natural human motions, we first employ whole-body motion retargeting to effectively transfer them to the robot. Subsequently, we train a generative model offline to predict future natural reference motions for the robot based on a conditional variational auto-encoder. During policy training, the generative motion prior serves as a frozen online motion generator, delivering precise and comprehensive supervision at the trajectory level, including joint angles and keypoint positions. The generative motion prior significantly enhances training stability and improves interpretability by offering detailed and dense guidance throughout the learning process. Experimental results in both simulation and real-world environments demonstrate that our method achieves superior motion naturalness compared to existing approaches. Project page can be found at https://sites.google.com/view/humanoid-gmp

Details

ICLR Conference 2025 Conference Paper

On the Crucial Role of Initialization for Matrix Factorization

  • Bingcong Li
  • Liang Zhang
  • Aryan Mokhtari
  • Niao He

This work revisits the classical low-rank matrix factorization problem and unveils the critical role of initialization in shaping convergence rates for such nonconvex and nonsmooth optimization. We introduce Nystrom initialization, which significantly improves the global convergence of Scaled Gradient Descent (ScaledGD) in both symmetric and asymmetric matrix factorization tasks. Specifically, we prove that ScaledGD with Nystrom initialization achieves quadratic convergence in cases where only linear rates were previously known. Furthermore, we extend this initialization to low-rank adapters (LoRA) commonly used for finetuning foundation models. Our approach, NoRA, i.e., LoRA with Nystrom initialization, demonstrates superior performance across various downstream tasks and model scales, from 1B to 7B parameters, in large language and diffusion models.

Details

NeurIPS Conference 2025 Conference Paper

PoLAR: Polar-Decomposed Low-Rank Adapter Representation

  • Kai Lion
  • Liang Zhang
  • Bingcong Li
  • Niao He

We show that low-rank adaptation of large-scale models suffers from a low stable rank that is well below the linear algebraic rank of the subspace, degrading fine-tuning performance. To mitigate the underutilization of the allocated subspace, we propose PoLAR, a parameterization inspired by the polar decomposition that factorizes the low-rank update into two direction matrices constrained to Stiefel manifolds and an unconstrained scale matrix. Our theory shows that PoLAR yields an exponentially faster convergence rate on a canonical low-rank adaptation problem. Pairing the parameterization with Riemannian optimization leads to consistent gains on three different benchmarks testing general language understanding, commonsense reasoning, and mathematical problem solving with base model sizes ranging from 350M to 27B.

PDF Details

EAAI Journal 2025 Journal Article

Process planning of parameter intelligent adjustment for batch machining based on historical data segmented modeling

  • Juan Lu
  • Shiying Tu
  • Ying Li
  • Liang Zhang
  • Xiaoping Liao

Improving machining benefit while maintaining stable quality and performance is a prerequisite for improving intelligent manufacturing and enterprise competitiveness. This paper attempts to explore the method of segmented optimization of process parameters to improve benefit (such as machining time and tool life) while maintaining quality and performance within a given range during the tool life cycle in batch machining. An intelligent parameter adjustment framework is proposed, which utilizes the historical machining data to construct the process parameter adjustment time series (i. e. , process planning) in the tool life cycle, and by online adding data from new tool machining processes, the time series can be modified. The key technology of the framework is to determine the adjustment point and the corresponding process parameters. A segmented prediction model based on historical data is presented to predict the adjustment point during the new tool machining process of, and a double depth Q-network (DDQN) is utilized to obtain the optimized machining parameters at the adjustment point. Based on a milling experiment of TC18, the effectiveness of the proposed method is validated by the prediction and optimization of the first adjustment point. The adjustment point is predicted to be the 58th machining of the new tool, after adjusting the process parameters obtained through DDQN at the adjustment point, the quality and cutting force remain within the set threshold, cutting time decreases by 0. 2% and tool life increases by 14. 42%, which shows that the proposed methods achieve quality assurance and benefit enhancement during the machining process.

Details DOI

NeurIPS Conference 2025 Conference Paper

Prompt-guided Disentangled Representation for Action Recognition

  • Tianci Wu
  • Guangming Zhu
  • Lu Jiang
  • Siyuan Wang
  • Ning Wang
  • Nuoye Xiong
  • Liang Zhang

Action recognition is a fundamental task in video understanding. Existing methods typically extract unified features to process all actions in one video, which makes it challenging to model the interactions between different objects in multi-action scenarios. To alleviate this issue, we explore disentangling any specified actions from complex scenes as an effective solution. In this paper, we propose Prompt-guided Disentangled Representation for Action Recognition (ProDA), a novel framework that disentangles any specified actions from a multi-action scene. ProDA leverages Spatio-temporal Scene Graphs (SSGs) and introduces Dynamic Prompt Module (DPM) to guide a Graph Parsing Neural Network (GPNN) in generating action-specific representations. Furthermore, we design a video-adapted GPNN that aggregates information using dynamic weights. Extensive experiments on two complex video action datasets, Charades and SportsHHI, demonstrate the effectiveness of our approach against state-of-the-art methods. Our code can be found in https: //github. com/iamsnaping/ProDA. git.

PDF Details

NeurIPS Conference 2025 Conference Paper

Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding

  • Ye Wang
  • Ziheng Wang
  • Boshen Xu
  • Yang Du
  • Kejun Lin
  • Zihan Xiao
  • Zihao Yue
  • Jianzhong Ju

Temporal Video Grounding (TVG), the task of locating specific video segments based on language queries, is a core challenge in long-form video understanding. While recent Large Vision-Language Models (LVLMs) have shown early promise in tackling TVG through supervised fine-tuning (SFT), their ability to generalize remains limited. To address this, we propose a novel post-training framework that enhances the generalization capabilities of LVLMs via reinforcement learning (RL). Specifically, our contributions span three key directions: (1) Time-R1: we introduce a reasoning-guided post-training framework via RL with verifiable reward to enhance capabilities of LVLMs on the TVG task. (2) TimeRFT: we explore post-training strategies on our curated RL-friendly dataset, which trains the model to progressively comprehend more difficult samples, leading to better generalization. (3) TVGBench: we carefully construct a small but comprehensive and balanced benchmark suitable for LVLM evaluation, which is sourced from available public benchmarks. Extensive experiments demonstrate that Time-R1 achieves state-of-the-art performance across multiple downstream datasets using significantly less training data than prior LVLM approaches, while improving its general video understanding capabilities. Project Page: https: //xuboshen. github. io/Time-R1/.

PDF Details

NeurIPS Conference 2025 Conference Paper

Venus-MAXWELL: Efficient Learning of Protein-Mutation Stability Landscapes using Protein Language Models

  • Yuanxi Yu
  • Fan Jiang
  • Xinzhu Ma
  • Liang Zhang
  • Bozitao Zhong
  • Wanli Ouyang
  • Guisheng Fan
  • Huiqun Yu

In-silico prediction of protein mutant stability, measured by the difference in Gibbs free energy change ($\Delta \Delta G$), is fundamental for protein engineering. Current sequence-to-label methods typically employ two-stage pipelines: (i) encoding mutant sequences using neural networks (e. g. , transformers), followed by (ii) the $\Delta \Delta G$ regression from the latent representations. Although these methods have demonstrated promising performance, their dependence on specialized neural network encoders significantly increases the complexity. Additionally, the requirement to compute latent representations individually for each mutant sequence negatively impacts computational efficiency and poses the risk of overfitting. This work proposes the Venus-MAXWELL framework, which reformulates mutation $\Delta \Delta G$ prediction as a sequence-to-landscape task. In Venus-MAXWELL, mutations of a protein and their corresponding $\Delta \Delta G$ values are organized into a landscape matrix, allowing our framework to learn the $\Delta \Delta G$ landscape of a protein with a single forward and backward pass during training. To this end, we curated a new $\Delta \Delta G$ benchmark dataset with strict controls on data leakage and redundancy to ensure robust evaluation. Leveraging the zero-shot scoring capability of protein language models (PLMs), Venus-MAXWELL effectively utilizes the evolutionary patterns learned by PLMs during pre-training. More importantly, Venus-MAXWELL is compatible with multiple protein language models. For example, when integrated with the ESM-IF, Venus-MAXWELL achieves higher accuracy than ThermoMPNN with 10$\times$ faster in inference speed (despite having 50$\times$ more parameters than ThermoMPNN). The training codes, model weights, and datasets are publicly available at https: //github. com/ai4protein/Venus-MAXWELL.

PDF Details

NeurIPS Conference 2025 Conference Paper

Zeroth-Order Optimization Finds Flat Minima

  • Liang Zhang
  • Bingcong Li
  • Kiran Thekumparampil
  • Sewoong Oh
  • Michael Muehlebach
  • Niao He

Zeroth-order methods are extensively used in machine learning applications where gradients are infeasible or expensive to compute, such as black-box attacks, reinforcement learning, and language model fine-tuning. Existing optimization theory focuses on convergence to an arbitrary stationary point, but less is known on the implicit regularization that provides a fine-grained characterization on which particular solutions are finally reached. We show that zeroth-order optimization with the standard two-point estimator favors solutions with small trace of Hessian, which is widely used in previous work to distinguish between sharp and flat minima. We further provide convergence rates of zeroth-order optimization to approximate flat minima for convex and sufficiently smooth functions, where flat minima are defined as the minimizers that achieve the smallest trace of Hessian among all optimal solutions. Experiments on binary classification tasks with convex losses and language model fine-tuning support our theoretical findings.

PDF Details

ICLR Conference 2024 Conference Paper

Causality-Inspired Spatial-Temporal Explanations for Dynamic Graph Neural Networks

  • Kesen Zhao
  • Liang Zhang

Dynamic Graph Neural Networks (DyGNNs) have gained significant popularity in the research of dynamic graphs, but are limited by the low transparency, such that human-understandable insights can hardly be drawn from their predictions. Although a number of existing research have been devoted to investigating the interpretability of graph neural networks (GNNs), achieving the interpretability of DyGNNs is pivotally challenging due to the complex spatial-temporal correlations in dynamic graphs. To this end, we propose an innovative causality-inspired generative model based on structural causal model (SCM), which explores the underlying philosophies of DyGNN predictions by identifying the trivial, static, and dynamic causal relationships. To reach this goal, two critical tasks need to be accomplished including (1) disentangling the complex causal relationships, and (2) fitting the spatial-temporal explanations of DyGNNs in the SCM architecture. To tackle these challenges, the proposed method incorporates a contrastive learning module to disentangle trivial and causal relationships, and a dynamic correlating module to disentangle dynamic and static causal relationships, respectively. A dynamic VGAE-based framework is further developed, which generates causal-and-dynamic masks for spatial interpretability, and recognizes dynamic relationships along the time horizon through causal invention for temporal interpretability. Comprehensive experiments have been conducted on both synthetic and real-world datasets, where our approach yields substantial improvements, thereby demonstrating significant superiority.

Details

AAAI Conference 2024 Conference Paper

Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment

  • Rui Zhao
  • Liang Zhang
  • Biao Fu
  • Cong Hu
  • Jinsong Su
  • Yidong Chen

Sign language translation (SLT) aims to convert continuous sign language videos into textual sentences. As a typical multi-modal task, there exists an inherent modality gap between sign language videos and spoken language text, which makes the cross-modal alignment between visual and textual modalities crucial. However, previous studies tend to rely on an intermediate sign gloss representation to help alleviate the cross-modal problem thereby neglecting the alignment across modalities that may lead to compromised results. To address this issue, we propose a novel framework based on Conditional Variational autoencoder for SLT (CV-SLT) that facilitates direct and sufficient cross-modal alignment between sign language videos and spoken language text. Specifically, our CV-SLT consists of two paths with two Kullback-Leibler (KL) divergences to regularize the outputs of the encoder and decoder, respectively. In the prior path, the model solely relies on visual information to predict the target text; whereas in the posterior path, it simultaneously encodes visual information and textual knowledge to reconstruct the target text. The first KL divergence optimizes the conditional variational autoencoder and regularizes the encoder outputs, while the second KL divergence performs a self-distillation from the posterior path to the prior path, ensuring the consistency of decoder outputs.We further enhance the integration of textual information to the posterior path by employing a shared Attention Residual Gaussian Distribution (ARGD), which considers the textual information in the posterior path as a residual component relative to the prior path. Extensive experiments conducted on public datasets demonstrate the effectiveness of our framework, achieving new state-of-the-art results while significantly alleviating the cross-modal representation discrepancy. The code and models are available at https://github.com/rzhao-zhsq/CV-SLT.

PDF Details DOI

EAAI Journal 2024 Journal Article

DCDSum: An interpretable extractive summarization framework based on contrastive learning method

  • Jiaqi Zhang
  • Ling Lu
  • Liang Zhang
  • Yinong Chen
  • Wanping Liu

As the phenomenon of knowledge overload becomes more and more obvious, the automatic summarization technology still needs to break through the bottleneck in order to improve the application value and expand the scope of the application. Traditional training paradigms for extractive summarization systems suffer from the inconsistency in training and evaluation. In this paper, we propose an innovative and interpretable contrastive learning based framework for extractive summarization called DCDSum, which comprises a Diverse Oracle evaluator, a Contrastive learning extractor, and a Dynamic Top-k selector. Different from previous models that consider the extractive summarization task as a sequence labeling problem, our contrastive learning extractor treats it as a sentence reranking problem and introduces contrastive loss to achieve it, which can bridge the gap between objective function and evaluation metrics. The experimental results demonstrate the outstanding performance of our approach on the CNN/DailyMail, XSum, and PubMed datasets, achieving highly competitive results. In particular, our method achieves ROUGE-1 of 44. 65, ROUGE-2 of 21. 32, and ROUGE-L of 40. 87 on the CNN/DailyMail dataset. The outcomes across various evaluation metrics substantiate that the Diverse Oracle extraction algorithm adeptly captures a broader array of sentences with reduced redundancy, consequently enhancing the interpretability of the DCDSum framework.

Details DOI

ICML Conference 2024 Conference Paper

DPZero: Private Fine-Tuning of Language Models without Backpropagation

  • Liang Zhang
  • Bingcong Li
  • Kiran Koshy Thekumparampil
  • Sewoong Oh
  • Niao He

The widespread practice of fine-tuning large language models (LLMs) on domain-specific data faces two major challenges in memory and privacy. First, as the size of LLMs continues to grow, the memory demands of gradient-based training methods via backpropagation become prohibitively high. Second, given the tendency of LLMs to memorize training data, it is important to protect potentially sensitive information in the fine-tuning data from being regurgitated. Zeroth-order methods, which rely solely on forward passes, substantially reduce memory consumption during training. However, directly combining them with standard differentially private gradient descent suffers more as model size grows. To bridge this gap, we introduce DPZero, a novel private zeroth-order algorithm with nearly dimension-independent rates. The memory efficiency of DPZero is demonstrated in privately fine-tuning RoBERTa and OPT on several downstream tasks. Our code is available at https: //github. com/Liang137/DPZero.

Details

AAAI Conference 2024 Conference Paper

Enhance Sketch Recognition’s Explainability via Semantic Component-Level Parsing

  • Guangming Zhu
  • Siyuan Wang
  • Tianci Wu
  • Liang Zhang

Free-hand sketches are appealing for humans as a universal tool to depict the visual world. Humans can recognize varied sketches of a category easily by identifying the concurrence and layout of the intrinsic semantic components of the category, since humans draw free-hand sketches based a common consensus that which types of semantic components constitute each sketch category. For example, an airplane should at least have a fuselage and wings. Based on this analysis, a semantic component-level memory module is constructed and embedded in the proposed structured sketch recognition network in this paper. The memory keys representing semantic components of each sketch category can be self-learned and enhance the recognition network's explainability. Our proposed networks can deal with different situations of sketch recognition, i.e., with or without semantic components labels of strokes. Experiments on the SPG and SketchIME datasets demonstrate the memory module's flexibility and the recognition network's explainability. The code and data are available at https://github.com/GuangmingZhu/SketchESC.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems

  • Bingcong Li
  • Liang Zhang
  • Niao He

Sharpness-aware minimization (SAM) improves generalization of various deep learning tasks. Motivated by popular architectures such as LoRA, we explore the implicit regularization of SAM for scale-invariant problems involving two groups of variables. Instead of focusing on commonly used sharpness, this work introduces a concept termed balancedness, defined as the difference between the squared norm of two variables. This allows us to depict richer global behaviors of SAM. In particular, our theoretical and empirical findings reveal that i) SAM promotes balancedness; and ii) the regularization on balancedness is data-responsive -- outliers have stronger impact. The latter coincides with empirical observations that SAM outperforms SGD in the presence of outliers. Leveraging the implicit regularization, we develop a resource-efficient SAM variant, balancedness-aware regularization (BAR), tailored for scale-invariant problems such as finetuning language models with LoRA. BAR saves 95% computational overhead of SAM, with enhanced test performance across various tasks on RoBERTa, GPT2, and OPT-1. 3B.

PDF Details DOI

ECAI Conference 2024 Conference Paper

Improving Non-Autoregressive Sign Language Translation with Random Ordering Progressive Prediction Pretraining

  • Pei Yu
  • Changhao Lai
  • Cong Hu
  • Shan Liu
  • Liang Zhang
  • Biao Fu
  • Yidong Chen 0001

Recently, the Non-AutoRegressive (NAR) decoding mechanism, effectively reducing the inference latency of text generation, has been applied to Sign Language Translation (SLT). Typically, the current best NAR SLT model using a Curriculum-based Non-autoregressive Decoder (CND) outperforms AutoRegressive (AR) baselines in speed and performance. Although it has been proven that AutoRegressive Pre-trained Language Models (AR-PLMs) further boost the performance of AR SLT models, combining NAR Pretrained Language Models (NAR-PLMs) with NAR SLT model remains challenge due to (1) existing NAR-PLMs’ inability to model token dependencies between decoder layers, crucial for NAR SLT models using CND; (2) the modality gap between the decoder’s inputs of the NAR-PLMs and NAR SLT models. To address these, we propose a Random Ordering Progressive Prediction Pre-training task for NAR SLT models using CND, enabling the decoder to predict target sequences in diverse orderings and enhancing the modeling of target token dependencies between layers. Moreover, we propose a CTC-enhanced Soft Copy method to incorporate target-side information in the decoder’s inputs, alleviating the modality gap. Experimental results on PHOENIX-2014T and CSL-Daily demonstrate that our model consistently outperforms all strong baselines and achieves competitive performance with AR SLT models equipped with AR-PLMs.

Details

AAAI Conference 2023 Conference Paper

Accommodating Audio Modality in CLIP for Multimodal Processing

  • Ludan Ruan
  • Anwen Hu
  • Yuqing Song
  • Liang Zhang
  • Sipeng Zheng
  • Qin Jin

Multimodal processing has attracted much attention lately especially with the success of pre-training. However, the exploration has mainly focused on vision-language pre-training, as introducing more modalities can greatly complicate model design and optimization. In this paper, we extend the state-of-the-art Vision-Language model CLIP to accommodate the audio modality for Vision-Language-Audio multimodal processing. Specifically, we apply inter-modal and intra-modal contrastive learning to explore the correlation between audio and other modalities in addition to the inner characteristics of the audio modality. Moreover, we further design an audio type token to dynamically learn different audio information type for different scenarios, as both verbal and nonverbal heterogeneous information is conveyed in general audios. Our proposed CLIP4VLA model is validated in different downstream tasks including video retrieval and video captioning, and achieves the state-of-the-art performance on the benchmark datasets of MSR-VTT, VATEX, and Audiocaps.The corresponding code and checkpoints will be released at https://github.com/ludanruan/CLIP4VLA.

PDF Details DOI

TCS Journal 2023 Journal Article

Complexity and approximation algorithms for two parallel dedicated machine scheduling with conflict constraints

  • An Zhang
  • Liang Zhang
  • Yong Chen
  • Guangting Chen
  • Xing Wang

We investigate two parallel dedicated machine scheduling with conflict constraints. The problem of minimizing the makespan has been shown to be NP-hard in the strong sense under the assumption that the processing sequence of jobs on one machine is given and fixed a priori. The problem without any fixed sequence was previously recognized as weakly NP-hard. In this paper, we first present a 9 5 -approximation algorithm for the problem with a fixed sequence. Then we show that the tight approximation ratios of the algorithm are 7 4 and 5 3 for two subproblems which remain strongly NP-hard. We also send an improved algorithm with approximation ratio 3 − 2 ≈ 1. 586 for one subproblem. Finally, we prove that the problem without any fixed sequence is actually strongly NP-hard, and design a 5 3 -approximation algorithm.

Details DOI

IJCAI Conference 2023 Conference Paper

Efficient Sign Language Translation with a Curriculum-based Non-autoregressive Decoder

  • Pei Yu
  • Liang Zhang
  • Biao Fu
  • Yidong Chen

Most existing studies on Sign Language Translation (SLT) employ AutoRegressive Decoding Mechanism (AR-DM) to generate target sentences. However, the main disadvantage of the AR-DM is high inference latency. To address this problem, we introduce Non-AutoRegressive Decoding Mechanism (NAR-DM) into SLT, which generates the whole sentence at once. Meanwhile, to improve its decoding ability, we integrate the advantages of curriculum learning and NAR-DM and propose a Curriculum-based NAR Decoder (CND). Specifically, the lower layers of the CND are expected to predict simple tokens that could be predicted correctly using source-side information solely. Meanwhile, the upper layers could predict complex tokens based on the lower layers' predictions. Therefore, our CND significantly reduces the model's inference latency while maintaining its competitive performance. Moreover, to further boost the performance of our CND, we propose a mutual learning framework, containing two decoders, i. e. , an AR decoder and our CND. We jointly train the two decoders and minimize the KL divergence between their outputs, which enables our CND to learn the forward sequential knowledge from the strengthened AR decoder. Experimental results on PHOENIX2014T and CSL-Daily demonstrate that our model consistently outperforms all competitive baselines and achieves 7. 92/8. 02× speed-up compared to the AR SLT model respectively. Our source code is available at https: //github. com/yp20000921/CND.

PDF Details DOI

IJCAI Conference 2023 Conference Paper

Exploring Effective Inter-Encoder Semantic Interaction for Document-Level Relation Extraction

  • Liang Zhang
  • Zijun Min
  • Jinsong Su
  • Pei Yu
  • Ante Wang
  • Yidong Chen

In document-level relation extraction (RE), the models are required to correctly predict implicit relations in documents via relational reasoning. To this end, many graph-based methods have been proposed for this task. Despite their success, these methods still suffer from several drawbacks: 1) their interaction between document encoder and graph encoder is usually unidirectional and insufficient; 2) their graph encoders often fail to capture the global context of nodes in document graph. In this paper, we propose a document-level RE model with a Graph-Transformer Network (GTN). The GTN includes two core sublayers: 1) the graph-attention sublayer that simultaneously models global and local contexts of nodes in the document graph; 2) the cross-attention sublayer, enabling GTN to capture the non-entity clue information from the document encoder. Furthermore, we introduce two auxiliary training tasks to enhance the bidirectional semantic interaction between the document encoder and GTN: 1) the graph node reconstruction that can effectively train our cross-attention sublayer to enhance the semantic transition from the document encoder to GTN; 2) the structure-aware adversarial knowledge distillation, by which we can effectively transfer the structural information of GTN to the document encoder. Experimental results on four benchmark datasets prove the effectiveness of our model. Our source code is available at https: //github. com/DeepLearnXMU/DocRE-BSI.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Exploring Self-Distillation Based Relational Reasoning Training for Document-Level Relation Extraction

  • Liang Zhang
  • Jinsong Su
  • Zijun Min
  • Zhongjian Miao
  • Qingguo Hu
  • Biao Fu
  • Xiaodong Shi
  • Yidong Chen

Document-level relation extraction (RE) aims to extract relational triples from a document. One of its primary challenges is to predict implicit relations between entities, which are not explicitly expressed in the document but can usually be extracted through relational reasoning. Previous methods mainly implicitly model relational reasoning through the interaction among entities or entity pairs. However, they suffer from two deficiencies: 1) they often consider only one reasoning pattern, of which coverage on relational triples is limited; 2) they do not explicitly model the process of relational reasoning. In this paper, to deal with the first problem, we propose a document-level RE model with a reasoning module that contains a core unit, the reasoning multi-head self-attention unit. This unit is a variant of the conventional multi-head self-attention and utilizes four attention heads to model four common reasoning patterns, respectively, which can cover more relational triples than previous methods. Then, to address the second issue, we propose a self-distillation training framework, which contains two branches sharing parameters. In the first branch, we first randomly mask some entity pair feature vectors in the document, and then train our reasoning module to infer their relations by exploiting the feature information of other related entity pairs. By doing so, we can explicitly model the process of relational reasoning. However, because the additional masking operation is not used during testing, it causes an input gap between training and testing scenarios, which would hurt the model performance. To reduce this gap, we perform conventional supervised training without masking operation in the second branch and utilize Kullback-Leibler divergence loss to minimize the difference between the predictions of the two branches. Finally, we conduct comprehensive experiments on three benchmark datasets, of which experimental results demonstrate that our model consistently outperforms all competitive baselines. Our source code is available at https://github.com/DeepLearnXMU/DocRE-SD

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimation

  • Zihao Yue
  • Anwen Hu
  • Liang Zhang
  • Qin Jin

Image captioning aims to describe visual content in natural language. As 'a picture is worth a thousand words', there could be various correct descriptions for an image. However, with maximum likelihood estimation as the training objective, the captioning model is penalized whenever its prediction mismatches with the label. For instance, when the model predicts a word expressing richer semantics than the label, it will be penalized and optimized to prefer more concise expressions, referred to as conciseness optimization. In contrast, predictions that are more concise than labels lead to richness optimization. Such conflicting optimization directions could eventually result in the model generating general descriptions. In this work, we introduce Semipermeable MaxImum Likelihood Estimation (SMILE), which allows richness optimization while blocking conciseness optimization, thus encouraging the model to generate longer captions with more details. Extensive experiments on two mainstream image captioning datasets MSCOCO and Flickr30K demonstrate that SMILE significantly enhances the descriptiveness of generated captions. We further provide in-depth investigations to facilitate a better understanding of how SMILE works.

PDF Details

AAMAS Conference 2023 Conference Paper

Model-Based Reinforcement Learning for Auto-bidding in Display Advertising

  • Shuang Chen
  • Qisen Xu
  • Liang Zhang
  • Yongbo Jin
  • Wenhao Li
  • Linjian Mo

Real-time bidding (RTB) achieves outstanding success in online display advertising, which has become one of the most influential businesses. Given historical ad impressions under the second price auction mechanism, the advertiser’s optimal bidding strategy is determined by the core parameter corresponding to the optimal solution of a constrained optimization problem. However, the sequentially arrived impressions in online display advertising make it highly non-trivial to obtain the optimal core parameter in advance without knowing the complete impression set. For this reason, recent methods have generally transformed the core parameter determination problem into a sequential parameter adjustment problem and solved it using reinforcement learning (RL). This paper proposes a simple and effective Model-Based Automatic Bidding algorithm, MBAB, which explicitly models the uncertainty of the dynamic auction environment and then uses the dynamic programming algorithm to obtain the current optimal adjustment of the core parameter. MBAB can avoid burdensome simulated environment construction and is more suitable for production deployment without the thorny sim-to-real issue than model-free methods. Furthermore, MBAB uses the optimal bidding formula to carry out coarse-grained modeling of the online market environment to alleviate the scalability problem caused by fine-grained environment modeling of previous model-based methods. In order to accurately describe the impression distribution and non-stationarity of the online market environment, we introduce the probabilistic modeling method and propose a novel monotonicity constraint to regulate the model output. Numerical experiments show that the proposed MBAB substantially outperforms existing baselines on various constrained RTB tasks in the production environment.

PDF

AAAI Conference 2023 Conference Paper

MPMQA: Multimodal Question Answering on Product Manuals

  • Liang Zhang
  • Anwen Hu
  • Jing Zhang
  • Shuo Hu
  • Qin Jin

Visual contents, such as illustrations and images, play a big role in product manual understanding. Existing Product Manual Question Answering (PMQA) datasets tend to ignore visual contents and only retain textual parts. In this work, to emphasize the importance of multimodal contents, we propose a Multimodal Product Manual Question Answering (MPMQA) task. For each question, MPMQA requires the model not only to process multimodal contents but also to provide multimodal answers. To support MPMQA, a large-scale dataset PM209 is constructed with human annotations, which contains 209 product manuals from 27 well-known consumer electronic brands. Human annotations include 6 types of semantic regions for manual contents and 22,021 pairs of question and answer. Especially, each answer consists of a textual sentence and related visual regions from manuals. Taking into account the length of product manuals and the fact that a question is always related to a small number of pages, MPMQA can be naturally split into two subtasks: retrieving most related pages and then generating multimodal answers. We further propose a unified model that can perform these two subtasks all together and achieve comparable performance with multiple task-specific models. The PM209 dataset is available at https://github.com/AIM3-RUC/MPMQA.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

Optimal Guarantees for Algorithmic Reproducibility and Gradient Complexity in Convex Optimization

  • Liang Zhang
  • Junchi YANG
  • Amin Karbasi
  • Niao He

Algorithmic reproducibility measures the deviation in outputs of machine learning algorithms upon minor changes in the training process. Previous work suggests that first-order methods would need to trade-off convergence rate (gradient complexity) for better reproducibility. In this work, we challenge this perception and demonstrate that both optimal reproducibility and near-optimal convergence guarantees can be achieved for smooth convex minimization and smooth convex-concave minimax problems under various error-prone oracle settings. Particularly, given the inexact initialization oracle, our regularization-based algorithms achieve the best of both worlds -- optimal reproducibility and near-optimal gradient complexity -- for minimization and minimax optimization. With the inexact gradient oracle, the near-optimal guarantees also hold for minimax optimization. Additionally, with the stochastic gradient oracle, we show that stochastic gradient descent ascent is optimal in terms of both reproducibility and gradient complexity. We believe our results contribute to an enhanced understanding of the reproducibility-convergence trade-off in the context of convex optimization.

PDF Details

NeurIPS Conference 2023 Conference Paper

UE4-NeRF:Neural Radiance Field for Real-Time Rendering of Large-Scale Scene

  • Jiaming Gu
  • Minchao Jiang
  • Hongsheng Li
  • Xiaoyuan Lu
  • Guangming Zhu
  • Syed Afaq Ali Shah
  • Liang Zhang
  • Mohammed Bennamoun

Neural Radiance Fields (NeRF) is a novel implicit 3D reconstruction method that shows immense potential and has been gaining increasing attention. It enables the reconstruction of 3D scenes solely from a set of photographs. However, its real-time rendering capability, especially for interactive real-time rendering of large-scale scenes, still has significant limitations. To address these challenges, in this paper, we propose a novel neural rendering system called UE4-NeRF, specifically designed for real-time rendering of large-scale scenes. We partitioned each large scene into different sub-NeRFs. In order to represent the partitioned independent scene, we initialize polygonal meshes by constructing multiple regular octahedra within the scene and the vertices of the polygonal faces are continuously optimized during the training process. Drawing inspiration from Level of Detail (LOD) techniques, we trained meshes of varying levels of detail for different observation levels. Our approach combines with the rasterization pipeline in Unreal Engine 4 (UE4), achieving real-time rendering of large-scale scenes at 4K resolution with a frame rate of up to 43 FPS. Rendering within UE4 also facilitates scene editing in subsequent stages. Furthermore, through experiments, we have demonstrated that our method achieves rendering quality comparable to state-of-the-art approaches. Project page: https: //jamchaos. github. io/UE4-NeRF/.

PDF Details

NeurIPS Conference 2022 Conference Paper

Bring Your Own Algorithm for Optimal Differentially Private Stochastic Minimax Optimization

  • Liang Zhang
  • Kiran K. Thekumparampil
  • Sewoong Oh
  • Niao He

We study differentially private (DP) algorithms for smooth stochastic minimax optimization, with stochastic minimization as a byproduct. The holy grail of these settings is to guarantee the optimal trade-off between the privacy and the excess population loss, using an algorithm with a linear time-complexity in the number of training samples. We provide a general framework for solving differentially private stochastic minimax optimization (DP-SMO) problems, which enables the practitioners to bring their own base optimization algorithm and use it as a black-box to obtain the near-optimal privacy-loss trade-off. Our framework is inspired from the recently proposed Phased-ERM method [22] for nonsmooth differentially private stochastic convex optimization (DP-SCO), which exploits the stability of the empirical risk minimization (ERM) for the privacy guarantee. The flexibility of our approach enables us to sidestep the requirement that the base algorithm needs to have bounded sensitivity, and allows the use of sophisticated variance-reduced accelerated methods to achieve near-linear time-complexity. To the best of our knowledge, these are the first near-linear time algorithms with near-optimal guarantees on the population duality gap for smooth DP-SMO, when the objective is (strongly-)convex--(strongly-)concave. Additionally, based on our flexible framework, we enrich the family of near-linear time algorithms for smooth DP-SCO with the near-optimal privacy-loss trade-off.

PDF Details

IJCAI Conference 2022 Conference Paper

FOGS: First-Order Gradient Supervision with Learning-based Graph for Traffic Flow Forecasting

  • Xuan Rao
  • Hao Wang
  • Liang Zhang
  • Jing Li
  • Shuo Shang
  • Peng Han

Traffic flow forecasting plays a vital role in the transportation domain. Existing studies usually manually construct correlation graphs and design sophisticated models for learning spatial and temporal features to predict future traffic states. However, manually constructed correlation graphs cannot accurately extract the complex patterns hidden in the traffic data. In addition, it is challenging for the prediction model to fit traffic data due to its irregularly-shaped distribution. To solve the above-mentioned problems, in this paper, we propose a novel learning-based method to learn a spatial-temporal correlation graph, which could make good use of the traffic flow data. Moreover, we propose First-Order Gradient Supervision (FOGS), a novel method for traffic flow forecasting. FOGS utilizes first-order gradients, rather than specific flows, to train prediction model, which effectively avoids the problem of fitting irregularly-shaped distributions. Comprehensive numerical evaluations on four real-world datasets reveal that the proposed methods achieve state-of-the-art performance and significantly outperform the benchmarks.

PDF Details DOI

NeurIPS Conference 2022 Conference Paper

Multi-Lingual Acquisition on Multimodal Pre-training for Cross-modal Retrieval

  • Liang Zhang
  • Anwen Hu
  • Qin Jin

Vision and diverse languages are important information sources in our living world. A model that understands multi-modalities and multi-languages can be applied to a wider range of real-life scenarios. To build such a multimodal and multilingual model, existing works try to ensemble vision-language data from multiple languages in pre-training. However, due to the large number of languages, these works often require huge computing resources and cannot be flexibly extended to new languages. In this work, we propose a MultiLingual Acquisition (MLA) framework that can easily empower a monolingual Vision-Language Pre-training (VLP) model with multilingual capability. Specifically, we design a lightweight language acquisition encoder based on state-of-the-art monolingual VLP models. We further propose a two-stage training strategy to optimize the language acquisition encoder, namely the Native Language Transfer stage and the Language Exposure stage. With much less multilingual training data and computing resources, our model achieves state-of-the-art performance on multilingual image-text and video-text retrieval benchmarks.

PDF Details

JBHI Journal 2021 Journal Article

Attention-Guided Deep Neural Network With Multi-Scale Feature Fusion for Liver Vessel Segmentation

  • Qingsen Yan
  • Bo Wang
  • Wei Zhang
  • Chuan Luo
  • Wei Xu
  • Zhengqing Xu
  • Yanning Zhang
  • Qinfeng Shi

Liver vessel segmentation is fast becoming a key instrument in the diagnosis and surgical planning of liver diseases. In clinical practice, liver vessels are normally manual annotated by clinicians on each slice of CT images, which is extremely laborious. Several deep learning methods exist for liver vessel segmentation, however, promoting the performance of segmentation remains a major challenge due to the large variations and complex structure of liver vessels. Previous methods mainly using existing UNet architecture, but not all features of the encoder are useful for segmentation and some even cause interferences. To overcome this problem, we propose a novel deep neural network for liver vessel segmentation, called LVSNet, which employs special designs to obtain the accurate structure of the liver vessel. Specifically, we design Attention-Guided Concatenation (AGC) module to adaptively select the useful context features from low-level features guided by high-level features. The proposed AGC module focuses on capturing rich complemented information to obtain more details. In addition, we introduce an innovative multi-scale fusion block by constructing hierarchical residual-like connections within one single residual block, which is of great importance for effectively linking the local blood vessel fragments together. Furthermore, we construct a new dataset containing 40 thin thickness cases (0. 625 mm) which consist of CT volumes and annotated vessels. To evaluate the effectiveness of the method with minor vessels, we also propose an automatic stratification method to split major and minor liver vessels. Extensive experimental results demonstrate that the proposed LVSNet outperforms previous methods on liver vessel segmentation datasets. Additionally, we conduct a series of ablation studies that comprehensively support the superiority of the underlying concepts.

Details DOI

YNIMG Journal 2021 Journal Article

The coupling of BOLD signal variability and degree centrality underlies cognitive functions and psychiatric diseases

  • Jintao Sheng
  • Liang Zhang
  • Junjiao Feng
  • Jing Liu
  • Anqi Li
  • Wei Chen
  • Yuedi Shen
  • Jinhui Wang

Brain signal variability has been consistently linked to functional integration; however, whether this coupling is associated with cognitive functions and/or psychiatric diseases has not been clarified. Using multiple multimodality datasets, including resting-state functional magnetic resonance imaging (rsfMRI) data from the Human Connectome Project (HCP: N = 927) and a Beijing sample (N = 416) and cerebral blood flow (CBF) and rsfMRI data from a Hangzhou sample (N = 29), we found that, compared with the existing variability measure (i. e. , SDBOLD), the mean-scaled (standardized) fractional standard deviation of the BOLD signal (mfSDBOLD) maintained very high test-retest reliability, showed greater cross-site reliability and was less affected by head motion. We also found strong reproducible couplings between the mfSDBOLD and functional integration measured by the degree centrality (DC), both cross-voxel and cross-subject, which were robust to scanning and preprocessing parameters. Moreover, both mfSDBOLD and DC were correlated with CBF, suggesting a common physiological basis for both measures. Critically, the degree of coupling between mfSDBOLD and long-range DC was positively correlated with individuals’ cognitive total composite scores. Brain regions with greater mismatches between mfSDBOLD and long-range DC were more vulnerable to brain diseases. Our results suggest that BOLD signal variability could serve as a meaningful index of local function that underlies functional integration in the human brain and that a strong coupling between BOLD signal variability and functional integration may serve as a hallmark of balanced brain networks that are associated with optimal brain functions.

Details DOI

ICRA Conference 2020 Conference Paper

A Connectivity-Prediction Algorithm and its Application in Active Cooperative Localization for Multi-Robot Systems

  • Liang Zhang
  • Zexu Zhang
  • Roland Siegwart
  • Jen Jen Chung

This paper presents a method for predicting the probability of future connectivity between mobile robots with range-limited communication. In particular, we focus on its application to active motion planning for cooperative localization (CL). The probability of connection is modeled by the distribution of quadratic forms in random normal variables and is computed by the infinite power series expansion theorem. A finite-term approximation is made to realize the computational feasibility and three more modifications are designed to handle the adverse impacts introduced by the omission of the higher order series terms. On the basis of this algorithm, an active and CL problem with leader-follower architecture is then reformulated into a Markov Decision Process (MDP) with a one-step planning horizon, and the optimal motion strategy is generated by minimizing the expected cost of the MDP. Extensive simulations and comparisons are presented to show the effectiveness and efficiency of both the proposed prediction algorithm and the MDP model.

Details

IJCAI Conference 2020 Conference Paper

An Attention-based Model for Conversion Rate Prediction with Delayed Feedback via Post-click Calibration

  • Yumin Su
  • Liang Zhang
  • Quanyu Dai
  • Bo Zhang
  • Jinyao Yan
  • Dan Wang
  • Yongjun Bao
  • Sulong Xu

Conversion rate (CVR) prediction is becoming increasingly important in the multi-billion dollar online display advertising industry. It has two major challenges: firstly, the scarce user history data is very complicated and non-linear; secondly, the time delay between the clicks and the corresponding conversions can be very large, e. g. , ranging from seconds to weeks. Existing models usually suffer from such scarce and delayed conversion behaviors. In this paper, we propose a novel deep learning framework to tackle the two challenges. Specifically, we extract the pre-trained embedding from impressions/clicks to assist in conversion models and propose an inner/self-attention mechanism to capture the fine-grained personalized product purchase interests from the sequential click data. Besides, to overcome the time-delay issue, we calibrate the delay model by learning dynamic hazard function with the abundant post-click data more in line with the real distribution. Empirical experiments with real-world user behavior data prove the effectiveness of the proposed method.

PDF Details DOI

NeurIPS Conference 2020 Conference Paper

How does Weight Correlation Affect Generalisation Ability of Deep Neural Networks?

  • Gaojie Jin
  • Xinping Yi
  • Liang Zhang
  • Lijun Zhang
  • Sven Schewe
  • Xiaowei Huang

This paper studies the novel concept of weight correlation in deep neural networks and discusses its impact on the networks' generalisation ability. For fully-connected layers, the weight correlation is defined as the average cosine similarity between weight vectors of neurons, and for convolutional layers, the weight correlation is defined as the cosine similarity between filter matrices. Theoretically, we show that, weight correlation can, and should, be incorporated into the PAC Bayesian framework for the generalisation of neural networks, and the resulting generalisation bound is monotonic with respect to the weight correlation. We formulate a new complexity measure, which lifts the PAC Bayes measure with weight correlation, and experimentally confirm that it is able to rank the generalisation errors of a set of networks more precisely than existing measures. More importantly, we develop a new regulariser for training, and provide extensive experiments that show that the generalisation error can be greatly reduced with our novel approach.

PDF Details

IJCAI Conference 2019 Conference Paper

GCN-LASE: Towards Adequately Incorporating Link Attributes in Graph Convolutional Networks

  • Ziyao Li
  • Liang Zhang
  • Guojie Song

Graph Convolutional Networks (GCNs) have proved to be a most powerful architecture in aggregating local neighborhood information for individual graph nodes. Low-rank proximities and node features are successfully leveraged in existing GCNs, however, attributes that graph links may carry are commonly ignored, as almost all of these models simplify graph links into binary or scalar values describing node connectedness. In our paper instead, links are reverted to hypostatic relationships between entities with descriptional attributes. We propose GCN-LASE (GCN with Link Attributes and Sampling Estimation), a novel GCN model taking both node and link attributes as inputs. To adequately captures the interactions between link and node attributes, their tensor product is used as neighbor features, based on which we define several graph kernels and further develop according architectures for LASE. Besides, to accelerate the training process, the sum of features in entire neighborhoods are estimated through Monte Carlo method, with novel sampling strategies designed for LASE to minimize the estimation variance. Our experiments show that LASE outperforms strong baselines over various graph datasets, and further experiments corroborate the informativeness of link attributes and our model's ability of adequately leveraging them.

PDF Details

AAAI Conference 2019 Conference Paper

SepNE: Bringing Separability to Network Embedding

  • Ziyao Li
  • Liang Zhang
  • Guojie Song

Many successful methods have been proposed for learning low dimensional representations on large-scale networks, while almost all existing methods are designed in inseparable processes, learning embeddings for entire networks even when only a small proportion of nodes are of interest. This leads to great inconvenience, especially on super-large or dynamic networks, where these methods become almost impossible to implement. In this paper, we formalize the problem of separated matrix factorization, based on which we elaborate a novel objective function that preserves both local and global information. We further propose SepNE, a simple and flexible network embedding algorithm which independently learns representations for different subsets of nodes in separated processes. By implementing separability, our algorithm reduces the redundant efforts to embed irrelevant nodes, yielding scalability to super-large networks, automatic implementation in distributed learning and further adaptations. We demonstrate the effectiveness of this approach on several real-world networks with different scales and subjects. With comparable accuracy, our approach significantly outperforms state-of-the-art baselines in running times on large networks.

PDF Details

NeurIPS Conference 2018 Conference Paper

Attention in Convolutional LSTM for Gesture Recognition

  • Liang Zhang
  • Guangming Zhu
  • Lin Mei
  • Peiyi Shen
  • Syed Afaq Ali Shah
  • Mohammed Bennamoun

Convolutional long short-term memory (LSTM) networks have been widely used for action/gesture recognition, and different attention mechanisms have also been embedded into the LSTM or the convolutional LSTM (ConvLSTM) networks. Based on the previous gesture recognition architectures which combine the three-dimensional convolution neural network (3DCNN) and ConvLSTM, this paper explores the effects of attention mechanism in ConvLSTM. Several variants of ConvLSTM are evaluated: (a) Removing the convolutional structures of the three gates in ConvLSTM, (b) Applying the attention mechanism on the input of ConvLSTM, (c) Reconstructing the input and (d) output gates respectively with the modified channel-wise attention mechanism. The evaluation results demonstrate that the spatial convolutions in the three gates scarcely contribute to the spatiotemporal feature fusion, and the attention mechanisms embedded into the input and output gates cannot improve the feature fusion. In other words, ConvLSTM mainly contributes to the temporal fusion along with the recurrent steps to learn the long-term spatiotemporal features, when taking as input the spatial or spatiotemporal features. On this basis, a new variant of LSTM is derived, in which the convolutional structures are only embedded into the input-to-state transition of LSTM. The code of the LSTM variants is publicly available.

PDF Details

IJCAI Conference 2017 Conference Paper

Adaptively Unified Semi-supervised Learning for Cross-Modal Retrieval

  • Liang Zhang
  • Bingpeng Ma
  • Jianfeng He
  • Guorong Li
  • Qingming Huang
  • Qi Tian

Motivated by the fact that both relevancy of class labels and unlabeled data can help to strengthen multi-modal correlation, this paper proposes a novel method for cross-modal retrieval. To make each sample moving to the direction of its relevant label while far away from that of its irrelevant ones, a novel dragging technique is fused into a unified linear regression model. By this way, not only the relation between embedded features and relevant class labels but also the relation between embedded features and irrelevant class labels can be exploited. Moreover, considering that some unlabeled data contain specific semantic information, a weighted regression model is designed to adaptively enlarge their contribution while weaken that of the unlabeled data with non-specific semantic information. Hence, unlabeled data can supply semantic information to enhance discriminant ability of classifier. Finally, we integrate the constraints into a joint minimization formulation and develop an efficient optimization algorithm to learn a discriminative common subspace for different modalities. Experimental results on Wiki, Pascal and NUS-WIDE datasets show that the proposed method outperforms the state-of-the-art methods even when we set 20% samples without class labels.

PDF Details

ICRA Conference 2013 Conference Paper

Geometric parameter identification for spherical actuator calibration based on torque formula

  • Liang Zhang
  • Weihai Chen
  • Jingmeng Liu
  • Xingming Wu

This paper presents a geometric calibration approach of a permanent magnet (PM) spherical actuator to improve its positioning accuracy. The proposed actuator consists of a ball-shaped rotor with multiple PM poles and a spherical-shell-shaped stator with circumferential air-core coils. Due to manufacturing and assembly restrictions, the actual geometric parameters of the spherical actuator differ from their nominal values. Hence, the identification of such errors is significant for high accuracy motion control. The calibration model is formulated based on the differential form of torque equation. To identify the position vector errors in the magnetization axes of PM poles and coils axes, an iterative least-squares algorithm is employed. The proposed calibration method can also be applied to other PM spherical actuators. To verify the robustness and effectiveness of the proposed calibration algorithm, simulations are conducted on the spherical actuator. The results have shown that the positioning accuracy of the spherical actuator is greatly improved after calibration.

Details

AAAI Conference 2011 Conference Paper

Online Updating the Generalized Inverse of Centered Matrices

  • Qing Wang
  • Liang Zhang

In this paper, we present the exact online updating formulae for the generalized inverse of centered matrices. The computational cost is O(mn) for matrices of size m × n. Experimental results validate the proposed method’s accuracy and efficiency.

PDF Details

IROS Conference 2011 Conference Paper

Trajectory planning and current control optimization of three degree-of-freedom spherical actuator

  • Liang Zhang
  • Weihai Chen
  • Liang Yan
  • Jingmeng Liu

The study in this paper covers torque modeling, trajectory planning and optimization control of current input of spherical actuators, in which the latter two are the major contributions. Trajectory planning is an effective way to improve the smoothness and stability of rotor motions. A novel three-dimensional (3D) orientation representation method based on manifold of S 2 is proposed to facilitate the trajectory planning of rotor. Current redundancy of spherical actuator is analyzed in detail, and optimization algorithm of current input is developed to improve the power efficiency and the fault tolerance capability of system. Simulation is then carried out to validate the proposed method and algorithm in this study. The simulation results indicate that by using the trajectory planning, the given torque values and the torque model, optimal current could be obtained to drive the rotor to achieve desired motions.

Details

TAAS Journal 2010 Journal Article

A self-organization mechanism based on cross-entropy method for P2P-like applications

  • Gang Chen
  • Abdolhossein Sarrafzadeh
  • Chor Ping Low
  • Liang Zhang

P2P-like applications are quickly gaining popularity in the Internet. Such applications are commonly modeled as graphs with nodes and edges. Usually nodes represent running processes that exchange information with each other through communication channels as represented by the edges. They often need to autonomously determine their suitable working mode or local status for the purpose of improving performance, reducing operation cost, or achieving system-level design goals. In order to achieve this objective, the concept of status configuration is introduced in this article and a mathematical correspondence is further established between status configuration and an optimization index ( OI ), which serves as a unified abstraction of any system design goals. Guided by this correspondence and inspired by the cross-entropy algorithm, a cross-entropy-driven self-organization mechanism (CESM) is proposed in this article. CESM exhibits the self-organization property since desirable status configurations that lead to high OI values will quickly emerge from purely localized interactions. Both theoretical and experimental analysis have been performed. The results strongly indicate that CESM is a simple yet effective technique which is potentially suitable for many P2P-like applications.

Details DOI

NeurIPS Conference 2008 Conference Paper

Fast Computation of Posterior Mode in Multi-Level Hierarchical Models

  • Liang Zhang
  • Deepak Agarwal

Multi-level hierarchical models provide an attractive framework for incorporating correlations induced in a response variable organized in a hierarchy. Model fitting is challenging, especially for hierarchies with large number of nodes. We provide a novel algorithm based on a multi-scale Kalman filter that is both scalable and easy to implement. For non-Gaussian responses, quadratic approximation to the log-likelihood results in biased estimates. We suggest a bootstrap strategy to correct such biases. Our method is illustrated through simulation studies and analyses of real world data sets in health care and online advertising.

PDF Details

TCS Journal 2003 Journal Article

Completion of codes with finite bi-decoding delays

  • Liang Zhang
  • K.P. Shum
  • Shou-Li Peng

Let A∗ be a free monoid generated by a set A and let X⊆A∗ be a code with property P. The embedding of X into a complete code Y⊆A∗ with the same property P is called the completion of X. The method of completion of rational bifix codes and codes with finite decoding delays have been investigated by a number of authors. In this paper, we provide a general method of construction for completing the codes with finite bi-decoding delays. As a consequence, the completion method of rational bifix codes and codes with finite decoding delays is extended and applied to codes with finite bi-decoding delays.

Details DOI

TCS Journal 1993 Journal Article

Decompositions of recognizable strong maximal codes

  • Liang Zhang
  • Weide Qiu

In this paper, we study the decompositions of recognizable strong maximal codes and obtain the following results: Every recognizable strong maximal code is a composition of a finite number of indecomposable (in the sense of strong codes) recognizable strong maximal codes. In particular, every solvable strong code is a composition of a finite number of indecomposable (in the sense of general codes) strong maximal codes and, for the latter, a structure formula is given.

Details DOI
v2026.09.13