Arrow Research search

Author name cluster

Cheng Zhang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

60 papers
2 author rows

Possible papers

60

YNIMG Journal 2026 Journal Article

Anatomically mediated variability of hippocampal electric fields in temporal interference stimulation: A predictive modeling approach

  • Weiyu Meng
  • Ziyan Zhu
  • Zhen Wu
  • Shuxiang Zhu
  • Cheng Zhang
  • Changzhe Wu
  • Xiaolin Huo
  • Hua Lin

BACKGROUND: Temporal interference stimulation (TIS) enables noninvasive targeting of deep brain regions. However, individual variability in the induced electric field (EF), particularly in terms of modulation depth (MD) and directional alignment, remains poorly understood. METHODS: We performed MRI-guided finite element modeling to simulate TIS-induced EF in the left hippocampus (LHippo), quantifying two metrics: MD and the angle between EF vectors and the hippocampal longitudinal axis. First, we evaluated these metrics across individuals stratified by age, sex, and cognitive status. To identify anatomical contributors to MD variability, we applied the least absolute shrinkage and selection operator (LASSO) regression for feature selection, followed by stepwise regression to construct a predictive model. We then conducted mediation analysis to determine whether the selected anatomical features accounted for the effects of age, sex, and cognitive status on MD. RESULTS: MD varied significantly with age, sex, and cognitive status, whereas the directional angle remained relatively stable under the fixed electrode configuration. The final predictive model, incorporating eight anatomical features (e.g., skull thickness, CSF-to-head ratio), demonstrated strong generalizability in two independent validation cohorts. Mediation analysis confirmed that the model-predicted MD significantly mediated the effects of age, sex, and cognitive status. CONCLUSIONS: These findings highlight the critical role of anatomical variability in shaping individual TIS outcomes and demonstrate that personalized MD could be efficiently estimated from clinically measurable features without subject-specific simulations. This anatomically informed framework supports scalable and interpretable strategies for precision-targeted brain stimulation.

Details DOI

AAAI Conference 2026 Conference Paper

Cook and Clean Together: Teaching Embodied Agents for Parallel Task Execution

  • Dingkang Liang
  • Cheng Zhang
  • Xiaopeng Xu
  • Jianzhong Ju
  • Zhenbo Luo
  • Xiang Bai

Task scheduling has become increasingly critical for embodied AI, where agents need to follow natural language instructions and execute actions efficiently in 3D physical worlds. Existing datasets for task planning in 3D environments often simplify the problem, lacking operations research knowledge for task scheduling and 3D grounding for real-world applications. In this work, we propose Operations Research Knowledge-based 3D Grounded Task Scheduling (OKS3D), a new task that requires synerization of language understanding, 3D grounding, and efficiency optimization for embodied agents. OKS3D reflects real-world demands by requiring agents to generate efficient, step-by-step schedules that are grounded in 3D space. To facilitate research on OKS3D, we construct a large-scale dataset called OKS3D-60K, comprising 60K tasks across 4K real-world scenes. Furthermore, we propose GRANT, an embodied multi-modal large language model equipped with a simple yet effective scheduling token mechanism to generate efficient task schedules and grounded actions. Extensive experiments on the OKS3D-60K dataset validate the effectiveness of GRANT across language understanding, 3D grounding, and scheduling efficiency.

PDF Details DOI

AAAI Conference 2026 Conference Paper

MSCFL: Model Structure-Aware Clustered Federated Learning for System Heterogeneity and Data Drift

  • Yang Xu
  • Xiaowei Wu
  • Zifeng Xu
  • Cheng Zhang
  • Ju Ren
  • Yaoxue Zhang

Federated Learning (FL) faces significant challenges arising from both data and system heterogeneity. While Clustered Federated Learning (CFL) mitigates data heterogeneity by grouping clients with similar data distributions, it remains vulnerable to system heterogeneity, which can slow convergence due to performance disparities among clients. Moreover, data drift may degrade clustering accuracy and training efficiency over time. In this work, we propose a Model Structure-aware Clustered Federated Learning (MSCFL) framework that simultaneously addresses the issues of data heterogeneity, system heterogeneity, and data drift. MSCFL incorporates model pruning (MP) into the CFL framework to enhance training efficiency under system heterogeneity. To enable this integration, we address the key challenge of performing effective clustering based on heterogeneous, pruned local models with varying structures. To this end, we design a model structure-based similarity computation algorithm to integrate CFL with MP. To effectively address data drift, we propose a dynamic cluster migration strategy that efficiently monitors model structures via Hamming Distance and triggers re-clustering only when necessary. Extensive experimental results show that MSCFL improves the accuracy and convergence speed of cluster models, outperforming traditional CFL in various settings.

PDF Details DOI

AAAI Conference 2026 Conference Paper

PanFlow: Decoupled Motion Control for Panoramic Video Generation

  • Cheng Zhang
  • Hanwen Liang
  • Donny Y. Chen
  • Qianyi Wu
  • Konstantinos N. Plataniotis
  • Camilo Cruz Gambardella
  • Jianfei Cai

Panoramic video generation has attracted growing attention due to its applications in virtual reality and immersive media. However, existing methods lack explicit motion control and struggle to generate scenes with large and complex motions. We propose PanFlow a novel approach that exploits the spherical nature of panoramas to decouple the highly dynamic camera rotation from the input optical flow condition, enabling more precise control over large and dynamic motions. We further introduce a spherical noise warping strategy to promote loop consistency in motion across panorama boundaries. To support effective training, we curate a large-scale, motion-rich panoramic video dataset with frame-level pose and flow annotations. We also showcase the effectiveness of our method in various applications, including motion transfer and video editing. Extensive experiments demonstrate that PanFlow significantly outperforms prior methods in motion fidelity, visual quality, and temporal coherence.

PDF Details DOI

AAAI Conference 2026 Conference Paper

SEMC: Structure-Enhanced Mixture-of-Experts Contrastive Learning for Ultrasound Standard Plane Recognition

  • Qing Cai
  • GuihaoYan
  • Fan Zhang
  • Cheng Zhang
  • Zhi Liu

Ultrasound standard plane recognition is essential for clinical tasks such as disease screening, organ evaluation, and biometric measurement. However, existing methods fail to effectively exploit shallow structural information and struggle to capture fine-grained semantic differences through contrastive samples generated by image augmentations, leading to poor recognition of structural and discriminative details in ultrasound standard planes. To address these issues, we propose Structure-Enhanced Mixture-of-Experts Contrastive Learning (SEMC), a novel framework that combines structure-aware feature fusion with expert-guided contrastive learning. Specifically, we propose a Semantic-Structure Fusion Module (SSFM) to exploit multi-scale structural information and enhance the model's ability to perceive fine-grained structural details by effectively aligning shallow and deep features. Meanwhile, a Mixture-of-Experts Contrastive Recognition Module (MCRM) is designed to perform hierarchical contrastive learning and classification across multi-level features using a mixture-of-experts (MoE) mechanism, further improving class separability and overall recognition performance. More importantly, we also curate a large-scale and meticulously annotated liver ultrasound dataset containing six standard planes. Extensive experimental results on our in-house dataset and two public datasets demonstrate that SEMC outperforms recent state-of-the-art methods across various metrics.

PDF Details DOI

TMLR Journal 2025 Journal Article

Amortized Inference of Causal Models via Conditional Fixed-Point Iterations

  • Divyat Mahajan
  • Jannes Gladrow
  • Agrin Hilmkil
  • Cheng Zhang
  • Meyer Scetbon

Structural Causal Models (SCMs) offer a principled framework to reason about interventions and support out-of-distribution generalization, which are key goals in scientific discovery. However, the task of learning SCMs from observed data poses formidable challenges, and often requires training a separate model for each dataset. In this work, we propose an amortized inference framework that trains a single model to predict the causal mechanisms of SCMs conditioned on their observational data and causal graph. We first use a transformer-based architecture for amortized learning of dataset embeddings, and then extend the Fixed-Point Approach (FiP) to infer the causal mechanisms conditionally on their dataset embeddings. As a byproduct, our method can generate observational and interventional data from novel SCMs at inference time, without updating parameters. Empirical results show that our amortized procedure performs on par with baselines trained specifically for each dataset on both in and out-of-distribution problems, and also outperforms them in scare data regimes.

PDF Details

ICML Conference 2025 Conference Paper

Continuous Semi-Implicit Models

  • Longlin Yu
  • Jiajun Zha
  • Tong Yang
  • Tianyu Xie 0001
  • Xiangyu Zhang
  • S. -H. Gary Chan
  • Cheng Zhang

Semi-implicit distributions have shown great promise in variational inference and generative modeling. Hierarchical semi-implicit models, which stack multiple semi-implicit layers, enhance the expressiveness of semi-implicit distributions and can be used to accelerate diffusion models given pretrained score networks. However, their sequential training often suffers from slow convergence. In this paper, we introduce CoSIM, a continuous semi-implicit model that extends hierarchical semi-implicit models into a continuous framework. By incorporating a continuous transition kernel, CoSIM enables efficient, simulation-free training. Furthermore, we show that CoSIM achieves consistency with a carefully designed transition kernel, offering a novel approach for multistep distillation of generative models at the distributional level. Extensive experiments on image generation demonstrate that CoSIM performs on par or better than existing diffusion model acceleration methods, achieving superior performance on FD-DINOv2.

Details

YNICL Journal 2025 Journal Article

Electric field variations across DLPFC targeting methods in TMS therapy for Alzheimer’s disease

  • Nianshuang Wu
  • Yuxuan Shao
  • Zhen Wu
  • Shuxiang Zhu
  • Penghao Wang
  • Ziyan Zhu
  • Cheng Zhang
  • Changzhe Wu

BACKGROUND: The dorsolateral prefrontal cortex (DLPFC) is crucial for cognitive control and a primary target for transcranial magnetic stimulation (TMS) in Alzheimer's disease (AD). However, understanding the distribution of TMS-induced electric field (E-field) across different targeting methods remains limited, as does its relationship to therapeutic outcomes. OBJECTIVE: This study assesses differences in TMS-induced E-field using functional versus anatomical targeting methods for DLPFC stimulation. METHODS: ), were calculated via SimNIBS software for comparisons across stimulation targets. RESULTS: Functional targeting showed greater spatial dispersion compared to anatomical targeting in both groups. Significant E-field differences were observed between the functional target and adjacent anatomical regions when the coil was positioned over the functional target in both groups. Optimal coil orientation exhibited directional specificity: parallel alignment with the LOI E-field produced higher field intensity in the functional target compared to the anatomical target (AD patients: P < 0.001; HCs: P = 0.052), while perpendicular orientation maintained functional stability with reduced anatomical interference (both groups: P < 0.001). And significant variations in E-field ratios were observed across coil orientations. CONCLUSION: This study reveals key E-field disparities across DLPFC targeting approaches and establishes coil orientation optimization as a critical strategy to improve TMS precision, offering actionable insights for developing personalized protocols in AD therapy that may enhance treatment efficacy while minimizing adverse effects.

Details DOI

ICML Conference 2025 Conference Paper

Hardware and Software Platform Inference

  • Cheng Zhang
  • Hanna Foerster
  • Robert D. Mullins
  • Yiren Zhao
  • Ilia Shumailov

It is now a common business practice to buy access to large language model (LLM) inference rather than self-host, because of significant upfront hardware infrastructure and energy costs. However, as a buyer, there is no mechanism to verify the authenticity of the advertised service including the serving hardware platform, e. g. that it is actually being served using an NVIDIA H100. Furthermore, there are reports suggesting that model providers may deliver models that differ slightly from the advertised ones, often to make them run on less expensive hardware. That way, a client pays premium for a capable model access on more expensive hardware, yet ends up being served by a (potentially less capable) cheaper model on cheaper hardware. In this paper we introduce hardware and software platform inference (HSPI) – a method for identifying the underlying GPU architecture and software stack of a (black-box) machine learning model solely based on its input-output behavior. Our method leverages the inherent differences of various GPU architectures and compilers to distinguish between different GPU types and software stacks. By analyzing the numerical patterns in the model’s outputs, we propose a classification framework capable of accurately identifying the GPU used for model inference as well as the underlying software configuration. Our findings demonstrate the feasibility of inferring GPU type from black-box models. We evaluate HSPI against models served on different real hardware and find that in a white-box setting we can distinguish between different GPUs with between 83. 9% and 100% accuracy. Even in a black-box setting we are able to achieve results that are up to three times higher than random guess accuracy.

Details

IS Journal 2025 Journal Article

Machine Learning Approaches for Micromobility User Behavior Analysis

  • Cheng Zhang
  • Bo Du
  • Qiuyun Luan
  • Jun Shen

With widespread adoption globally, micromobility like bikes, e-scooters, and e-bikes has attracted increasing attention due to its ability to complement existing transportation modes and promote sustainable transportation. Understanding micromobility user behaviors in urban areas is essential for improving safety and comfort, as well as for informing infrastructure development and policy. Prior investigations on micromobility user behaviors primarily relied on statistical and kinematic modeling approaches. Although these methods have proven effective in characterizing user behaviors at both macroscopic and microscopic levels, the advent of artificial intelligence (AI)-powered data analytics and behavioral modeling is revolutionizing the field. Recently, advanced machine learning models, such as gradient boosting decision tree, graph convolutional network, and inverse reinforcement learning, has introduced new momentum into micromobility user behavior research. This article explores recent developments, research opportunities, and future directions in this field, leveraging the power of more generic AI approaches.

Details DOI

AAMAS Conference 2025 Conference Paper

Modeling the Collaborative Edge Data Caching Problem via a Dynamic DCOP

  • Ziyang Song
  • Ziyu Chen
  • Jinhui Huang
  • Cheng Zhang
  • Jingyuan He

The Collaborative Edge Data Caching (CEDC) problem poses a significant challenge in Mobile Edge Computing (MEC). It’s a research focus to address the problem from the service providers’ perspective that requires the optimal caching strategy for service providers to maximize their caching revenue, subject to capacity and latency constraints. However, current research primarily focuses on centralized methods, neglecting the distributed and dynamic nature of CEDC. Accordingly, we first propose to use a Dynamic Distributed Constraint Optimization Problem (D-DCOP) to model the problem in a distributed manner, where capacity, latency constraints and caching revenue are dynamically mapped into local hard constraints and constraint utilities between edge servers according to changes in user requests. The proposed model enables each edge server to make its caching strategy through information exchange with neighboring edge servers. We further present a local search framework for CEDC to handle local hard constraints in the model and apply it to two classic local search algorithms, DSA and MGM, along with specific modifications to avoid repetitive computation. We empirically confirm the superiority of our distributed model and algorithms over state-of-the-art centralized solvers for CEDC.

PDF

ICLR Conference 2025 Conference Paper

PhyloVAE: Unsupervised Learning of Phylogenetic Trees via Variational Autoencoders

  • Tianyu Xie 0001
  • Harry Richman
  • Jiansi Gao
  • Frederick A. Matsen IV
  • Cheng Zhang

Learning informative representations of phylogenetic tree structures is essential for analyzing evolutionary relationships. Classical distance-based methods have been widely used to project phylogenetic trees into Euclidean space, but they are often sensitive to the choice of distance metric and may lack sufficient resolution. In this paper, we introduce *phylogenetic variational autoencoders* (PhyloVAEs), an unsupervised learning framework designed for representation learning and generative modeling of tree topologies. Leveraging an efficient encoding mechanism inspired by autoregressive tree topology generation, we develop a deep latent-variable generative model that facilitates fast, parallelized topology generation. PhyloVAE combines this generative model with a collaborative inference model based on learnable topological features, allowing for high-resolution representations of phylogenetic tree samples. Extensive experiments demonstrate PhyloVAE's robust representation learning capabilities and fast generation of phylogenetic tree topologies.

Details

NeurIPS Conference 2025 Conference Paper

Provable Sample-Efficient Transfer Learning Conditional Diffusion Models via Representation Learning

  • Ziheng Cheng
  • Tianyu Xie
  • Shiyue Zhang
  • Cheng Zhang

While conditional diffusion models have achieved remarkable success in various applications, they require abundant data to train from scratch, which is often infeasible in practice. To address this issue, transfer learning has emerged as an essential paradigm in small data regimes. Despite its empirical success, the theoretical underpinnings of transfer learning conditional diffusion models remain unexplored. In this paper, we take the first step towards understanding the sample efficiency of transfer learning conditional diffusion models through the lens of representation learning. Inspired by practical training procedures, we assume that there exists a low-dimensional representation of conditions shared across all tasks. Our analysis shows that with a well-learned representation from source tasks, the sample complexity of target tasks can be reduced substantially. Numerical experiments are also conducted to verify our results.

PDF Details

ICLR Conference 2025 Conference Paper

QERA: an Analytical Framework for Quantization Error Reconstruction

  • Cheng Zhang
  • Jeffrey T. H. Wong
  • Can Xiao
  • George Anthony Constantinides
  • Yiren Zhao

The growing number of parameters and computational demands of large language models (LLMs) present significant challenges for their efficient deployment. Recently, there is an increasing interest in quantizing weights to extremely low precision while offsetting the resulting error with low-rank, high-precision error reconstruction terms. The combination of quantization and low-rank approximation is now popular in both adapter-based, parameter-efficient fine-tuning methods such as LoftQ and low-precision inference techniques including ZeroQuant-V2. Usually, the low-rank terms are calculated via the singular value decomposition (SVD) of the weight quantization error, minimizing the Frobenius and spectral norms of the weight approximation error. Recent methods like LQ-LoRA and LQER introduced hand-crafted heuristics to minimize errors in layer outputs (activations) rather than weights, resulting improved quantization results. However, these heuristic methods lack an analytical solution to guide the design of quantization error reconstruction terms. In this paper, we revisit this problem and formulate an analytical framework, named Quantization Error Reconstruction Analysis (QERA), and offer a closed-form solution to the problem. We show QERA benefits both existing low-precision fine-tuning and inference methods -- QERA achieves a fine-tuned accuracy gain of $\Delta_{\text{acc}}$ = 6.05\% of 2-bit RoBERTa-base on GLUE compared to LoftQ; and obtains $\Delta_{\text{acc}}$ = 2.97\% higher post-training quantization accuracy of 4-bit Llama-3.1-70B on average than ZeroQuant-V2 and $\Delta_{\text{ppl}}$ = $-$ 0.28 lower perplexity on WikiText2 than LQER.

Details

IJCAI Conference 2025 Conference Paper

Rethinking Removal Attack and Fingerprinting Defense for Model Intellectual Property Protection: A Frequency Perspective

  • Cheng Zhang
  • Yang Xu
  • Tingqiao Huang
  • Zixing Zhang

Training deep neural networks is resource-intensive, making it crucial to protect their intellectual property from infringement. However, current model ownership resolution (MOR) methods predominantly address general removal attacks that involve weight modifications, with limited research considering alternative attack perspectives. In this work, we propose a frequency-based model ownership removal attack, grounded in a key observation: modifying a model's high-frequency coefficients does not significantly impact its performance but does alter its weights and decision boundary. This change invalidates the existing MOR methods. We further propose a frequency-based fingerprinting technique as a defense mechanism. By extracting frequency-domain characteristics instead of decision boundary or model weights, our fingerprinting defense effectively against the proposed frequency-based removal attack and demonstrates robustness against existing general removal attacks. The experimental results show that the frequency-based removal attack can easily defeat state-of-the-art white-box watermarking and fingerprinting schemes while preserving model performance, and the proposed defense method is also effective. Our code is released at: https: //github. com/huangtingqiao/RRA-IJCAI25.

PDF Details DOI

TAAS Journal 2025 Journal Article

Tail-Learning: Adaptive Learning Method for Mitigating Tail Latency in Autonomous Edge Systems

  • Cheng Zhang
  • Yinuo Deng
  • Hailiang Zhao
  • Tianlv Chen
  • Shuiguang Deng

In the field of edge computing, the increasing demand for high Quality of Service (QoS), particularly in dynamic multimedia streaming applications (e.g., Augmented Reality/Virtual Reality and online gaming), has prompted the need for effective solutions. Nevertheless, adopting an edge paradigm grounded in distributed computing has exacerbated the issue of tail latency. Given a limited variety of multimedia services supported by edge servers and the dynamic nature of user requests, employing traditional queuing methods to model tail latency in distributed edge computing is challenging, substantially exacerbating head-of-line (HoL) blocking. In response to this challenge, we have developed a learning-based scheduling method to mitigate the overall tail latency, which adaptively selects appropriate edge servers for execution as incoming distributed tasks vary with unknown size. To optimize the utilization of the edge computing paradigm, we leverage the Laplace transform to theoretically derive an upper bound for the response time of edge servers. Subsequently, we integrate this upper bound into reinforcement learning to facilitate tail-learning and enable informed decisions for autonomous distributed scheduling. The experiment results demonstrate the efficiency in reducing tail latency compared to existing methods.

Details DOI

JMLR Journal 2024 Journal Article

A Variational Approach to Bayesian Phylogenetic Inference

  • Cheng Zhang
  • Frederick A. Matsen IV

Bayesian phylogenetic inference is currently done via Markov chain Monte Carlo with simple proposal mechanisms. This hinders exploration efficiency and often requires long runs to deliver accurate posterior estimates. In this paper, we present an alternative approach: a variational framework for Bayesian phylogenetic analysis. We propose combining subsplit Bayesian networks, an expressive graphical model for tree topology distributions, and a structured amortization of the branch lengths over tree topologies for a suitable variational family of distributions. We train the variational approximation via stochastic gradient ascent and adopt gradient estimators for continuous and discrete variational parameters separately to deal with the composite latent space of phylogenetic models. We show that our variational approach provides competitive performance to MCMC, while requiring much fewer (though more costly) iterations due to a more efficient exploration mechanism enabled by variational inference. Experiments on a benchmark of challenging real data Bayesian phylogenetic inference problems demonstrate the effectiveness and efficiency of our methods. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2024. ( edit, beta )

PDF Details

ICRA Conference 2024 Conference Paper

Bio-Inspired Pupal-Mode Actuator with Ultra-Crossing Capability for Soft Robots

  • Zhenxing Wang
  • Xiao He
  • Yuhang Zhang
  • Cheng Zhang
  • Lei Sun
  • Zhidong Wang
  • Shun Xu
  • Hao Liu

Robot-assisted Natural Orifice Translu-minal Endoscopic Surgery (NOTES) represents a paradigm shift in surgical practice, significantly mini-mizing patient morbidity. However, the variability of inner diameter and the inter-luminal crossing within the luminal tracts lead to challenge for effective robotic intervention. Inspired by the motion of the chrysalis during its transformation, we designed an innovative pupal-mode actuator for NOTES robots. Through the manipulation of its internal air chambers, this actuator is capable of replicating wriggle-like movements. Through experimental analysis, we have acquired the constitutive characteristics of this actuator. Subsequently, an innovative gastric endoscopy robot is developed base the actuator and tested in a phantom. The results of the task simulations substantiate that the pupal-mode actuator has the capability to reduce resistance and enhance the safety of the endoscopic intervention.

Details

TMLR Journal 2024 Journal Article

Deep End-to-end Causal Inference

  • Tomas Geffner
  • Javier Antoran
  • Adam Foster
  • Wenbo Gong
  • Chao Ma
  • Emre Kiciman
  • Amit Sharma
  • Angus Lamb

Causal inference is essential for data-driven decision-making across domains such as business engagement, medical treatment, and policy making. However, in practice, causal inference suffers from many limitations including unknown causal graphs, missing data problems, and mixed data types. To tackle those challenges, we develop Deep End-to-end Causal Inference (DECI) framework, a flow based non-linear additive noise model combined with variational inference, which can perform both Bayesian causal discovery and inference. Theoretically, we show that DECI unifies many existing structural equation model (SEM) based causal inference techniques and can recover the ground truth mechanism under standard assumptions. Motivated by the challenges in the real world, we further extend DECI to heterogeneous, mixed-type data with missing values, allowing for both continuous and discrete treatment decisions. Empirically, we conduct extensive experiments (over a thousand) to show the competitive performance of DECI when compared to relevant baselines for both causal discovery and inference with both synthetic and causal machine learning benchmarks across data types and levels of missingness.

PDF Details

AAAI Conference 2024 System Paper

EasyTS: The Express Lane to Long Time Series Forecasting

  • Tiancheng Zhang
  • Shaoyuan Huang
  • Cheng Zhang
  • Xiaofei Wang
  • Wenyu Wang

Responding to the escalating interest in long-term forecasting within the industry, we introduce EasyTS, a comprehensive toolkit engineered to streamline data collection, analysis, and model creation procedures. EasyTS acts as a unified solution, driving progress in long-term time series forecasting. The platform provides effortless access to various time series datasets, including a newly open-sourced multi-scenario dataset in the electricity domain. Integrated visualization and analysis tools help unveil inherent data features and relationships. EasyTS facilitates a user-friendly model validation approach with versatile evaluation criteria. This toolkit allows researchers to compare their models proficiently against renowned benchmarks. With our ongoing commitment to expanding our dataset collection and enhancing toolkit functionalities, we aspire to contribute significantly to the time series forecasting domain. Code is available at this repository: https://github.com/EdgeBigBang/EasyTS.git.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Functional Gradient Flows for Constrained Sampling

  • Shiyue Zhang
  • Longlin Yu
  • Ziheng Cheng
  • Cheng Zhang

Recently, through a unified gradient flow perspective of Markov chain Monte Carlo (MCMC) and variational inference (VI), particle-based variational inference methods (ParVIs) have been proposed that tend to combine the best of both worlds. While typical ParVIs such as Stein Variational Gradient Descent (SVGD) approximate the gradient flow within a reproducing kernel Hilbert space (RKHS), many attempts have been made recently to replace RKHS with more expressive function spaces, such as neural networks. While successful, these methods are mainly designed for sampling from unconstrained domains. In this paper, we offer a general solution to constrained sampling by introducing a boundary condition for the gradient flow which would confine the particles within the specific domain. This allows us to propose a new functional gradient ParVI method for constrained sampling, called constrained functional gradient flow (CFG), with provable continuous-time convergence in total variation (TV). We also present novel numerical strategies to handle the boundary integral term arising from the domain constraints. Our theory and experiments demonstrate the effectiveness of the proposed framework.

PDF Details DOI

ICML Conference 2024 Conference Paper

Kernel Semi-Implicit Variational Inference

  • Ziheng Cheng
  • Longlin Yu
  • Tianyu Xie 0001
  • Shiyue Zhang 0002
  • Cheng Zhang

Semi-implicit variational inference (SIVI) extends traditional variational families with semi-implicit distributions defined in a hierarchical manner. Due to the intractable densities of semi-implicit distributions, classical SIVI often resorts to surrogates of evidence lower bound (ELBO) that would introduce biases for training. A recent advancement in SIVI, named SIVI-SM, utilizes an alternative score matching objective made tractable via a minimax formulation, albeit requiring an additional lower-level optimization. In this paper, we propose kernel SIVI (KSIVI), a variant of SIVI-SM that eliminates the need for the lower-level optimization through kernel tricks. Specifically, we show that when optimizing over a reproducing kernel Hilbert space (RKHS), the lower-level problem has an explicit solution. This way, the upper-level objective becomes the kernel Stein discrepancy (KSD), which is readily computable for stochastic gradient descent due to the hierarchical structure of semi-implicit variational distributions. An upper bound for the variance of the Monte Carlo gradient estimators of the KSD objective is derived, which allows us to establish novel convergence guarantees of KSIVI. We demonstrate the effectiveness and efficiency of KSIVI on both synthetic distributions and a variety of real data Bayesian inference tasks.

Details

ICML Conference 2024 Conference Paper

LQER: Low-Rank Quantization Error Reconstruction for LLMs

  • Cheng Zhang
  • Jianyi Cheng
  • George Anthony Constantinides
  • Yiren Zhao

Post-training quantization of Large Language Models (LLMs) is challenging. In this work, we introduce L ow-rank Q uantization E rror R eduction (LQER), which combines quantization and low-rank approximation to recover the model capability. LQER leverages an activation-induced scale matrix to drive the singular value distribution of quantization error towards a desirable distribution, which enables nearly-lossless W4A8 quantization on various LLMs and downstream tasks without the need for knowledge distillation, grid search, or gradient-based iterative optimization. Unlike existing methods, the computation pattern of LQER eliminates the need for specialized Scatter and Gather processes to collect high-precision weights from irregular memory locations. Our W4A8 LLMs achieve near-lossless performance on six popular downstream tasks, while using $1. 36 \times$ fewer hardware resources than the leading state-of-the-art method. We will open-source our framework at https: //github. com/ChengZhang-98/lqer

Details

AAAI Conference 2024 Conference Paper

ProAgent: Building Proactive Cooperative Agents with Large Language Models

  • Ceyao Zhang
  • Kaijie Yang
  • Siyi Hu
  • Zihao Wang
  • Guanghe Li
  • Yihang Sun
  • Cheng Zhang
  • Zhaowei Zhang

Building agents with adaptive behavior in cooperative tasks stands as a paramount goal in the realm of multi-agent systems. Current approaches to developing cooperative agents rely primarily on learning-based methods, whose policy generalization depends heavily on the diversity of teammates they interact with during the training phase. Such reliance, however, constrains the agents' capacity for strategic adaptation when cooperating with unfamiliar teammates, which becomes a significant challenge in zero-shot coordination scenarios. To address this challenge, we propose ProAgent, a novel framework that harnesses large language models (LLMs) to create proactive agents capable of dynamically adapting their behavior to enhance cooperation with teammates. ProAgent can analyze the present state, and infer the intentions of teammates from observations. It then updates its beliefs in alignment with the teammates' subsequent actual behaviors. Moreover, ProAgent exhibits a high degree of modularity and interpretability, making it easily integrated into various of coordination scenarios. Experimental evaluations conducted within the Overcooked-AI environment unveil the remarkable performance superiority of ProAgent, outperforming five methods based on self-play and population-based training when cooperating with AI agents. Furthermore, in partnered with human proxy models, its performance exhibits an average improvement exceeding 10% compared to the current state-of-the-art method. For more information about our project, please visit https://pku-proagent.github.io.

PDF Details DOI

ICML Conference 2024 Conference Paper

Reflected Flow Matching

  • Tianyu Xie 0001
  • Yu Zhu 0004
  • Longlin Yu
  • Tong Yang
  • Ziheng Cheng
  • Shiyue Zhang 0002
  • Xiangyu Zhang
  • Cheng Zhang

Continuous normalizing flows (CNFs) learn an ordinary differential equation to transform prior samples into data. Flow matching (FM) has recently emerged as a simulation-free approach for training CNFs by regressing a velocity model towards the conditional velocity field. However, on constrained domains, the learned velocity model may lead to undesirable flows that result in highly unnatural samples, e. g. , oversaturated images, due to both flow matching error and simulation error. To address this, we add a boundary constraint term to CNFs, which leads to reflected CNFs that keep trajectories within the constrained domains. We propose reflected flow matching (RFM) to train the velocity model in reflected CNFs by matching the conditional velocity fields in a simulation-free manner, similar to the vanilla FM. Moreover, the analytical form of conditional velocity fields in RFM avoids potentially biased approximations, making it superior to existing score-based generative models on constrained domains. We demonstrate that RFM achieves comparable or better results on standard image benchmarks and produces high-quality class-conditioned samples under high guidance weight.

Details

NeurIPS Conference 2024 Conference Paper

Visual Data Diagnosis and Debiasing with Concept Graphs

  • Rwiddhi Chakraborty
  • Yinong Wang
  • Jialu Gao
  • Runkai Zheng
  • Cheng Zhang
  • Fernando De la Torre

The widespread success of deep learning models today is owed to the curation of extensive datasets significant in size and complexity. However, such models frequently pick up inherent biases in the data during the training process, leading to unreliable predictions. Diagnosing and debiasing datasets is thus a necessity to ensure reliable model performance. In this paper, we present ConBias, a novel framework for diagnosing and mitigating Concept co-occurrence Biases in visual datasets. ConBias represents visual datasets as knowledge graphs of concepts, enabling meticulous analysis of spurious concept co-occurrences to uncover concept imbalances across the whole dataset. Moreover, we show that by employing a novel clique-based concept balancing strategy, we can mitigate these imbalances, leading to enhanced performance on downstream tasks. Extensive experiments show that data augmentation based on a balanced concept distribution augmented by ConBias improves generalization performance across multiple datasets compared to state-of-the-art methods.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

ARTree: A Deep Autoregressive Model for Phylogenetic Inference

  • Tianyu Xie
  • Cheng Zhang

Designing flexible probabilistic models over tree topologies is important for developing efficient phylogenetic inference methods. To do that, previous works often leverage the similarity of tree topologies via hand-engineered heuristic features which would require domain expertise and may suffer from limited approximation capability. In this paper, we propose a deep autoregressive model for phylogenetic inference based on graph neural networks (GNNs), called ARTree. By decomposing a tree topology into a sequence of leaf node addition operations and modeling the involved conditional distributions based on learnable topological features via GNNs, ARTree can provide a rich family of distributions over tree topologies that have simple sampling algorithms, without using heuristic features. We demonstrate the effectiveness and efficiency of our method on a benchmark of challenging real data tree topology density estimation and variational Bayesian phylogenetic inference problems.

PDF Details

NeurIPS Conference 2023 Conference Paper

BayesDAG: Gradient-Based Posterior Inference for Causal Discovery

  • Yashas Annadani
  • Nick Pawlowski
  • Joel Jennings
  • Stefan Bauer
  • Cheng Zhang
  • Wenbo Gong

Bayesian causal discovery aims to infer the posterior distribution over causal models from observed data, quantifying epistemic uncertainty and benefiting downstream tasks. However, computational challenges arise due to joint inference over combinatorial space of Directed Acyclic Graphs (DAGs) and nonlinear functions. Despite recent progress towards efficient posterior inference over DAGs, existing methods are either limited to variational inference on node permutation matrices for linear causal models, leading to compromised inference accuracy, or continuous relaxation of adjacency matrices constrained by a DAG regularizer, which cannot ensure resulting graphs are DAGs. In this work, we introduce a scalable Bayesian causal discovery framework based on a combination of stochastic gradient Markov Chain Monte Carlo (SG-MCMC) and Variational Inference (VI) that overcomes these limitations. Our approach directly samples DAGs from the posterior without requiring any DAG regularization, simultaneously draws function parameter samples and is applicable to both linear and nonlinear causal models. To enable our approach, we derive a novel equivalence to the permutation-based DAG learning, which opens up possibilities of using any relaxed gradient estimator defined over permutations. To our knowledge, this is the first framework applying gradient-based MCMC sampling for causal discovery. Empirical evaluation on synthetic and real-world datasets demonstrate our approach's effectiveness compared to state-of-the-art baselines.

PDF Details

NeurIPS Conference 2023 Conference Paper

Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration

  • Longlin Yu
  • Tianyu Xie
  • Yu Zhu
  • Tong Yang
  • Xiangyu Zhang
  • Cheng Zhang

Semi-implicit variational inference (SIVI) has been introduced to expand the analytical variational families by defining expressive semi-implicit distributions in a hierarchical manner. However, the single-layer architecture commonly used in current SIVI methods can be insufficient when the target posterior has complicated structures. In this paper, we propose hierarchical semi-implicit variational inference, called HSIVI, which generalizes SIVI to allow more expressive multi-layer construction of semi-implicit distributions. By introducing auxiliary distributions that interpolate between a simple base distribution and the target distribution, the conditional layers can be trained by progressively matching these auxiliary distributions one layer after another. Moreover, given pre-trained score networks, HSIVI can be used to accelerate the sampling process of diffusion models with the score matching objective. We show that HSIVI significantly enhances the expressiveness of SIVI on several Bayesian inference problems with complicated target distributions. When used for diffusion model acceleration, we show that HSIVI can produce high quality samples comparable to or better than the existing fast diffusion model based samplers with a small number of function evaluations on various datasets.

PDF Details

NeurIPS Conference 2023 Conference Paper

High Precision Causal Model Evaluation with Conditional Randomization

  • Chao Ma
  • Cheng Zhang

The gold standard for causal model evaluation involves comparing model predictions with true effects estimated from randomized controlled trials (RCT). However, RCTs are not always feasible or ethical to perform. In contrast, conditionally randomized experiments based on inverse probability weighting (IPW) offer a more realistic approach but may suffer from high estimation variance. To tackle this challenge and enhance causal model evaluation in real-world conditional randomization settings, we introduce a novel low-variance estimator for causal error, dubbed as the pairs estimator. By applying the same IPW estimator to both the model and true experimental effects, our estimator effectively cancels out the variance due to IPW and achieves a smaller asymptotic variance. Empirical studies demonstrate the improved of our estimator, highlighting its potential on achieving near-RCT performance. Our method offers a simple yet powerful solution to evaluate causal inference models in conditional randomization settings without complicated modification of the IPW estimator itself, paving the way for more robust and reliable model assessments.

PDF Details

TMLR Journal 2023 Journal Article

Learn the Time to Learn: Replay Scheduling in Continual Learning

  • Marcus Klasson
  • Hedvig Kjellstrom
  • Cheng Zhang

Replay methods are known to be successful at mitigating catastrophic forgetting in continual learning scenarios despite having limited access to historical data. However, storing historical data is cheap in many real-world settings, yet replaying all historical data is often prohibited due to processing time constraints. In such settings, we propose that continual learning systems should learn the time to learn and schedule which tasks to replay at different time steps. We first demonstrate the benefits of our proposal by using Monte Carlo tree search to find a proper replay schedule, and show that the found replay schedules can outperform fixed scheduling policies when combined with various replay methods in different continual learning settings. Additionally, we propose a framework for learning replay scheduling policies with reinforcement learning. We show that the learned policies can generalize better in new continual learning scenarios compared to equally replaying all seen tasks, without added computational cost. Our study reveals the importance of learning the time to learn in continual learning, which brings current research closer to real-world needs.

PDF Details

ICLR Conference 2023 Conference Paper

Learnable Topological Features For Phylogenetic Inference via Graph Neural Networks

  • Cheng Zhang

Structural information of phylogenetic tree topologies plays an important role in phylogenetic inference. However, finding appropriate topological structures for specific phylogenetic inference tasks often requires significant design effort and domain expertise. In this paper, we propose a novel structural representation method for phylogenetic inference based on learnable topological features. By combining the raw node features that minimize the Dirichlet energy with modern graph representation learning techniques, our learnable topological features can provide efficient structural information of phylogenetic trees that automatically adapts to different downstream tasks without requiring domain expertise. We demonstrate the effectiveness and efficiency of our method on a simulated data tree probability estimation task and a benchmark of challenging real data variational Bayesian phylogenetic inference problems.

Details

TMLR Journal 2023 Journal Article

Optimistic Optimization of Gaussian Process Samples

  • Julia Grosse
  • Cheng Zhang
  • Philipp Hennig

Bayesian optimization is a popular formalism for global optimization, but its computational costs limit it to expensive-to-evaluate functions. A competing, computationally more effi- cient, global optimization framework is optimistic optimization, which exploits prior knowl- edge about the geometry of the search space in form of a dissimilarity function. We investi- gate to which degree the conceptual advantages of Bayesian Optimization can be combined with the computational efficiency of optimistic optimization. By mapping the kernel to a dissimilarity, we obtain an optimistic optimization algorithm for the Bayesian Optimization setting with a run-time of up to $O(N log N )$. As a high-level take-away we find that, when using stationary kernels on objectives of low evaluation cost, optimistic optimization can be preferable over Bayesian optimization, while for strongly coupled and parametric models, Bayesian optimization can perform much better, even at low evaluation cost. As a concep- tual takeaway, our results demonstrate that balancing exploration and exploitation under Gaussian process assumptions does not require computing a posterior.

PDF Details

NeurIPS Conference 2023 Conference Paper

Particle-based Variational Inference with Generalized Wasserstein Gradient Flow

  • Ziheng Cheng
  • Shiyue Zhang
  • Longlin Yu
  • Cheng Zhang

Particle-based variational inference methods (ParVIs) such as Stein variational gradient descent (SVGD) update the particles based on the kernelized Wasserstein gradient flow for the Kullback-Leibler (KL) divergence. However, the design of kernels is often non-trivial and can be restrictive for the flexibility of the method. Recent works show that functional gradient flow approximations with quadratic form regularization terms can improve performance. In this paper, we propose a ParVI framework, called generalized Wasserstein gradient descent (GWG), based on a generalized Wasserstein gradient flow of the KL divergence, which can be viewed as a functional gradient method with a broader class of regularizers induced by convex functions. We show that GWG exhibits strong convergence guarantees. We also provide an adaptive version that automatically chooses Wasserstein metric to accelerate convergence. In experiments, we demonstrate the effectiveness and efficiency of the proposed framework on both simulated and real data problems.

PDF Details

ICLR Conference 2023 Conference Paper

Semi-Implicit Variational Inference via Score Matching

  • Longlin Yu
  • Cheng Zhang

Semi-implicit variational inference (SIVI) greatly enriches the expressiveness of variational families by considering implicit variational distributions defined in a hierarchical manner. However, due to the intractable densities of variational distributions, current SIVI approaches often use surrogate evidence lower bounds (ELBOs) or employ expensive inner-loop MCMC runs for unbiased ELBOs for training. In this paper, we propose SIVI-SM, a new method for SIVI based on an alternative training objective via score matching. Leveraging the hierarchical structure of semi-implicit variational families, the score matching objective allows a minimax formulation where the intractable variational densities can be naturally handled with denoising score matching. We show that SIVI-SM closely matches the accuracy of MCMC and outperforms ELBO-based SIVI methods in a variety of Bayesian inference tasks.

Details

CLeaR Conference 2022 Conference Paper

Local Constraint-Based Causal Discovery under Selection Bias

  • Philip Versteeg
  • Joris Mooij
  • Cheng Zhang

We consider the problem of discovering causal relations from independence constraints selection bias in addition to confounding is present. While the seminal FCI algorithm is sound and complete in this setup, no criterion for the causal interpretation of its output under selection bias is presently known. We focus instead on local patterns of independence relations, where we find no sound method for only three variable that can include background knowledge. Y-Structure patterns are shown to be sound in predicting causal relations from data under selection bias, where cycles may be present. We introduce a finite-sample scoring rule for Y-Structures that is shown to successfully predict causal relations in simulation experiments that include selection mechanisms. On real-world microarray data, we show that a Y-Structure variant performs well across different datasets, potentially circumventing spurious correlations due to selection bias.

PDF Details

NeurIPS Conference 2022 Conference Paper

Simultaneous Missing Value Imputation and Structure Learning with Groups

  • Pablo Morales-Alvarez
  • Wenbo Gong
  • Angus Lamb
  • Simon Woodhead
  • Simon Peyton Jones
  • Nick Pawlowski
  • Miltiadis Allamanis
  • Cheng Zhang

Learning structures between groups of variables from data with missing values is an important task in the real world, yet difficult to solve. One typical scenario is discovering the structure among topics in the education domain to identify learning pathways. Here, the observations are student performances for questions under each topic which contain missing values. However, most existing methods focus on learning structures between a few individual variables from the complete data. In this work, we propose VISL, a novel scalable structure learning approach that can simultaneously infer structures between groups of variables under missing data and perform missing value imputations with deep learning. Particularly, we propose a generative model with a structured latent space and a graph neural network-based architecture, scaling to a large number of variables. Empirically, we conduct extensive experiments on synthetic, semi-synthetic, and real-world education data sets. We show improved performances on both imputation and structure learning accuracy compared to popular and recent approaches.

PDF Details

AAAI Conference 2021 Conference Paper

Estimating α-Rank by Maximizing Information Gain

  • Tabish Rashid
  • Cheng Zhang
  • Kamil Ciosek

Game theory has been increasingly applied in settings where the game is not known outright, but has to be estimated by sampling. For example, meta-games that arise in multi-agent evaluation can only be accessed by running a succession of expensive experiments that may involve simultaneous deployment of several agents. In this paper, we focus on α-rank, a popular game-theoretic solution concept designed to perform well in such scenarios. We aim to estimate the α-rank of the game using as few samples as possible. Our algorithm maximizes information gain between an epistemic belief over the α-ranks and the observed payoff. This approach has two main benefits. First, it allows us to focus our sampling on the entries that matter the most for identifying the α-rank. Second, the Bayesian formulation provides a facility to build in modeling assumptions by using a prior over game payoffs. We show the benefits of using information gain as compared to the confidence interval criterion of ResponseGraphUCB, and provide theoretical results justifying our method.

PDF Details

NeurIPS Conference 2021 Conference Paper

Identifiable Generative models for Missing Not at Random Data Imputation

  • Chao Ma
  • Cheng Zhang

Real-world datasets often have missing values associated with complex generative processes, where the cause of the missingness may not be fully observed. This is known as missing not at random (MNAR) data. However, many imputation methods do not take into account the missingness mechanism, resulting in biased imputation values when MNAR data is present. Although there are a few methods that have considered the MNAR scenario, their model's identifiability under MNAR is generally not guaranteed. That is, model parameters can not be uniquely determined even with infinite data samples, hence the imputation results given by such models can still be biased. This issue is especially overlooked by many modern deep generative models. In this work, we fill in this gap by systematically analyzing the identifiability of generative models under MNAR. Furthermore, we propose a practical deep generative model which can provide identifiability guarantees under mild assumptions, for a wide range of MNAR mechanisms. Our method demonstrates a clear advantage for tasks on both synthetic data and multiple real-world scenarios with MNAR data.

PDF Details

NeurIPS Conference 2021 Conference Paper

On Model Calibration for Long-Tailed Object Detection and Instance Segmentation

  • Tai-Yu Pan
  • Cheng Zhang
  • Yandong Li
  • Hexiang Hu
  • Dong Xuan
  • Soravit Changpinyo
  • Boqing Gong
  • Wei-Lun Chao

Vanilla models for object detection and instance segmentation suffer from the heavy bias toward detecting frequent objects in the long-tailed setting. Existing methods address this issue mostly during training, e. g. , by re-sampling or re-weighting. In this paper, we investigate a largely overlooked approach --- post-processing calibration of confidence scores. We propose NorCal, Normalized Calibration for long-tailed object detection and instance segmentation, a simple and straightforward recipe that reweighs the predicted scores of each class by its training sample size. We show that separately handling the background class and normalizing the scores over classes for each proposal are keys to achieving superior performance. On the LVIS dataset, NorCal can effectively improve nearly all the baseline models not only on rare classes but also on common and frequent classes. Finally, we conduct extensive analysis and ablation studies to offer insights into various modeling choices and mechanisms of our approach. Our code is publicly available at https: //github. com/tydpan/NorCal.

PDF Details

UAI Conference 2021 Conference Paper

Probabilistic DAG search

  • Julia Grosse
  • Cheng Zhang
  • Philipp Hennig

Exciting contemporary machine learning problems have recently been phrased in the classic formalism of tree search — most famously, the game of Go. Interestingly, the state-space underlying these sequential decision-making problems often posses a more general latent structure than can be captured by a tree. In this work, we develop a probabilistic framework to exploit a search space’s latent structure and thereby share information across the search tree. The method is based on a combination of approximate inference in jointly Gaussian models for the explored part of the problem, and an abstraction for the unexplored part that imposes a reduction of complexity ad hoc. We empirically find our algorithm to compare favorably to existing non-probabilistic alternatives in Tic-Tac-Toe and a feature selection application.

Details

NeurIPS Conference 2021 Conference Paper

Sparse Uncertainty Representation in Deep Learning with Inducing Weights

  • Hippolyt Ritter
  • Martin Kukla
  • Cheng Zhang
  • Yingzhen Li

Bayesian Neural Networks and deep ensembles represent two modern paradigms of uncertainty quantification in deep learning. Yet these approaches struggle to scale mainly due to memory inefficiency, requiring parameter storage several times that of their deterministic counterparts. To address this, we augment each weight matrix with a small inducing weight matrix, projecting the uncertainty quantification into a lower dimensional space. We further extend Matheron’s conditional Gaussian sampling rule to enable fast weight sampling, which enables our inference method to maintain reasonable run-time as compared with ensembles. Importantly, our approach achieves competitive performance to the state-of-the-art in prediction and uncertainty estimation tasks with fully connected neural networks and ResNets, while reducing the parameter size to $\leq 24. 3\%$ of that of a single neural network.

PDF Details

NeurIPS Conference 2020 Conference Paper

A Causal View on Robustness of Neural Networks

  • Cheng Zhang
  • Kun Zhang
  • Yingzhen Li

We present a causal view on the robustness of neural networks against input manipulations, which applies not only to traditional classification tasks but also to general measurement data. Based on this view, we design a deep causal manipulation augmented model (deep CAMA) which explicitly models possible manipulations on certain causes leading to changes in the observed effect. We further develop data augmentation and test-time fine-tuning methods to improve deep CAMA's robustness. When compared with discriminative deep neural networks, our proposed model shows superior robustness against unseen manipulations. As a by-product, our model achieves disentangled representation which separates the representation of manipulations from those of other latent causes.

PDF Details

NeurIPS Conference 2020 Conference Paper

How do fair decisions fare in long-term qualification?

  • Xueru Zhang
  • Ruibo Tu
  • Yang Liu
  • Mingyan Liu
  • Hedvig Kjellstrom
  • Kun Zhang
  • Cheng Zhang

Although many fairness criteria have been proposed for decision making, their long-term impact on the well-being of a population remains unclear. In this work, we study the dynamics of population qualification and algorithmic decisions under a partially observed Markov decision problem setting. By characterizing the equilibrium of such dynamics, we analyze the long-term impact of static fairness constraints on the equality and improvement of group well-being. Our results show that static fairness constraints can either promote equality or exacerbate disparity depending on the driving factor of qualification transitions and the effect of sensitive attributes on feature distributions. We also consider possible interventions that can effectively improve group qualification or promote equality of group qualification. Our theoretical results and experiments on static real-world datasets with simulated dynamics show that our framework can be used to facilitate social science studies.

PDF Details

NeurIPS Conference 2020 Conference Paper

Improved Variational Bayesian Phylogenetic Inference with Normalizing Flows

  • Cheng Zhang

Variational Bayesian phylogenetic inference (VBPI) provides a promising general variational framework for efficient estimation of phylogenetic posteriors. However, the current diagonal Lognormal branch length approximation would significantly restrict the quality of the approximating distributions. In this paper, we propose a new type of VBPI, VBPI-NF, as a first step to empower phylogenetic posterior estimation with deep learning techniques. By handling the non-Euclidean branch length space of phylogenetic models with carefully designed permutation equivariant transformations, VBPI-NF uses normalizing flows to provide a rich family of flexible branch length distributions that generalize across different tree topologies. We show that VBPI-NF significantly improves upon the vanilla VBPI on a benchmark of challenging real data Bayesian phylogenetic inference problems. Further investigation also reveals that the structured parameterization in those permutation equivariant transformations can provide additional amortization benefit.

PDF Details

AAAI Conference 2020 Conference Paper

Learning from Easy to Complex: Adaptive Multi-Curricula Learning for Neural Dialogue Generation

  • Hengyi Cai
  • Hongshen Chen
  • Cheng Zhang
  • Yonghao Song
  • Xiaofang Zhao
  • Yangxi Li
  • Dongsheng Duan
  • Dawei Yin

Current state-of-the-art neural dialogue systems are mainly data-driven and are trained on human-generated responses. However, due to the subjectivity and open-ended nature of human conversations, the complexity of training dialogues varies greatly. The noise and uneven complexity of query-response pairs impede the learning efficiency and effects of the neural dialogue generation models. What is more, so far, there are no unified dialogue complexity measurements, and the dialogue complexity embodies multiple aspects of attributes— specificity, repetitiveness, relevance, etc. Inspired by human behaviors of learning to converse, where children learn from easy dialogues to complex ones and dynamically adjust their learning progress, in this paper, we first analyze five dialogue attributes to measure the dialogue complexity in multiple perspectives on three publicly available corpora. Then, we propose an adaptive multi-curricula learning framework to schedule a committee of the organized curricula. The framework is established upon the reinforcement learning paradigm, which automatically chooses different curricula at the evolving learning process according to the learning status of the neural dialogue generation model. Extensive experiments conducted on five state-of-the-art models demonstrate its learning efficiency and effectiveness with respect to 13 automatic evaluation metrics and human judgments.

PDF Details

NeurIPS Conference 2020 Conference Paper

VAEM: a Deep Generative Model for Heterogeneous Mixed Type Data

  • Chao Ma
  • Sebastian Tschiatschek
  • Richard Turner
  • José Miguel Hernández-Lobato
  • Cheng Zhang

Deep generative models often perform poorly in real-world applications due to the heterogeneity of natural data sets. Heterogeneity arises from data containing different types of features (categorical, ordinal, continuous, etc. ) and features of the same type having different marginal distributions. We propose an extension of variational autoencoders (VAEs) called VAEM to handle such heterogeneous data. VAEM is a deep generative model that is trained in a two stage manner, such that the first stage provides a more uniform representation of the data to the second stage, thereby sidestepping the problems caused by heterogeneous data. We provide extensions of VAEM to handle partially observed data, and demonstrate its performance in data generation, missing data prediction and sequential feature selection tasks. Our results show that VAEM broadens the range of real-world applications where deep generative models can be successfully deployed.

PDF Details

AAAI Conference 2020 Conference Paper

Weakly-Supervised Fine-Grained Event Recognition on Social Media Texts for Disaster Management

  • Wenlin Yao
  • Cheng Zhang
  • Shiva Saravanan
  • Ruihong Huang
  • Ali Mostafavi

People increasingly use social media to report emergencies, seek help or share information during disasters, which makes social networks an important tool for disaster management. To meet these time-critical needs, we present a weakly supervised approach for rapidly building high-quality classi- fiers that label each individual Twitter message with finegrained event categories. Most importantly, we propose a novel method to create high-quality labeled data in a timely manner that automatically clusters tweets containing an event keyword and asks a domain expert to disambiguate event word senses and label clusters quickly. In addition, to process extremely noisy and often rather short user-generated messages, we enrich tweet representations using preceding context tweets and reply tweets in building event recognition classifiers. The evaluation on two hurricanes, Harvey and Florence, shows that using only 1-2 person-hours of human supervision, the rapidly trained weakly supervised classifiers outperform supervised classifiers trained using more than ten thousand annotated tweets created in over 50 person-hours.

PDF Details

AAAI Conference 2019 Conference Paper

Active Mini-Batch Sampling Using Repulsive Point Processes

  • Cheng Zhang
  • Cengiz Öztireli
  • Stephan Mandt
  • Giampiero Salvi

The convergence speed of stochastic gradient descent (SGD) can be improved by actively selecting mini-batches. We explore sampling schemes where similar data points are less likely to be selected in the same mini-batch. In particular, we prove that such repulsive sampling schemes lower the variance of the gradient estimator. This generalizes recent work on using Determinantal Point Processes (DPPs) for mini-batch diversification (Zhang et al. , 2017) to the broader class of repulsive point processes. We first show that the phenomenon of variance reduction by diversified sampling generalizes in particular to non-stationary point processes. We then show that other point processes may be computationally much more efficient than DPPs. In particular, we propose and investigate Poisson Disk sampling—frequently encountered in the computer graphics community—for this task. We show empirically that our approach improves over standard SGD both in terms of convergence speed as well as final model performance.

PDF Details

NeurIPS Conference 2019 Conference Paper

Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck

  • Maximilian Igl
  • Kamil Ciosek
  • Yingzhen Li
  • Sebastian Tschiatschek
  • Cheng Zhang
  • Sam Devlin
  • Katja Hofmann

The ability for policies to generalize to new environments is key to the broad application of RL agents. A promising approach to prevent an agent’s policy from overfitting to a limited set of training environments is to apply regularization techniques originally developed for supervised learning. However, there are stark differences between supervised learning and RL. We discuss those differences and propose modifications to existing regularization techniques in order to better adapt them to RL. In particular, we focus on regularization techniques relying on the injection of noise into the learned function, a family that includes some of the most widely used approaches such as Dropout and Batch Normalization. To adapt them to RL, we propose Selective Noise Injection (SNI), which maintains the regularizing effect the injected noise has, while mitigating the adverse effects it has on the gradient quality. Furthermore, we demonstrate that the Information Bottleneck (IB) is a particularly well suited regularization technique for RL as it is effective in the low-data regime encountered early on in training RL agents. Combining the IB with SNI, we significantly outperform current state of the art results, including on the recently proposed generalization benchmark Coinrun.

PDF Details

NeurIPS Conference 2019 Conference Paper

Icebreaker: Element-wise Efficient Information Acquisition with a Bayesian Deep Latent Gaussian Model

  • Wenbo Gong
  • Sebastian Tschiatschek
  • Sebastian Nowozin
  • Richard Turner
  • José Miguel Hernández-Lobato
  • Cheng Zhang

In this paper, we address the ice-start problem, i. e. , the challenge of deploying machine learning models when only a little or no training data is initially available, and acquiring each feature element of data is associated with costs. This setting is representative of the real-world machine learning applications. For instance, in the health care domain, obtaining every single measurement comes with a cost. We propose Icebreaker, a principled framework for elementwise training data acquisition. Icebreaker introduces a full Bayesian Deep Latent Gaussian Model (BELGAM) with a novel inference method, which combines recent advances in amortized inference and stochastic gradient MCMC to enable fast and accurate posterior inference. By utilizing BELGAM’s ability to fully quantify model uncertainty, we also propose two information acquisition functions for imputation and active prediction problems. We demonstrate that BELGAM performs significantly better than previous variational autoencoder (VAE) based models, when the data set size is small, using both machine learning benchmarks and real world recommender systems and health-care applications. Moreover, Icebreaker not only demonstrates improved performance compared to baselines, but it is also capable of achieving better test performance with less training data available.

PDF Details

NeurIPS Conference 2019 Conference Paper

Neuropathic Pain Diagnosis Simulator for Causal Discovery Algorithm Evaluation

  • Ruibo Tu
  • Kun Zhang
  • Bo Bertilson
  • Hedvig Kjellstrom
  • Cheng Zhang

Discovery of causal relations from observational data is essential for many disciplines of science and real-world applications. However, unlike other machine learning algorithms, whose development has been greatly fostered by a large amount of available benchmark datasets, causal discovery algorithms are notoriously difficult to be systematically evaluated because few datasets with known ground-truth causal relations are available. In this work, we handle the problem of evaluating causal discovery algorithms by building a flexible simulator in the medical setting. We develop a neuropathic pain diagnosis simulator, inspired by the fact that the biological processes of neuropathic pathophysiology are well studied with well-understood causal influences. Our simulator exploits the causal graph of the neuropathic pain pathology and its parameters in the generator are estimated from real-life patient cases. We show that the data generated from our simulator have similar statistics as real-world data. As a clear advantage, the simulator can produce infinite samples without jeopardizing the privacy of real-world patients. Our simulator provides a natural tool for evaluating various types of causal discovery algorithms, including those to deal with practical issues in causal discovery, such as unknown confounders, selection bias, and missing data. Using our simulator, we have evaluated extensively causal discovery algorithms under various settings.

PDF Details

JBHI Journal 2019 Journal Article

Nonrigid Image Registration Using Spatially Region-Weighted Correlation Ratio and GPU-Acceleration

  • Lun Gong
  • Cheng Zhang
  • Luwen Duan
  • Xueying Du
  • Hanqiu Liu
  • Xinjian Chen
  • Jian Zheng

Objective: Nonrigid image registration with high accuracy and efficiency remains a challenging task for medical image analysis. In this paper, we present the spatially region-weighted correlation ratio (SRWCR) as a novel similarity measure to improve the registration performance. Methods: SRWCR is rigorously deduced from a three-dimension joint probability density function combining the intensity channels with an extra spatial information channel. SRWCR estimates the optimal functional dependence between the intensities for each spatial bin, in which the spatial distribution modeled by a cubic B-spline function is used to differentiate the contribution of voxels. We also analytically derive the gradient of SRWCR with respect to the transformation parameters and optimize it using a quasi-Newton approach. Furthermore, we propose a GPUbased parallel mechanism to accelerate the computation of SRWCR and its derivatives. Results: The experiments on synthetic images, public four-dimensional thoracic computed tomography (CT) dataset, retinal optical coherence tomography data, and clinical CT and positron emission tomography images confirm that SRWCR significantly outperforms some state-of-the-art techniques such as spatially encoded mutual information and Robust PaTch-based cOrrelation Ration. Conclusion: This study demonstrates the advantages of SRWCR in tackling the practical difficulties due to distinct intensity changes, serious speckle noise, or different imaging modalities. Significance: The proposed registration framework might be more reliable to correct the nonrigid deformations and more potential for clinical applications.

Details DOI

AAAI Conference 2018 Conference Paper

An Ant-Based Algorithm to Solve Distributed Constraint Optimization Problems

  • Ziyu Chen
  • Tengfei Wu
  • Yanchen Deng
  • Cheng Zhang

As an important population-based algorithm, ant colony optimization (ACO) has been successfully applied into various combinatorial optimization problems. However, much existing work in ACO focuses on solving centralized problems. In this paper, we present a novel algorithm that takes the power of ants to solve Distributed Constraint Optimization Problems (DCOPs), called ACO DCOP. In ACO DCOP, a new mechanism that captures local benefits is proposed to compute heuristic factors and a new method that considers the cost structure of DCOPs is proposed to compute pheromone deltas appropriately. Moreover, pipelining technique is introduced to make full use of the computational capacity and improve the efficiency. In our theoretical analysis, we prove that ACO DCOP is an anytime algorithm. Our empirical evaluation indicates that ACO DCOP is able to find solutions of equal or significantly higher quality than state-of-the-art DCOP algorithms.

PDF Details

AAAI Conference 2018 Conference Paper

Emphasizing 3D Properties in Recurrent Multi-View Aggregation for 3D Shape Retrieval

  • Cheng Xu
  • Biao Leng
  • Cheng Zhang
  • Xiaochen Zhou

Multi-view based shape descriptors have achieved impressive performance for 3D shape retrieval. The core of view-based methods is to interpret 3D structures through 2D observations. However, most existing methods pay more attention to discriminative models and none of them necessarily incorporate the 3D properties of the objects. To resolve this problem, we propose an encoder-decoder recurrent feature aggregation network (ERFA-Net) to emphasize the 3D properties of 3D shapes in multi-view features aggregation. In our network, a view sequence of the shape is trained to encode a discriminative shape embedding and estimate unseen rendered views of any viewpoints. This generation task gives an effective supervision which makes the network exploit 3D properties of shapes through various 2D images. During feature aggregation, a discriminative feature representation across multiple views is effectively exploited based on LSTM network. The proposed 3D representation has following advantages against other state-of-the-art: 1) it performs robust discrimination under the existence of noise such as view missing and occlusion, because of the improvement brought by 3D properties. 2) it has strong generative capabilities, which is useful for various 3D shape tasks. We evaluate ERFA-Net on two popular 3D shape datasets, ModelNet and ShapeNetCore55, and ERFA-Net outperforms the state-of-the-art methods significantly. Extensive experiments show the effectiveness and robustness of the proposed 3D representation.

PDF Details

NeurIPS Conference 2018 Conference Paper

Generalizing Tree Probability Estimation via Bayesian Networks

  • Cheng Zhang
  • Frederick A Matsen IV

Probability estimation is one of the fundamental tasks in statistics and machine learning. However, standard methods for probability estimation on discrete objects do not handle object structure in a satisfactory manner. In this paper, we derive a general Bayesian network formulation for probability estimation on leaf-labeled trees that enables flexible approximations which can generalize beyond observations. We show that efficient algorithms for learning Bayesian networks can be easily extended to probability estimation on this challenging structured space. Experiments on both synthetic and real data show that our methods greatly outperform the current practice of using the empirical distribution, as well as a previous effort for probability estimation on trees.

PDF Details

NeurIPS Conference 2017 Conference Paper

Perturbative Black Box Variational Inference

  • Robert Bamler
  • Cheng Zhang
  • Manfred Opper
  • Stephan Mandt

Black box variational inference (BBVI) with reparameterization gradients triggered the exploration of divergence measures other than the Kullback-Leibler (KL) divergence, such as alpha divergences. In this paper, we view BBVI with generalized divergences as a form of estimating the marginal likelihood via biased importance sampling. The choice of divergence determines a bias-variance trade-off between the tightness of a bound on the marginal likelihood (low bias) and the variance of its gradient estimators. Drawing on variational perturbation theory of statistical physics, we use these insights to construct a family of new variational bounds. Enumerated by an odd integer order $K$, this family captures the standard KL bound for $K=1$, and converges to the exact marginal likelihood as $K\to\infty$. Compared to alpha-divergences, our reparameterization gradients have a lower variance. We show in experiments on Gaussian Processes and Variational Autoencoders that the new bounds are more mass covering, and that the resulting posterior covariances are closer to the true posterior and lead to higher likelihoods on held-out data.

PDF Details

ICML Conference 2017 Conference Paper

Probabilistic Path Hamiltonian Monte Carlo

  • Vu Dinh
  • Arman Bilge
  • Cheng Zhang
  • Frederick A. Matsen IV

Hamiltonian Monte Carlo (HMC) is an efficient and effective means of sampling posterior distributions on Euclidean space, which has been extended to manifolds with boundary. However, some applications require an extension to more general spaces. For example, phylogenetic (evolutionary) trees are defined in terms of both a discrete graph and associated continuous parameters; although one can represent these aspects using a single connected space, this rather complex space is not suitable for existing HMC algorithms. In this paper, we develop Probabilistic Path HMC (PPHMC) as a first step to sampling distributions on spaces with intricate combinatorial structure. We define PPHMC on orthant complexes, show that the resulting Markov chain is ergodic, and provide a promising implementation for the case of phylogenetic trees in open-source software. We also show that a surrogate function to ease the transition across a boundary on which the log-posterior has discontinuous derivatives can greatly improve efficiency.

Details

AAAI Conference 2010 Conference Paper

G-Optimal Design with Laplacian Regularization

  • Chun Chen
  • Zhengguang Chen
  • Jiajun Bu
  • Can Wang
  • Lijun Zhang
  • Cheng Zhang

In many real world applications, labeled data are usually expensive to get, while there may be a large amount of unlabeled data. To reduce the labeling cost, active learning attempts to discover the most informative data points for labeling. Recently, Optimal Experimental Design (OED) techniques have attracted an increasing amount of attention. OED is concerned with the design of experiments that minimizes variances of a parameterized model. Typical design criteria include D-, A-, and E-optimality. However, all these criteria are based on an ordinary linear regression model which aims to minimize the empirical error whereas the geometrical structure of the data space is not well respected. In this paper, we propose a novel optimal experimental design approach for active learning, called Laplacian G-Optimal Design (LapGOD), which considers both discriminating and geometrical structures. By using Laplacian Regularized Least Squares which incorporates manifold regularization into linear regression, our proposed algorithm selects those data points that minimizes the maximum variance of the predicted values on the data manifold. We also extend our algorithm to nonlinear case by using kernel trick. The experimental results on various image databases have shown that our proposed LapGOD active learning algorithm can significantly enhance the classification accuracy if the selected data points are used as training data.

PDF Details

IROS Conference 2009 Conference Paper

A design of the electromagnetic driver for the "internal force-static friction" capsubot

  • Gang Su
  • Cheng Zhang
  • Renjia Tan
  • Hongyi Li

A capsule robot that moves utilizing the internal force between the shell and the sliding mass and the friction between the shell and the environment is named "internal force-static friction" capsubot. The new driving mode is used in the active gastrointestinal examination robot system, which is a novel and meaningful attempt. In this paper, for the operation mechanism and movement characteristics of capsubot, a new type of electromagnetic driver has been designed. The driver is optimized and analyzed by magnetic circuit method and finite element method. Then we manufacture the prototype of the capsubot. The driver has big output power, a small size and a high weight proportion of the sliding mass and the shell, which has been proved by the experiment. The speed on the table is 45. 8mm/s.

Details
v2026.09.13