Hao Zhang

AAAI Conference 2026 Conference Paper

Audio-Thinker: Guiding Large Audio Language Model When and How to Think via Reinforcement Learning

• Shu Wu
• Chenxing Li
• Wenfu Wang
• Hao Zhang
• Hualei Wang
• Meng Yu
• Dong Yu

Recent advancements in large language models, multimodal large language models, and large audio language models (LALMs) have significantly improved their reasoning capabilities through reinforcement learning utilizing rule-based rewards. However, the explicit reasoning process has not yet yielded substantial benefits for audio question answering, and effectively leveraging deep reasoning remains an open challenge, with LALMs still falling short of achieving human-level auditory-language reasoning. To address these limitations, we propose Audio-Thinker, a reinforcement learning framework designed to enhance the reasoning capabilities of LALMs through improved adaptability, consistency, and effectiveness. Our approach introduces an adaptive think accuracy reward, enabling the model to adjust its reasoning strategies based on task complexity. Furthermore, we incorporate an external reward model to evaluate the overall consistency and quality of the reasoning process, complemented by think-based rewards that assist the model in distinguishing between valid and flawed reasoning paths during training. Experimental results demonstrate that Audio-Thinker models outperform existing reasoning-oriented LALMs across various benchmark tasks, exhibiting superior reasoning and generalization capabilities.

AAAI Conference 2026 Conference Paper

Bidirectional Noise Injection: Enhancing Diffusion Models via Coordinated Input-Output Perturbation

• Tianyi Zheng
• Jiayang Gao
• Peng-tao Jiang
• Fengxiang Yang
• Ben Wan
• Hao Zhang
• Jinwei Chen
• Jia Wang

Diffusion models have demonstrated remarkable success in image generation, yet a persistent challenge remains: the bias between model predictions and the target distribution. In this paper, we propose a Bidirectional Noise Injection framework for enhancing diffusion models, implemented via Coordinated Input-Output Perturbation (CIOP). Our approach mitigates this bias by randomly applying synchronized noise injection to both the model inputs and the prediction targets during the training stage. This stochastic, synchronized noise injected acts as a smoothing mechanism that effectively reduces the 2-Wasserstein distance between the predicted and target distributions, as substantiated by our theoretical analysis based on optimal transport theory. Extensive experiments on multiple benchmark datasets and various generative tasks demonstrate that our method improves generation quality and training efficiency without incurring additional computational cost. Furthermore, the design of CIOP enables seamless integration with existing diffusion model improvements and advanced frameworks, thereby broadening its applicability. These results highlight the potential of Bidirectional Noise Injection via CIOP to alleviate bias in diffusion-based generative models across a wide range of settings.

EAAI Journal 2026 Journal Article

Channel Clustering-based Attention Network for interpretable hard landing prediction

• Hao Zhang
• Huabo Sun
• Yu Liu
• Xinbin Zhao
• Xu Li
• Jiaxing Shang
• Linjiang Zheng

Hard landing incidents are common flight safety events during the landing phase and are of significant concern in the aviation industry. Recent hard landing prediction methods tend to overemphasize temporal features while overlooking the landing process along the altitude dimension, importantly, altitude-based alignment enables more practical interpretability. Additionally, they often fail to capture dependencies between contributing factors and offer limited interpretability under fixed time windows. To address the above issues, we propose a Channel Clustering based Attention Network, termed CCAN, to predict hard landing incidents and identify their potential causes. Specifically, we resample and interpolate different flight parameters along the altitude dimension to align the landing process across different flights into a common reference frame for the subsequent interpretability. Subsequently, we design a channel clustering module that groups flight parameters into distinct clusters based on a predefined assignment threshold. Then, we employ graph attention network (GAT) to capture the dependencies between different flight parameters within and across clusters. To further reveal the interactions between flight parameters throughout the landing process, we incorporate attention mechanism into Gated Recurrent Units (GRUs) to extract informative temporal features. We conducted experiments on a real-world quick access recorders (QAR) dataset with 44, 729 Airbus A321 flights. Experimental results demonstrate that CCAN outperforms the baseline models in hard landing predictions and offers practical interpretability for hard landings by visualizing the dependencies between flight parameters and their interactions under the altitude reference during the landing process.

YNIMG Journal 2026 Journal Article

Converse or reverse? Machine-learning modeling for disease progression: A study based on Alzheimer’s disease continuum cohort

• Yujing Huang
• Hao Zhang
• Buqing Ma
• Zhe Yu
• Shenyi Dai
• Lu Cheng
• Li Su
• Gaoyi Yang

INTRODUCTION: Longitudinal trajectories from healthy aging to Mild Cognitive Impairment and Alzheimer's Disease involve complex mechanisms. METHODS: We evaluated five machine learning approaches (Random Forest, Support Vector Machines, Radial Basis Function Networks, Backpropagation Networks, Convolutional Neural Network) to assess the importance of potential predictive markers across the health-to-dementia continuum. Using the ADNI cohort across four phases (ADNI1, ADNIGO, ADNI2, ADNI3), we analyzed participants with distinct trajectories: stable, convertible, and reverse progression. RESULTS: Random Forest outperformed other models across key effectiveness metrics and achieved a macro-averaged sensitivity of 70.8 % and specificity of 96.8 % across all participant groups. Random Forest identified visuospatial and memory-related cognitive dysfunction as key predictive clinical features and several amyloid-related neuroimaging biomarkers - including temporal variations of amyloid uptake within inferior lateral ventricles, para-hippocampus-for classifying participant groups. Additionally, plasma APOE4 and long neurofilament light chain levels emerged as promising predictors for tracking progression. CONCLUSION: These findings highlight the potential of machine learning in classifying disease trajectories.

AAAI Conference 2026 Conference Paper

CycleChemist: A Dual-Pronged Machine Learning Framework for Organic Photovoltaic Discovery

• Hou Hei Lam
• Jiangjie Qiu
• Xiuyuan Hu
• Wentao Li
• Fankun Zeng
• Siwei Fu
• Hao Zhang
• Xiaonan Wang

Organic photovoltaic (OPV) materials offer a promising pathway for sustainable energy generation. However, their development is hindered by the challenge of identifying high-performance donor-acceptor pairs with optimal power conversion efficiencies (PCEs). Most existing design strategies focus exclusively on either the donor or the acceptor, rather than employing a unified model capable of designing both components. In this work, we introduce a dual-pronged machine learning framework for OPV discovery, integrating predictive modeling and generative molecular design. In this study, we propose the newly curated Organic Photovoltaic Donor-Acceptor Dataset (OPV²D), the largest of its kind, comprising 2,000 experimentally characterized donor-acceptor pairs. This dataset serves as a comprehensive foundation for model training and evaluation. To enable accurate property prediction in organic photovoltaic (OPV) materials, we first introduce the Organic Photovoltaic Classifier (OPVC) to predict the likelihood that a given material exhibits OPV behavior. Complementing this, we develop a hierarchical graph neural network framework that integrates multi-task learning and cross-modal donor–acceptor interaction modeling. This framework includes the Molecular Orbital Energy Estimator (MOE²) for predicting the highest occupied molecular orbital–lowest unoccupied molecular orbital (HOMO–LUMO) energy levels, and the Photovoltaic Performance Predictor (P³) for estimating power conversion efficiency (PCE). In addition, we introduce the Material Generative Pretrained Transformer (MatGPT) to generate synthetically accessible organic semiconductors. Building on this, we propose a reinforcement learning strategy with three-objective policy optimization that guides molecular generation while preserving chemical validity. By bridging molecular representation learning with device performance prediction, our framework advances computational OPV material discovery.

AAAI Conference 2026 Conference Paper

Diff-NAT: Better Naturalistic and Aggressive Adversarial Attacks via Class-Optimized Diffusion for Object Detection

• Qinglong Yan
• Tong Zou
• Xunpeng Yi
• Xinyu Xiang
• Xuying Wu
• Hao Zhang
• Jiayi Ma

Recent advances in naturalistic physical adversarial patch generation show great promise in protecting personal privacy against detector-based malicious surveillance while remaining inconspicuous to human observers. In this work, we present the first systematic categorization and in-depth re-examination of existing methods into three representative paradigms, revealing a pervasive imbalance: enforcing naturalness constraints inherently restricts the adversarial search space, thus limiting attack performance. To address this challenge, we propose a novel paradigm based on class-optimized diffusion, termed Diff-NAT. Diff-NAT leverages pretrained diffusion models as powerful natural image priors and introduces a unified iterative framework that jointly optimizes two complementary components: semantic-level textual prompts and instance-level latent codes. Specifically, prompt optimization enables broad traversal across inter-class semantic regions, while latent refinement allows for fine-grained manipulation within class objectives. This dual-level optimization facilitates progressive navigation toward adversarial distributions embedded within the natural semantic manifold. Extensive experiments in both digital and physical settings demonstrate that Diff-NAT outperforms existing SOTA approaches in terms of both visual realism and aggressiveness.

JBHI Journal 2026 Journal Article

Heterophily-Aware Spectral GCN for Population-Level Brain Disorder Prediction

• Hao Zhang
• Liping Wang
• Yitian Zhao
• Jianyang Xie
• Tiyu Fang
• Ran Song
• Wei Zhang

Integrating resting-state functional magnetic resonance imaging (rs-fMRI) and phenotypic data is a promising way to build a comprehensive population graph for the prediction of brain disorders using graph neural networks (GNNs). However, existing GNN-based methods face two limitations: the complexity of relationships between subjects poses challenges in constructing a well-defined population graph, and the inherent node heterophily within the population graph is often overlooked. To address them, we propose a population graph with a phenotypic encoder, which leverages rs-fMRI and phenotypic data to model complex relationships between subjects and enables GNN to learn population-level features. We also design a heterophily-aware spectral graph convolution network that incorporates local similarity-based learning to assess node homophily and addresses the heterophily issue. Experiments demonstrate that our method performs well in classifying both Alzheimer's Disease and Autism Spectrum Disorder. In addition, it can distinguish between progressive and stable mild cognitive impairment, facilitating timely interventions for the diseases.

AIIM Journal 2026 Journal Article

IKDP: Implicit Knowledge Enhanced Disease Prediction via heterogeneous admission sequence graphs

• Zongbao Yang
• Yuchen Lin
• Yichen He
• Jinlong Hu
• Ruxin Wang
• Hao Zhang
• Shoubin Dong

Despite significant advances in deep learning for electronic health record (EHR) modeling, accurately representing complex disease relationships and admission trajectories remains challenging. Current approaches that leverage external knowledge graphs to learn patient representations are often limited by incomplete knowledge coverage. Furthermore, these methods frequently overlook implicit information within patient data, such as inter-patient similarities and latent disease correlations, and often discard patients with only a single admission, thereby losing valuable clinical insights. To address these limitations, we introduce the Implicit Knowledge Enhanced Disease Prediction model (IKDP) via heterogeneous admission sequence graphs (SeqGs), which harnesses implicit knowledge from comprehensive patient admission data. IKDP integrates an auxiliary pre-training strategy with end-to-end optimization to effectively process multi-dimensional patient data and compute inter-patient similarities as complementary knowledge. Specifically, the model constructs SeqGs for each patient, which capture complex disease dependencies and the dynamic evolution of health status. Moreover, critical paths extracted from the SeqGs, combined with similar patient analysis and historical admission records, are utilized to elucidate the reasoning behind predictions. The code is available at https: //github. com/SCUT-CCNL/IKDP.

AAAI Conference 2026 Conference Paper

Invariant Feature Learning for Counterfactual Watch-time Prediction in Video Recommendation

• Chenghou Jin
• Yixin Ren
• Hongxu Ma
• Yewei Xia
• Yi Guan
• Hao Zhang
• Jiandong Ding
• Jihong Guan

Video recommendation systems heavily rely on user watch time feedback, making accurate watch time prediction a crucial task. However, this task inherently suffers from bias, as recommendation models tend to favor long-duration videos to maximize watch time. This issue, known as duration bias in the watch-time prediction context, can be explained from a causal perspective, where video duration acts as a confounder. Recent works address this bias using backdoor adjustment, isolating the direct effect of content on watch time from observational data. These methods typically discretize video duration into groups, estimate group-wise effects, and then aggregate them via a unified prediction model. However, this aggregation strategy is prone to model misspecification due to feature distribution shift across groups. In this paper, we reinterpret the problem through the lens of invariant learning and propose a novel framework: Duration-Invariant Feature Learning (DIFL). DIFL employs a kernel-based regularization that enforces representation invariance across duration groups, reducing sensitivity to group design and improving generalization. This enables more accurate modeling of the direct causal effect and making counterfactual inference. Extensive experiments on both public and real large-scale production datasets demonstrate the effectiveness of our approach, which achieves SOTA performance.

AAAI Conference 2026 Conference Paper

MoLoRA: Boosting LLM-based End-to-end Speech Translation with Mixture of Low-rank Experts

• Hao Zhang
• Yaqi Chen
• Nianwen Si
• XuKui Yang
• Wenlin Zhang
• Dan Qu

Recently, End-to-End Speech Translation (E2E-ST) methods leveraging large language models (LLMs) have demonstrated strong generalization capabilities and excellent scalability by integrating pre-trained speech encoders with LLMs, where Low-Rank Adaptation (LoRA) is commonly used for parameter-efficient fine-tuning to reduce training costs. However, LoRA's low-rank assumption often fails in multilingual tasks, as the inherent complexity of cross-lingual semantic relationships and syntactic variations exceeds the representational capacity of low-rank matrices. This leads to parameter conflicts across languages, resulting in suboptimal performance. To address this issue, we propose Mixture of Low-Rank Adaptations (MoLoRA), which integrates the Mixture of Experts (MoE) mechanism with LoRA. MoLoRA effectively enhances the model's expressive capacity while maintaining parameter efficiency during training. Specifically, we treat multiple LoRA modules as low-rank experts and introduce a routing mechanism to dynamically activate language-specific experts. Additionally, shared experts are incorporated and consistently activated to model cross-lingual general knowledge. Furthermore, to enhance the robustness and accuracy of speech representations, we propose a Multi-Granularity Representation Fusion module (MGRF). This module mitigates local distortions in frame-level speech representations caused by noise by fusing frame-level and sentence-level features, thereby providing the LLM with more accurate high-level semantic information. We conduct multilingual experiments on the MuST-C and CoVoST-2 datasets. Our method achieves an average BLEU score of 32.2 across eight language pairs on the MuST-C dataset and an average of 36.3 across three language pairs on the CoVoST-2 dataset, establishing a new state-of-the-art (SOTA) performance.

AAAI Conference 2026 Conference Paper

PKR-QA: A Benchmark for Procedural Knowledge Reasoning with Knowledge Module Learning

• Thanh-Son Nguyen
• Hong Yang
• Tzeh Yuan Neoh
• Hao Zhang
• Ee Yeo Keat
• Basura Fernando

We introduce PKR-QA (Procedural Knowledge Reasoning Question Answering), a new benchmark for question answering over procedural tasks that require structured reasoning. PKR-QA is constructed semi-automatically using a procedural knowledge graph (PKG), which encodes task-specific knowledge across diverse domains. The PKG is built by curating and linking information from the COIN instructional video dataset and the ontology, enriched with commonsense knowledge from ConceptNet and structured outputs from Large Language Models (LLMs), followed by manual verification. To generate question-answer pairs, we design graph traversal templates where each template is applied systematically over PKG. To enable interpretable reasoning, we propose a neurosymbolic approach called Knowledge Module Learning (KML), which learns procedural relations via neural modules and composes them for structured reasoning with LLMs. Experiments demonstrate that this paradigm improves reasoning performance on PKR-QA and enables step-by-step reasoning traces that facilitate interpretability.

JBHI Journal 2026 Journal Article

Rich-Neighborhood Contrastive Learning Framework for Drug Repositioning via Structural and Semantic Neighbor Fusion

• Yuhua Yao
• Hao Zhang
• Xianfang Tang
• Jincan Li
• Junlin Xu
• Jialiang Yang
• Yajie Meng

Drug repositioning accelerates therapeutic discovery by identifying new indications for approved drugs, substantially reducing the time and cost associated with drug development. However, graph collaborative filtering (GCF)–based methods for predicting drug–disease associations are limited by data sparsity and structural noise, impeding the modeling of latent high-order and semantic relationships. We hypothesize that jointly leveraging complementary information from structural and semantic neighborhoods can alleviate data sparsity and improve predictive performance. To this end, we propose a unified framework, Rich–Neighborhood Contrastive Learning for Drug Repositioning (RCL–DR), which integrates both structural and semantic neighborhood modeling into a LightGCN–based collaborative filtering backbone and optimizes semantic prototypes via an Expectation–Maximization (EM) algorithm. Experiments on three public datasets using 10 × 10- fold cross-validation demonstrate that RCL–DR outperforms representative baselines, achieving an area under the receiver operating characteristic curve (AUROC) of 0. 9419 and an area under the precision–recall curve (AUPR) of 0. 5126, representing absolute improvements of 0. 0345 and 0. 0138, respectively. Furthermore, RCL–DR identifies promising drug candidates for Alzheimer's disease (e. g. , buspirone) and Parkinson's disease (e. g. , trihexyphenidyl) by predicting previously unknown drug–disease associations on the Fdataset and validating them against authoritative databases. In summary, RCL–DR provides a unified contrastive learning framework for robust drug repositioning and precision pharmacology.

AAAI Conference 2026 Conference Paper

Robust Fusion Controller: Degradation-Aware Image Fusion with Fine-Grained Language Instructions

• Hao Zhang
• Yanping Zha
• Qingwei Zhuang
• Zhenfeng Shao
• Jiayi Ma

Current image fusion methods struggle to adapt to real-world environments encompassing diverse degradations with spatially varying characteristics. To address this challenge, we propose a robust fusion controller (RFC) capable of achieving degradation-aware image fusion through fine-grained language instructions, ensuring its reliable application in adverse environments. Specifically, RFC first parses language instructions to innovatively derive the functional condition and the spatial condition, where the former specifies the degradation type to remove, while the latter defines its spatial coverage. Then, a composite control priori is generated through a multi-condition coupling network, achieving a seamless transition from abstract language instructions to latent control variables. Subsequently, we design a hybrid attention-based fusion network to aggregate multi-modal information, in which the obtained composite control priori is deeply embedded to linearly modulate the intermediate fused features. To ensure the alignment between language instructions and control outcomes, we introduce a novel language-feature alignment loss, which constrains the consistency between feature-level gains and the composite control priori. Extensive experiments on publicly available datasets demonstrate that our RFC is robust against various composite degradations, particularly in highly challenging flare scenarios.

TMLR Journal 2026 Journal Article

Seeing is Simulating: Differentiable Physics for Interaction-Aware Material Estimation

• Chun Feng
• Hao Zhang
• Haolan Xu
• Narendra Ahuja

Modeling human-object interactions is crucial for creating immersive virtual experiences. However, synthesizing 3D object dynamics conditioned on actions remains a challenging problem. Existing approaches equip static 3D objects with motion priors distilled from video diffusion models. This methodology has two drawbacks: (i) video diffusion models are not physically grounded. Thus, the generated videos may contain physical inaccuracies; (ii) video diffusion models cannot generate complex dynamics where multiple objects interact under actions with long durations and large spatial extent. We present $\textbf{PhysInteract}$, a physics-based framework that (i) models interactions with a representation that captures their duration and contact information; (ii) estimates object material properties (e.g., Young's modulus) from objects' deformation caused by interactions; (iii) uses physics simulation to reproduce realistic object dynamics based on estimated interactions and material properties. We highlight that PhysInteract is fully differentiable, enabling joint optimization of interaction representations and object material properties. PhysInteract achieves better performance than existing methods. We demonstrate its superiority by quantitatively testing PhysInteract on a curated dataset. In conjunction with an additional user study, our method shows a step towards more realistic and immersive virtual experiences.

EAAI Journal 2026 Journal Article

Semi-supervised image segmentation via selective self-ensembling and boundary uncertainty suppression

• Xiaoguo Yang
• Yabo Wu
• Shouxiang Ni
• Hongmei He
• Hao Zhang
• Yu Chen
• Ke Yan
• Chuoying Tan

Image segmentation plays a key role in many image-guided clinical applications and deep learning technology has proven effective for this task when sufficient labeled images are available. However, it is very time-consuming and labor-intensive to obtain adequate pixel-level image labels. To alleviate the scarcity of labeled images, we propose a novel semi-supervised segmentation method based on the available uncertainty-aware mean teacher (UAMT) framework by introducing two different strategies, i. e. , selective self-ensembling (SSE) and boundary uncertainty suppression (BUS). The SSE dynamically selects multiple best student models across different training steps to update the teacher model's weights, while the BUS reduces boundary segmentation errors and improves the quantitative potential of loss functions through a unique uncertainty estimation function. With the two strategies, our proposed method was able to obtain promising segmentation performance with limited labeled images and abundant unlabeled ones. We trained and validated our proposed method by segmenting multiple objects from three public datasets (i. e. , PROMISE, REFUGE, and RETA). Extensive experiments showed that our proposed method achieved better segmentation performance than the UAMT, along with the average Dice score (DSC) of 0. 7990 for three different objects, and can compete with several existing semi-supervised methods (i. e. , HCMT, SASSNet, and DTC).

AAAI Conference 2026 Conference Paper

SGPFeat: Semantic and Geometric Priors for Multi-modal Image Matching

• Yuxin Deng
• Botian Wang
• Kaining Zhang
• Hao Zhang
• Jiayi Ma

Multi-modal image matching is a fundamental task in multi-view and multi-modal image processing. Its key challenge lies in extracting features that remain consistent despite drastic appearance variations across modalities. However, the learning of the feature is hindered by the scarcity and the inaccurate alignment of existing multi-modal datasets. To address this, we propose a knowledge distillation framework termed SGPFeat that transfers rich prior knowledge from large-scale unimodal tasks to enhance multi-modal representation learning. Specifically, semantic priors from a vision foundation model guide the feature extractor to identify shared semantic structures across modalities, enabling better generalization under large appearance gaps. In parallel, geometric priors derived from accurately aligned visible-light datasets improve detection precision on noisy aligned multi-modal pairs. Furthermore, we introduce a Heterogeneous Feature Aggregation (HFA) module to facilitate effective distillation and feature representation. Extensive experiments demonstrate that semantic and geometric priors bring significant improvement for our SGPFeat across diverse multi-modal image matching benchmarks.

AAAI Conference 2026 Conference Paper

Stop Mixing Things Up! BISCUIT Teaches Vision-Language Models to Learn New Concepts from Images on the Spot

• Jiahua Bao
• Siyao Cheng
• Jiaxing Du
• Yuhang Jia
• Boyang Niu
• Zeming Lang
• Changjiang He
• Hao Zhang

Vision-Language Models (VLMs) have achieved impressive performance across various tasks, but often struggle to apply newly introduced visual concepts during inference. A common failure pattern is what we call Mixing Things Up: VLMs frequently confuse concept names, resulting in vague descriptions and failure to ground the concept correctly. Existing approaches mainly address person-related concepts through text prompts or tokenizer modifications. However, VLMs still miss or misinterpret untrained visual concepts, underscoring the need to learn new concepts directly from visual input, without relying on prior textual injection. To overcome these limitations, we propose BISCUIT (Basis-aligned Inference through Structured Concept Unification and Identification-aware Tuning), a two-step training method. Step I proposes a dual-stream structure-aware vision encoder that fuses RGB and edge-based embeddings within a shared basis space to enhance concept recognition. Step II enhances generation quality through identification-aware tuning, which encourages alignment between the generated text and the newly introduced visual concepts. Existing methods mainly focus on person concepts and lack comprehensive evaluation across diverse visual categories. We further propose a benchmark BiscuitVQA to evaluate VLMs performance on recognizing and applying novel image-introduced concepts across diverse concept types and task types, including real people, cartoons, animals, and symbolic content. We apply BISCUIT to LLaVA-1.5 and Qwen2.5-VL, achieving competitive results among open-source models and narrowing the gap to Gemini-2.5 and GPT-4o. Interestingly, our BISCUIT maintains strong generalization, showing minimal degradation on other downstream tasks.

EAAI Journal 2026 Journal Article

Uncertainty- and hardness-weighted loss functions for medical image segmentation

• Yanyan Zheng
• Yabo Wu
• Jie Chen
• Xiaoguo Yang
• Hao Zhang
• Quanyong Yi
• Jiantao Pu
• Lei Wang

Accurate segmentation of medical images is essential for various image processing tasks and is now predominantly achieved using deep learning techniques. However, existing approaches often employ loss functions that fail to account for pixel-level differences in prediction uncertainty or hardness. This limitation frequently results in relatively large segmentation errors, particularly in object boundary regions. To address the limitation, we developed a novel class of uncertainty-/hardness-weighted loss functions by introducing two distinct pixel-wise weighting schemes: probability-guided uncertainty (PGU) and region-enhanced hardness (REH) weights. These weights, derived from the differences between network predictions and their corresponding ground truths, were designed to emphasize challenging pixels while reducing segmentation uncertainties. We validated these loss functions by integrating them with two classical neural networks, i. e. , Swin Transformer based U-shape network (Swin-Unet) and V-shape network (V-Net) to segment two- and three-dimensional target objects across four different images datasets, including Retinal Fundus Glaucoma Challenge (REFUGE) dataset, Retinal Vascular Tree Analysis (RETA) dataset, optical coherence tomography (OCT) dataset, and Atria Segmentation Challenge (ASC) dataset. Extensive experiments demonstrated that our developed loss functions outperformed classical losses, such as cross-entropy (CE) and Dice losses, along with their variants, highlighting the effectiveness and generalization of the introduced weighting schemes. The source code is available at https: //github. com/wmuLei/uhLoss.

EAAI Journal 2025 Journal Article

A streaming variable neural speech codec

• Huaifeng Zhang
• Pengfei Wu
• Guigeng Li
• Yuan An
• Hao Zhang

This paper presents a variable bit rate streaming neural speech codec designed for ultra-low bit rate scenarios, based on the SoundStream network framework. The codec employs the vector quantized variational auto-encoder (VQ-VAE) algorithm to capture the temporal structure and spectral characteristics of the speech signal, and constructs a latent space codebook to facilitate the effective mapping of feature vectors to discrete vectors. Based on the harmonic characteristics of speech signals and the inherent defects of single-scale discriminators, we introduce multi-period discriminators and multi-scale discriminators. The training process uses a balanced training strategy to ensure the balance between codebook utilization and training weights, and utilizes the Short-Time Fourier Transform (STFT) spectrum that can provide more accurate time–frequency resolution to compute the reconstruction loss. We introduce codebook loss to improve the utilization rate of the codebook and accelerate the convergence of the model. In the inference process, we use a quantizer selection strategy to achieve adaptive adjustment of variable bitrate. Objective and subjective experiments demonstrate that our proposed new neural speech codec outperforms traditional classical speech codecs and existing neural speech codecs in terms of reconstructed speech naturalness and quality while maintaining the low latency characteristic of neural speech codecs. With a multi-stimulus test with hidden reference and anchor (MUSHRA) score of 87, it is highly suitable for ultra-low bit rate speech compression applications such as satellite speech communication and narrowband instant messaging. The demo has been publicly released at https: //svcodec. github. io/.

AAAI Conference 2025 Conference Paper

Advancing Comprehensive Aesthetic Insight with Multi-Scale Text-Guided Self-Supervised Learning

• Yuti Liu
• Shice Liu
• Junyuan Gao
• Peng-tao Jiang
• Hao Zhang
• Jinwei Chen
• Bo Li

Image Aesthetic Assessment (IAA) is a vital and intricate task that entails analyzing and assessing an image's aesthetic values, and identifying its highlights and areas for improvement. Traditional methods of IAA often concentrate on a single aesthetic task and suffer from inadequate labeled datasets, thus impairing in-depth aesthetic comprehension. Despite efforts to overcome this challenge through the application of Multi-modal Large Language Models (MLLMs), such models remain underdeveloped for IAA purposes. To address this, we propose a comprehensive aesthetic MLLM capable of nuanced aesthetic insight. Central to our approach is an innovative multi-scale text-guided self-supervised learning technique. This technique features a multi-scale feature alignment module and capitalizes on a wealth of unlabeled data in a self-supervised manner to structurally and functionally enhance aesthetic ability. The empirical evidence indicates that accompanied with extensive instruct-tuning, our model sets new state-of-the-art benchmarks across multiple tasks, including aesthetic scoring, aesthetic commenting, and personalized image aesthetic assessment. Remarkably, it also demonstrates zero-shot learning capabilities in the emerging task of aesthetic suggesting. Furthermore, for personalized image aesthetic assessment, we harness the potential of in-context learning and showcase its inherent advantages.

NeurIPS Conference 2025 Conference Paper

Alleviating Hallucinations in Large Language Models through Multi-Model Contrastive Decoding and Dynamic Hallucination Detection

• Chenyu Zhu
• Yefeng Liu
• Hao Zhang
• Aowen Wang
• Guanhua Chen
• Longyue Wang
• Weihua Luo
• Kaifu Zhang

Despite their outstanding performance in numerous applications, large language models (LLMs) remain prone to hallucinations, generating content inconsistent with their pretraining corpora. Currently, almost all contrastive decoding approaches alleviate hallucinations by introducing a model susceptible to hallucinations and appropriately widening the contrastive logits gap between hallucinatory tokens and target tokens. However, although existing contrastive decoding methods mitigate hallucinations, they lack enough confidence in the factual accuracy of the generated content. In this work, we propose Multi-Model Contrastive Decoding (MCD), which integrates a pretrained language model with an evil model and a truthful model for contrastive decoding. Intuitively, a token is assigned a high probability only when deemed potentially hallucinatory by the evil model while being considered factual by the truthful model. This decoding strategy significantly enhances the model’s confidence in its generated responses and reduces potential hallucinations. Furthermore, we introduce a dynamic hallucination detection mechanism that facilitates token-by-token identification of hallucinations during generation and a tree-based revision mechanism to diminish hallucinations further. Extensive experimental evaluations demonstrate that our MCD strategy effectively reduces hallucinations in LLMs and outperforms state-of-the-art methods across various benchmarks.

EAAI Journal 2025 Journal Article

An adaptive integrated learning-based virtual sensing framework for temperature prediction in aircraft brake monitoring engineering

• Lin Lin
• Yin Chen
• Song Fu
• Hao Zhang
• Jinlei Wu

In commercial aviation, effective deceleration relies on accurate brake temperature trends. Traditional thermocouple sensors exhibit discontinuous responses, while machine learning-based methods often struggle with abnormal fluctuations due to high-dimensional signal overlap during low-speed braking. This study proposes an Adaptive Thermal Damping Integrated (ATDI) virtual sensor based on integrated learning to enhance brake temperature monitoring accuracy and address these challenges. A novel feature selection method is proposed that includes extracting baseline variables from five brake-related physical models and identifying significant latent variables using Shapley values, which are calculated by integrating three correlation indices based on linear, nonlinear, and rank consistency differences from aircraft flight records. The ATDI framework employs a time-lagged loss function and self-attention mechanism for dynamic weight assignment in the feature space, capturing critical temporal correlations to determine the thermal trend. A digital approach, adaptive to aircraft kinetic energy and called virtual thermal damping, is incorporated at the end of the prediction to mitigate fluctuations and ensure compliance with thermal conduction principles in braking systems. Experimental results demonstrate that the ATDI framework outperforms comparison networks in temperature value continuity and anomaly fluctuation suppression, especially with Long Short-Term Memory (LSTM) as the meta-learner, across three evaluation metrics. Additionally, application strategies for the ATDI framework in aircraft operation and maintenance are proposed.

AAAI Conference 2025 Conference Paper

Boosting Vision State Space Model with Fractal Scanning

• Haoke Xiao
• Lv Tang
• Peng-tao Jiang
• Hao Zhang
• Jinwei Chen
• Bo Li

Recently, foundational models have significantly advanced in different tasks, accompanied by Transformer as the general backbone. However, Transformer's quadratic complexity poses challenges for handling longer sequences and higher resolution images, which may limit foundational models further development. To alleviate this issue, various efficient State Space Models (SSMs) like Mamba have emerged, initially matching Transformer performance and gradually surpassing it. To improve the performance of SSMs in computer vision tasks, one crucial viewpoint is effective serialization of images. Existing vision Mambas, which rely on a linear scanning mechanism, often struggle to capture complex spatial relationships in 2D images. This results in feature loss during serialization and negatively impacts model performance. To overcome this limitation, we propose the use of fractal scanning curves for image serialization to enhance the Mambas’ ability to accurately model complex spatial dependencies. Additionally, unlike existing vision Mambas, which are designed with various curve scanning directions that increase the complexity, contradicting the original intent of Mamba to enhance model performance. We novelty introduce the Fractal Fusion Pathway (FFP) for our FractalMamba, which can enhance its performance efficiently. Extensive experiments underscore the superiority of our proposed FractalMamba.

AAAI Conference 2025 Conference Paper

Cross-Modal Stealth: A Coarse-to-Fine Attack Framework for RGB-T Tracker

• Xinyu Xiang
• Qinglong Yan
• Hao Zhang
• Jianfeng Ding
• Han Xu
• Zhongyuan Wang
• Jiayi Ma

Current research on adversarial attacks mainly focuses on RGB trackers, with no existing methods for attacking RGB-T cross-modal trackers. To fill this gap and overcome its challenges, we propose a progressive adversarial patch generation framework and achieve cross-modal stealth. On the one hand, we design a coarse-to-fine architecture grounded in the latent space to progressively and precisely uncover the vulnerabilities of RGB-T trackers. On the other hand, we introduce a correlation-breaking loss that disrupts the modal coupling within trackers, spanning from the pixel to the semantic level. These two design elements ensure that the proposed method can overcome the obstacles posed by cross-modal information complementarity in implementing attacks. Furthermore, to enhance the reliable application of the adversarial patches in real world, we develop a point tracking-based reprojection strategy that effectively mitigates performance degradation caused by multi-angle distortion during imaging. Extensive experiments demonstrate the superiority of our method.

JBHI Journal 2025 Journal Article

Cyclic Contrastive Representation Learning for Incomplete Multi-modal Medical Image Segmentation

• Shihuan He
• Zongbao Yang
• Jianbo Zhao
• Hao Zhang
• Ruxin Wang

Accurate segmentation of multimodal medical images with missing modalities remains a critical challenge due to incomplete data often encountered in clinical practice. Lack of modality-specific information often leads to significant performance degradation in scenarios with severely missing modalities. To address this problem, we focus on modeling the relationships between modality-specific features. We propose a joint representation learning framework, named as Cyclic Contrastive Latent Representation Segmentation (CLRS), which incorporates cyclic modality-specific representation generation and contrastive feature alignment for robust 3D medical image segmentation under missing modality conditions. CLRS first extracts feature from available modalities using a unified encoder, then generates missing latent representations conditioned on the encoded features via an elaborately designed synthesis strategy. Meanwhile, a channel-wise attention mechanism is introduced to enhance the specific features of the modality. In addition, modality-specific contrastive learning enforces cross-modal discrimination between the generated and encoded representations, which effectively disentangles modality-specific information from shared patterns and enhances the segmentation robustness in missing modality scenarios. Extensive experiments on three 3D multimodal datasets demonstrate the superior performance of CLRS, particularly in scenarios with severe modality absence. For instance, with only a single modality available on ProstateZS dataset, CLRS improves the state-of-the-art (SOTA) by over 4. 06% for peripheral zone, 2. 20% for central gland. The code is available at https://github.com/comphsh/CLRS.

ICRA Conference 2025 Conference Paper

Depth Estimation Based on 3D Gaussian Splatting Siamese Defocus

• Jinchang Zhang
• Ningning Xu
• Hao Zhang
• Guoyu Lu 0001

Depth estimation is a fundamental task in 3D geometry. While stereo depth estimation can be achieved through triangulation methods, it is not as straightforward for monocular methods, which require the integration of global and local information. The Depth from Defocus (DFD) method utilizes camera lens models and parameters to recover depth information from blurred images and has been proven to perform well. However, these methods rely on All-In-Focus (AIF) images for depth estimation, which is nearly impossible to obtain in real-world applications. To address this issue, we propose a self-supervised framework based on 3D Gaussian splatting and Siamese networks. By learning the blur levels at different focal distances of the same scene in the focal stack, the framework predicts the defocus map and Circle of Confusion (CoC) from a single defocused image, using the defocus map as input to DepthNet for monocular depth estimation. The 3D Gaussian splatting model renders defocused images using the predicted CoC, and the differences between these and the real defocused images provide additional supervision signals for the Siamese Defocus self-supervised network. This framework has been validated on both artificially synthesized and real blurred datasets. Subsequent quantitative and visualization experiments demonstrate that our proposed framework is highly effective as a DFD method.

IJCAI Conference 2025 Conference Paper

Efficient Constraint-based Window Causal Graph Discovery in Time Series with Multiple Time Lags

• Yewei Xia
• Yixin Ren
• Hong Cheng
• Hao Zhang
• Jihong Guan
• Minchuan Xu
• Shuigeng Zhou

We address the identification of direct causes in time series with multiple time lags, and propose a constraint-based window causal graph discovery method. A key advantage of our method is that the number of required conditional independence (CI) tests scales quadratically with the number of sub-series. The method first uses CI tests to find the minimum trek lag between two arbitrary sub-series, followed by designing an efficient CI testing strategy to identify the direct causes between them. We show that the method is both sound and complete under some graph constraints. We compare the proposed method with typical baselines on various datasets. Experimental results show that our method outperforms all the counterparts in both accuracy and running speed.

EAAI Journal 2025 Journal Article

Efficient kriging-based wall deflection prediction in braced excavation considering model and measurement errors

• Xiong Xiao
• Quanwang Li
• Hao Zhang

The deflection prediction of diaphragm walls stands as a critical aspect of safety management for excavation construction. Bayesian updating holds a prominent position among several existing prediction methods, owing to its probabilistic predictions and the ability to simultaneously consider prior knowledge and observational data. This paper develops a comprehensive Bayesian updating framework for wall deflection prediction in braced excavation, taking into account both model error associated with the prediction model chosen and measurement error. Further, both of these errors typically exhibit depth dependency and spatial correlation and thus are modeled as stochastic processes. An innovative efficient Kriging (e-Kriging) surrogate model and an Expectation-Maximum algorithm based adaptive importance sampling (AIS) method are proposed to improve computational efficiency. The prediction performance of this proposed framework is verified via an exhaustively reported Taipei National Enterprise Center (TNEC) excavation project. Moreover, this paper compares the influences of different selections of prior distributions and measurement data on the predictions. Results suggest that using only a subset of the middle portion of measurement data for updating is often a more appropriate choice.

NeurIPS Conference 2025 Conference Paper

Efficiently Scaling LLM Reasoning Programs with Certaindex

• Yichao Fu
• Junda Chen
• Siqi Zhu
• Fu Fu
• Zhongdongming Dai
• Yonghao Zhuang
• Yian Ma
• Aurick Qiao

Test-time reasoning algorithms such as chain-of-thought, self-consistency, and MCTS enhance LLM problem-solving but can wastefully generate many tokens without improving accuracy. At the same time, we observe that these algorithms exhibit answer stabilization: their intermediate solutions often cease to change after a certain point, and further investment of compute does not change their final answer. To quantify this phenomenon, we introduce Certaindex, an algorithm-agnostic metric measuring this evolving stability, signaling when further computation is unlikely to alter the final result. Certaindex is lightweight, can accelerate reasoning program inference via early exit, and further enables dynamic token allocation, gang scheduling, and many opportunities when integrated with real-world LLM serving systems. To quantify real-world benefits, we built Certaindex as a scheduler into Dynasor, our reasoning-aware LLM serving system, and demonstrate up to 50\% compute savings and 3. 3$\times$ higher throughput in real workloads with no accuracy drop. Our code is available at https: //github. com/hao-ai-lab/Dynasor. git

NeurIPS Conference 2025 Conference Paper

Faster Video Diffusion with Trainable Sparse Attention

• Peiyuan Zhang
• Yongqi Chen
• Haofeng Huang
• Will Lin
• Zhengzhong Liu
• Ion Stoica
• Eric Xing
• Hao Zhang

Scaling video diffusion transformers (DiTs) is limited by their quadratic 3D attention, even though most of the attention mass concentrates on a small subset of positions. We turn this observation into VSA, a trainable, hardware-efficient sparse attention that replaces full attention at both training and inference. In VSA, a lightweight coarse stage pools tokens into tiles and identifies high-weight critical tokens; a fine stage computes token-level attention only inside those tiles subjecting to block computing layout to ensure hard efficiency. This leads to a single differentiable kernel that trains end-to-end, requires no post-hoc profiling, and sustains 85\% of FlashAttention3 MFU. We perform a large sweep of ablation studies and scaling-law experiments by pretraining DiTs from 60M to 1. 4B parameters. VSA reaches a Pareto point that cuts training FLOPS by 2. 53$\times$ with no drop in diffusion loss. Retrofitting the open-source Wan2. 1-1. 3B model speeds up attention time by 6$\times$ and lowers end-to-end generation time from 31s to 18s with comparable quality, while for the 14B model, end-to-end generation time is reduced from 1274s to 576s. Furthermore, we introduce a preliminary study of Sparse-Distill, the first method to enable sparse attention and distillation concurrently, achieving 50. 9x speed up for Wan-1. 3B while maintaining quality. These results establish trainable sparse attention as a practical alternative to full attention and a key enabler for further scaling of video diffusion models. Code is available at https: //github. com/hao-ai-lab/FastVideo.

ICML Conference 2025 Conference Paper

FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates

• Xinyi Lu
• Hao Zhang
• Chenglin Li
• Weijia Lu
• Zhifei Yang 0005
• Wenrui Dai
• Xiaodong Zhang
• Xiaofeng Ma

The significant communication overhead and client data heterogeneity have posed an important challenge to current federated learning (FL) paradigm. Existing compression-based and optimization-based FL algorithms typically focus on addressing either the model compression challenge or the data heterogeneity issue individually, rather than tackling both of them. In this paper, we observe that by symbolizing the client model updates to be uploaded (i. e. , normalizing the magnitude for each model parameter at local clients), the model heterogeneity, essentially stemmed from data heterogeneity, can be mitigated, and thereby helping improve the overall generalization performance of the globally aggregated model at the server. Inspired with this observation, and further motivated by the success of Lion optimizer in achieving the optimal performance on most tasks in the centralized learning, we propose a new FL algorithm, called FedSMU, which simultaneously reduces the communication overhead and alleviates the data heterogeneity issue. Specifically, FedSMU splits the standard Lion optimizer into the local updates and global execution, where only the symbol of client model updates commutes between the client and server. We theoretically prove the convergence of FedSMU for the general non-convex settings. Through extensive experimental evaluations on several benchmark datasets, we demonstrate that our FedSMU algorithm not only reduces the communication overhead, but also achieves a better generalization performance than the other compression-based and optimization-based baselines.

ICLR Conference 2025 Conference Paper

Frame-Voyager: Learning to Query Frames for Video Large Language Models

• Sicheng Yu
• Chengkai Jin
• Huanyu Wang
• Zhenghao Chen
• Sheng Jin
• Zhongrong Zuo
• Xiaolei Xu
• Zhenbang Sun

Video Large Language Models (Video-LLMs) have made remarkable progress in video understanding tasks. However, they are constrained by the maximum length of input tokens, making it impractical to input entire videos. Existing frame selection approaches, such as uniform frame sampling and text-frame retrieval, fail to account for the information density variations in the videos or the complex instructions in the tasks, leading to sub-optimal performance. In this paper, we propose Frame-Voyager that learns to query informative frame combinations, based on the given textual queries in the task. To train Frame-Voyager, we introduce a new data collection and labeling pipeline, by ranking frame combinations using a pre-trained Video-LLM. Given a video of M frames, we traverse its T-frame combinations, feed them into a Video-LLM, and rank them based on Video-LLM's prediction losses. Using this ranking as supervision, we train Frame-Voyager to query the frame combinations with lower losses. In experiments, we evaluate Frame-Voyager on four Video Question Answering benchmarks by plugging it into two different Video-LLMs. The experimental results demonstrate that Frame-Voyager achieves impressive results in all settings, highlighting its potential as a plug-and-play solution for Video-LLMs.

ICRA Conference 2025 Conference Paper

Geometry and Force-Informed Robotic Assembly with Small Relative Initial Deviations for Circular Electrical Connectors

• Zhenyu Wang
• Xiangfei Li
• Huan Zhao 0001
• Lingjun Shao
• Hao Zhang
• Han Ding 0001

Circular electrical connectors (CECs) have a wide range of applications in scenarios that require reliable connections. However, sockets are often located in narrow scenes with random spatial orientations, complex lighting conditions, and obstructions from cables, making it difficult to accurately locate them through cameras. Besides, due to the complex geometric structure of CECs and the presence of electrode protection slots, the existing research on the assembly of cylindrical or polygonal pegs and holes may not be applicable to the assembly of such components. To this end, this article proposes a novel robotic assembly strategy for CECs with small relative initial deviations, whose core is to design a search trajectory and heuristic force strategy to perceive force/pose (F/P) discontinuity characteristics under different geometric constraints. This assembly strategy is independent of the CEC's size and is not affected by the socket's spatial orientation. The experiments with two different sizes of CECs on a robot equipped with a 6-dimensional force/torque ( $\mathbf{F} / \mathbf{T}$ ) sensor are conducted, and the effectiveness and robustness of the proposed assembly strategy for CECs are demonstrated.

IROS Conference 2025 Conference Paper

HEATS: A Hierarchical Framework for Efficient Autonomous Target Search with Mobile Manipulators

• Hao Zhang
• Yifei Wang
• Weifan Zhang
• Yu Wang
• Haoyao Chen

Utilizing robots for autonomous target search in complex and unknown environments can greatly improve the efficiency of search and rescue missions. However, existing methods have shown inadequate performance due to hardware platform limitations, inefficient viewpoint selection strategies, and conservative motion planning. In this work, we propose HEATS, which enhances the search capability of mobile manipulators in complex and unknown environments. We design a target viewpoint planner tailored to the strengths of mobile manipulators, ensuring efficient and comprehensive viewpoint planning. Supported by this, a whole-body motion planner integrates global path search with local IPC optimization, enabling the mobile manipulator to safely and agilely visit target viewpoints, significantly improving search performance. We present extensive simulated and real-world tests, in which our method demonstrates reduced search time, higher target search completeness, and lower movement cost compared to classic and state-of-the-art approaches. Our method will be open-sourced for community benefit 3.

IJCAI Conference 2025 Conference Paper

Identifying Causal Mechanism Shifts Under Additive Models with Arbitrary Noise

• Yewei Xia
• Xueliang Cui
• Hao Zhang
• Yixin Ren
• Feng Xie
• Jihong Guan
• Ruxin Wang
• Shuigeng Zhou

In many real-world scenarios, the goal is to identify variables whose causal mechanisms change across related datasets. For example, detecting abnormal root nodes in manufacturing, and identifying key genes that influence cancer by analyzing differences in gene regulatory mechanisms between healthy individuals and cancer patients. This can be done by recovering the causal structure for each dataset independently and then comparing them to identify differences, but the performance is often suboptimal. Typically, existing methods directly identify causal mechanism shifts based on linear additive noise models (ANMs) or by imposing restrictive assumptions on the noise distribution. In this paper, we introduce CMSI, a novel and more general algorithm based on nonlinear ANMs that identifies variables with shifting causal mechanisms under arbitrary noise distributions. Evaluated on various synthetic datasets, CMSI consistently outperforms existing baselines in terms of F1 score. Additionally, we demonstrate CMSI's applicability on gene expression datasets of ovarian cancer patients at different disease stages.

ICML Conference 2025 Conference Paper

LBI-FL: Low-Bit Integerized Federated Learning with Temporally Dynamic Bit-Width Allocation

• Li Ding
• Hao Zhang
• Wenrui Dai
• Chenglin Li
• Weijia Lu
• Zhifei Yang 0005
• Xiaodong Zhang
• Xiaofeng Ma

Federated learning (FL) is greatly challenged by the communication bottleneck and computation limitation on clients. Existing methods based on quantization for FL cannot simultaneously reduce the uplink and downlink communication cost and mitigate the computation burden on clients. To address this problem, in this paper, we propose the first low-bit integerized federated learning (LBI-FL) framework that quantizes the weights, activations, and gradients to lower than INT8 precision to evidently reduce the communication and computational costs. Specifically, we achieve dynamical temporal bit-width allocation for weights, activations, and gradients along the training trajectory via reinforcement learning. An agent is trained to determine bit-width allocation by comprehensively considering the states like current bit-width, training stage, and quantization loss as the state. The agent efficiently trained on small-scale datasets can be well generalized to train varying network architectures on non-independent and identically distributed datasets. Furthermore, we demonstrated in theory that federated learning with gradient quantization achieves an equivalent convergence rate to FedAvg. The proposed LBI-FL can reduce the communication costs by 8 times compared to full-precision FL. Extensive experiments show that the proposed LBI-FL achieves a reduction of more than 50% BitOPs per client on average for FL with less than 2% accuracy loss compared to low-bit training with INT8 precision.

EAAI Journal 2025 Journal Article

Learning to generate and evaluate fact-checking explanations with transformers

• Darius Feher
• Abdullah Khered
• Hao Zhang
• Riza Batista-Navarro
• Viktor Schlegel

In an era increasingly dominated by digital platforms, the spread of misinformation poses a significant challenge, highlighting the need for solutions capable of assessing information veracity. Our research contributes to the field of Explainable Artificial Antelligence (XAI) by developing transformer-based fact-checking models that contextualise and justify their decisions by generating human-accessible explanations. Importantly, we also develop models for automatic evaluation of explanations for fact-checking verdicts across different dimensions such as (self)-contradiction, hallucination, convincingness and overall quality. By introducing human-centred evaluation methods and developing specialised datasets, we emphasise the need for aligning Artificial Intelligence (AI)-generated explanations with human judgements. This approach not only advances theoretical knowledge in XAI but also holds practical implications by enhancing the transparency, reliability and users’ trust in AI-driven fact-checking systems. Furthermore, the development of our metric learning models is a first step towards potentially increasing efficiency and reducing reliance on extensive manual assessment. Based on experimental results, our best performing generative model achieved a Recall-Oriented Understudy for Gisting Evaluation-1 (ROUGE-1) score of 47. 77 demonstrating superior performance in generating fact-checking explanations, particularly when provided with high-quality evidence. Additionally, the best performing metric learning model showed a moderately strong correlation with human judgements on objective dimensions such as (self)-contradiction and hallucination, achieving a Matthews Correlation Coefficient (MCC) of around 0. 7.

TMLR Journal 2025 Journal Article

LLaVA-OneVision: Easy Visual Task Transfer

• Bo Li
• Yuanhan Zhang
• Dong Guo
• Renrui Zhang
• Feng Li
• Hao Zhang
• Kaichen Zhang
• Peiyuan Zhang

We present LLaVA-OneVision, a family of open large multimodal models (LMMs) developed by consolidating our insights into data, models, and visual representations in the LLaVA-NeXT blog series. Our experimental results demonstrate that LLaVA-OneVision is the first single model that can simultaneously push the performance boundaries of open LMMs in three important computer vision scenarios: single-image, multi-image, and video scenarios. Importantly, the design of LLaVA-OneVision allows strong transfer learning across different modalities/scenarios, yielding new emerging capabilities. In particular, strong video understanding and cross-scenario capabilities are demonstrated through task transfer from images to videos.

NeurIPS Conference 2025 Conference Paper

Local Learning for Covariate Selection in Nonparametric Causal Effect Estimation with Latent Variables

• Zheng Li
• Xichen Guo
• Feng Xie
• Yan Zeng
• Hao Zhang
• Zhi Geng

Estimating causal effects from nonexperimental data is a fundamental problem in many fields of science. A key component of this task is selecting an appropriate set of covariates for confounding adjustment to avoid bias. Most existing methods for covariate selection often assume the absence of latent variables and rely on learning the global causal structure among variables. However, identifying the global structure can be unnecessary and inefficient, especially when our primary interest lies in estimating the effect of a treatment variable on an outcome variable. To address this limitation, we propose a novel local learning approach for covariate selection in nonparametric causal effect estimation, which accounts for the presence of latent variables. Our approach leverages testable independence and dependence relationships among observed variables to identify a valid adjustment set for a target causal relationship, ensuring both soundness and completeness under standard assumptions. We validate the effectiveness of our algorithm through extensive experiments on both synthetic and real-world data.

TMLR Journal 2025 Journal Article

MagicPose4D: Crafting Articulated Models with Appearance and Motion Control

• Hao Zhang
• Di Chang
• Fang Li
• Mohammad Soleymani
• Narendra Ahuja

With the success of 2D and 3D visual generative models, there is growing interest in generating 4D content. Existing methods primarily rely on text prompts to produce 4D content, but they often fall short of accurately defining complex or rare motions. To address this limitation, we propose MagicPose4D, a novel framework for refined control over both appearance and motion in 4D generation. Unlike current 4D generation methods, MagicPose4D accepts monocular videos or mesh sequences as motion prompts, enabling precise and customizable motion control. MagicPose4D comprises two key modules: (i) Dual-Phase 4D Reconstruction Module which operates in two phases. The first phase focuses on capturing the model's shape using accurate 2D supervision and less accurate but geometrically informative 3D pseudo-supervision without imposing skeleton constraints. The second phase extracts the 3D motion (skeleton poses) using more accurate pseudo-3D supervision, obtained in the first phase, and introduces kinematic chain-based skeleton constraints to ensure physical plausibility. Additionally, we propose a Global-local Chamfer loss that aligns the overall distribution of predicted mesh vertices with the supervision while maintaining part-level alignment without extra annotations. (ii) Cross-category Motion Transfer Module leverages the extracted motion from the 4D reconstruction module and uses a kinematic-chain-based skeleton to achieve cross-category motion transfer. It ensures smooth transitions between frames through dynamic rigidity, facilitating robust generalization without additional training. Through extensive experiments, we demonstrate that MagicPose4D significantly improves the accuracy and consistency of 4D content generation, outperforming existing methods in various benchmarks.

ICLR Conference 2025 Conference Paper

MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection

• Bokai Lin
• Zihao Zeng
• Zipeng Xiao
• Siqi Kou
• TianQi Hou
• Xiaofeng Gao 0001
• Hao Zhang
• Zhijie Deng

KV cache has become a *de facto* technique for the inference of large language models (LLMs), where tensors of shape (layer number, head number, sequence length, feature dimension) are introduced to cache historical information for self-attention. As the size of the model and data grows, the KV cache can, yet, quickly become a bottleneck within the system in both storage and memory transfer. To address this, prior studies usually focus on the first three axes of the cache tensors for compression. This paper supplements them, focusing on the feature dimension axis, by utilizing low-rank projection matrices to transform the cache features into spaces with reduced dimensions. We begin by investigating the canonical orthogonal projection method for data compression through principal component analysis (PCA). We identify the drawback of PCA projection that model performance degrades rapidly under relatively low compression rates (less than 60%). This phenomenon is elucidated by insights derived from the principles of attention mechanisms. To bridge the gap, we propose to directly tune the orthogonal projection matrix on the continual pre-training or supervised fine-tuning datasets with an elaborate Matryoshka learning strategy. Thanks to such a strategy, we can adaptively search for the optimal compression rates for various layers and heads given varying compression budgets. Compared to Multi-head Latent Attention (MLA), our method can easily embrace pre-trained LLMs and hold a smooth tradeoff between performance and compression rate. We witness the high data efficiency of our training procedure and find that our method can sustain over 90\% performance with an average KV cache compression rate of 60% (and up to 75% in certain extreme scenarios) for popular LLMs like LLaMA2 and Mistral.

NeurIPS Conference 2025 Conference Paper

MoPFormer: Motion-Primitive Transformer for Wearable-Sensor Activity Recognition

• Hao Zhang
• Zhan Zhuang
• Xuehao Wang
• Xiaodong Yang
• Yu Zhang

Human Activity Recognition (HAR) with wearable sensors is challenged by limited interpretability, which significantly impacts cross-dataset generalization. To address this challenge, we propose Motion-Primitive Transformer (MoPFormer), a novel self-supervised framework that enhances interpretability by tokenizing inertial measurement unit signals into semantically meaningful motion primitives and leverages a Transformer architecture to learn rich temporal representations. MoPFormer comprises two stages. The first stage is to partition multi-channel sensor streams into short segments and quantize them into discrete ``motion primitive'' codewords, while the second stage enriches those tokenized sequences through a context-aware embedding module and then processes them with a Transformer encoder. The proposed MoPFormer can be pre-trained using a masked motion-modeling objective that reconstructs missing primitives, enabling it to develop robust representations across diverse sensor configurations. Experiments on six HAR benchmarks demonstrate that MoPFormer not only outperforms state-of-the-art methods but also successfully generalizes across multiple datasets. More importantly, the learned motion primitives significantly enhance both interpretability and cross-dataset performance by capturing fundamental movement patterns that remain consistent across similar activities, regardless of dataset origin.

EAAI Journal 2025 Journal Article

Multidimensional chaotic signals generation using deep learning and its application in image encryption

• Shuang Zhou
• Zhiji Tao
• Uğur Erkan
• Abdurrahim Toktas
• Herbert Ho-Ching Iu
• Yingqian Zhang
• Hao Zhang

In this paper, we propose a novel artificial intelligence implemented approach to generate multi-dimensional chaotic signals using the Long- and Short-Term Time-Series Network (LSTNet) for a newly contrived Two-Stage pixel/bit level Scrambling and Dynamic Diffusion (TSSDD) color image encryption. Initially, we employ the hyperchaotic Lorenz and Chen chaotic systems to produce chaotic signals. Subsequently, the LSTNet model is trained to predict these produced multi-dimensional chaotic sequences and then it generates new multi-dimensional chaotic signals. Through analysis involving phase diagrams, largest Lyapunov exponent (LE), 0–1 test, Permutation Entropy (PE), Sample Entropy (SE), Correlation Dimension (CD) and National Institute of Standards and Technology (NIST), we observe that these applied artificial intelligence signals exhibit high chaotic states and randomness. Finally, we apply these signals to demonstrate the proposed TSSDD color image encryption wherein simulation experiments indicate competitive performance against common attacks.

IROS Conference 2025 Conference Paper

Non-Overlap-Aware Egocentric Pose Estimation for Collaborative Perception in Connected Autonomy

• Hong Huang
• Dongkuan Xu
• Hao Zhang
• Peng Gao

Egocentric pose estimation is a fundamental capability for multi-robot collaborative perception in connected autonomy, such as connected autonomous vehicles. During multi-robot operations, a robot needs to know the relative pose between itself and its teammates with respect to its own coordinates. However, different robots usually observe completely different views that contains similar objects, which leads to wrong pose estimation. In addition, it is unrealistic to allow robots to share their raw observations to detect overlap due to the limited communication bandwidth constraint. In this paper, we introduce a novel method for Non-Overlap-Aware Egocentric Pose Estimation (NOPE), which performs egocentric pose estimation in a multi-robot team while identifying the non-overlap views and satifying the communication bandwidth constraint. NOPE is built upon an unified hierarchical learning framework that integrates two levels of robot learning: (1) high-level deep graph matching for correspondence identification, which allows to identify if two views are overlapping or not, (2) low-level position-aware cross-attention graph learning for egocentric pose estimation. To evaluate NOPE, we conduct extensive experiments in both high-fidelity simulation and real-world scenarios. Experimental results have demonstrated that NOPE enables the novel capability for non-overlapping-aware egocentric pose estimation and achieves state-of-art performance compared with the existing methods.

ICLR Conference 2025 Conference Paper

Palmbench: a comprehensive Benchmark of Compressed Large Language Models on Mobile Platforms

• Yilong Li
• Jingyu Liu
• Hao Zhang
• M. Badri Narayanan
• Utkarsh Sharma
• Shuai Zhang
• Yijing Zeng
• Jayaram Raghuram

Deploying large language models (LLMs) locally on mobile devices is advantageous in scenarios where transmitting data to remote cloud servers is either undesirable due to privacy concerns or impractical due to network connection. Recent advancements have facilitated the local deployment of LLMs. However, local deployment also presents challenges, particularly in balancing quality (generative performance), latency, and throughput within the hardware constraints of mobile devices. In this paper, we introduce our lightweight, all-in-one automated benchmarking framework that allows users to evaluate LLMs on mobile devices. We provide a comprehensive benchmark of various popular LLMs with different quantization configurations (both weights and activations) across multiple mobile platforms with varying hardware capabilities. Unlike traditional benchmarks that assess full-scale models on high-end GPU clusters, we focus on evaluating resource efficiency (memory and power consumption) and harmful output for compressed models on mobile devices. Our key observations include: i) differences in energy efficiency and throughput across mobile platforms; ii) the impact of quantization on memory usage, GPU execution time, and power consumption; and iii) accuracy and performance degradation of quantized models compared to their non-quantized counterparts; and iv) the frequency of hallucinations and toxic content generated by compressed LLMs on mobile devices.

EAAI Journal 2025 Journal Article

Progressive epipolar geometry for robust light field super-resolution

• Hao Zhang
• Hao Sheng
• Rongshan Chen
• Da Yang
• Ruixuan Cong
• Zhenglong Cui
• Xuefei Huang
• Guanqun Su

Light field technology captures both spatial and angular information, densely sampled high-resolution light field images contain abundant 3-dimensional geometric information, enabling widespread applications in various industrial fields. However, existing methods for joint spatial and angular super-resolution of light field images face significant challenges when reconstructing scenes with large disparities and significant occlusions. To address this issue, we propose a geometric information enhancement module that enables efficient extraction and preservation of geometric information, and then we propose a non-disparity-based, one-stage approach that allows for the direct reconstruction of densely sampled high-resolution light field images from sparsely sampled low-resolution counterparts. Specifically, we decompose the original 4-dimensional light field image into three distinct 2-dimensional representations: spatial, angular, and epipolar plane image. Among these representations, the epipolar plane image contains abundant geometric information, which inspires us to propose a more efficient feature extractor. In addition, we introduce a progressive feature fusion strategy that better preserves geometric information extracted from epipolar plane images. Finally, to avoid errors introduced by warping operations that complicate the process of enhancing geometric information, we introduce a spatial-angular integrated upsampling module. Extensive experimental results on public datasets demonstrate that our proposed method significantly outperforms state-of-the-art approaches both quantitatively and qualitatively. Specifically, on the Occlusions dataset, our method achieved significant improvement in performance while reducing inference time by approximately 80% compared to the current best method. This efficiency gain is particularly beneficial for practical applications of the artificial intelligence algorithm.

NeurIPS Conference 2025 Conference Paper

Projection-Manifold Regularized Latent Diffusion for Robust General Image Fusion

• Lei Cao
• Hao Zhang
• Chunyu Li
• Jiayi Ma

This study proposes PDFuse, a robust, general training-free image fusion framework built on pre-trained latent diffusion models with projection–manifold regularization. By redefining fusion as a diffusion inference process constrained by multiple source images, PDFuse can adapt to varied image modalities and produce high-fidelity outputs utilizing the diffusion prior. To ensure both source consistency and full utilization of generative priors, we develop novel projection–manifold regularization, which consists of two core mechanisms. On the one hand, the Multi-source Information Consistency Projection (MICP) establishes a projection system between diffusion latent representations and source images, solved efficiently via conjugate gradients to inject multi-source information into the inference. On the other hand, the Latent Manifold-preservation Guidance (LMG) aligns the latent distribution of diffusion variables with that of the sources, guiding generation to respect the model’s manifold prior. By alternating these mechanisms, PDFuse strikes an optimal balance between fidelity and generative quality, achieving superior fusion performance across diverse tasks. Moreover, PDFuse constructs a canonical interference operator set. It synergistically incorporates it into the aforementioned dual mechanisms, effectively leveraging generative priors to address various degradation issues during the fusion process without requiring clean data for supervising training. Extensive experimental evidence substantiates that PDFuse achieves highly competitive performance across diverse image fusion tasks. The code is publicly available at https: //github. com/Leiii-Cao/PDFuse.

JBHI Journal 2025 Journal Article

Re-Visible Dual-Domain Self-Supervised Deep Unfolding Network for MRI Reconstruction

• Hao Zhang
• Qi Wang
• Jian Sun
• Zhijie Wen
• Jun Shi
• Shihui Ying

Magnetic Resonance Imaging (MRI) is widely used in clinical practice, but suffers from prolonged acquisition time. Although deep learning methods have been proposed to accelerate acquisition and demonstrate promising performance, they rely on high-quality fully-sampled datasets for training in a supervised manner. However, such datasets are time-consuming and expensive-to-collect, which constrains their broader applications. On the other hand, self-supervised methods offer an alternative by enabling learning from under-sampled data alone, but most existing methods rely on further partitioned under-sampled k-space data as model's input for training, which causes an input distribution shift between the the training stage and the inference stage. Additionally, their models have not effectively incorporated comprehensive image priors, leading to degraded reconstruction performance. In this paper, we propose a novel re-visible dual-domain self-supervised deep unfolding network to address these issues when only under-sampled datasets are available. Specifically, by incorporating re-visible dual-domain loss, all under-sampled k-space data are utilized during training to mitigate the input distribution shift caused by further partitioning. This design enables the model to implicitly adapt to all under-sampled k-space data as input. Additionally, we design a Deep Unfolding Network based on Chambolle and Pock Proximal Point Algorithm (DUN-CP-PPA) to achieve end-to-end reconstruction. By employing a Spatial-Frequency Feature Extraction (SFFE) block to capture both global and local representations, the model effectively integrates imaging physics with comprehensive image priors to enhance reconstruction performance. Experiments on both single-coil and multi-coil datasets demonstrate that our method outperforms state-of-the-art approaches in terms of reconstruction performance and generalization capability.

AIJ Journal 2025 Journal Article

Regression-based conditional independence test with adaptive kernels

• Yixin Ren
• Juncai Zhang
• Yewei Xia
• Ruxin Wang
• Feng Xie
• Jihong Guan
• Hao Zhang
• Shuigeng Zhou

We propose a novel framework for regression-based conditional independence (CI) test with adaptive kernels, where the task of CI test is reduced to regression and statistical independence test while proving that the test power of CI can be maximized by adaptively learning parameterized kernels of the independence test if the consistency of regression can be guaranteed. For the adaptively learning kernel of independence test, we first address the pitfall inherent in the existing signal-to-noise ratio criterion by modeling the change of the null distribution during the learning process, then design a new class of kernels that can adaptively focus on the significant dimensions of variables to judge independence, which makes the tests more flexible than using simple kernels that are adaptive only in length-scale, and especially suitable for high-dimensional complex data. Theoretically, we demonstrate the consistency of the proposed tests, and show that the non-convex objective function used for learning fits the L-smoothing condition, thus benefiting the optimization. Experimental results on both synthetic and real data show the superiority of our method. The source code and datasets are available at https: //github. com/hzsiat/AdaRCIT.

NeurIPS Conference 2025 Conference Paper

RESAnything: Attribute Prompting for Arbitrary Referring Segmentation

• Ruiqi Wang
• Hao Zhang

We present an open-vocabulary and zero-shot method for arbitrary referring expression segmentation (RES), targeting input expressions that are more general than what prior works were designed to handle. Specifically, our inputs encompass both object- and part-level labels as well as implicit references pointing to properties or qualities of object/part function, design, style, material, etc. Our model, coined RESAnything, leverages Chain-of-Thoughts (CoT) reasoning, where the key idea is attribute prompting. We generate detailed descriptions of object/part attributes including shape, color, and location for potential segment proposals through systematic prompting of a large language model (LLM), where the proposals are produced by a foundational image segmentation model. Our approach encourages deep reasoning about object or part attributes related to function, style, design, etc. , enabling the system to handle implicit queries without any part annotations for training or fine-tuning. As the first zero-shot and LLM-based RES method, RESAnything achieves clearly superior performance among zero-shot methods on traditional RES benchmarks and significantly outperforms existing methods on challenging scenarios involving implicit queries and complex part-level relations. Finally, we contribute a new benchmark dataset to offer ~3K carefully curated RES instances to assess part-level, arbitrary RES solutions.

IJCAI Conference 2025 Conference Paper

Rethinking Federated Graph Learning: A Data Condensation Perspective

• Hao Zhang
• Xunkai Li
• Yinlin Zhu
• Lianglin Hu

Federated graph learning is a widely recognized technique that promotes collaborative training of graph neural networks (GNNs) by multi-client graphs. However, existing approaches heavily rely on the communication of model parameters or gradients for federated optimization and fail to adequately address the data heterogeneity introduced by intricate and diverse graph distributions. Although some methods attempt to share additional messages among the server and clients to improve federated convergence during communication, they introduce significant privacy risks and increase communication overhead. To address these issues, we introduce the concept of a condensed graph as a novel optimization carrier to address FGL data heterogeneity and propose a new FGL paradigm called FedGM. Specifically, we utilize a generalized condensation graph consensus to aggregate comprehensive knowledge from distributed graphs, while minimizing communication costs and privacy risks through a single transmission of the condensed data. Extensive experiments on six public datasets consistently demonstrate the superiority of FedGM over state-of-the-art baselines, highlighting its potential for a novel FGL paradigm.

NeurIPS Conference 2025 Conference Paper

RGB-Only Supervised Camera Parameter Optimization in Dynamic Scenes

• Fang Li
• Hao Zhang
• Narendra Ahuja

Although COLMAP has long remained the predominant method for camera parameter optimization in static scenes, it is constrained by its lengthy runtime and reliance on ground truth (GT) motion masks for application to dynamic scenes. Many efforts attempted to improve it by incorporating more priors as supervision such as GT focal length, motion masks, 3D point clouds, camera poses, and metric depth, which, however, are typically unavailable in casually captured RGB videos. In this paper, we propose a novel method for more accurate and efficient camera parameter optimization in dynamic scenes solely supervised by a single RGB video, dubbed $\textbf{\textit{ROS-Cam}}$. Our method consists of three key components: (1) Patch-wise Tracking Filters, to establish robust and maximally sparse hinge-like relations across the RGB video. (2) Outlier-aware Joint Optimization, for efficient camera parameter optimization by adaptive down-weighting of moving outliers, without reliance on motion priors. (3) A Two-stage Optimization Strategy, to enhance stability and optimization speed by a trade-off between the Softplus limits and convex minima in losses. We visually and numerically evaluate our camera estimates. To further validate accuracy, we feed the camera estimates into a 4D reconstruction method and assess the resulting 3D scenes, and rendered 2D RGB and depth maps. We perform experiments on 4 real-world datasets (NeRF-DS, DAVIS, iPhone, and TUM-dynamics) and 1 synthetic dataset (MPI-Sintel), demonstrating that our method estimates camera parameters more efficiently and accurately with a single RGB video as the only supervision.

ICRA Conference 2025 Conference Paper

RINA: Rapid Introspective Neural Adaptation for Out-of-Distribution Payload Configurations on Quadruped Robots

• Oscar Youngquist
• Hao Zhang

Adaptive locomotion is a fundamental capability for quadruped robots, particularly in real-world scenarios when they must transport novel or out-of-distribution (O. O. D.) payloads across diverse terrains. Previous learning-based methods often tightly couple a locomotion controller's learned parameters with the adaptation process, which requires extensive pre-training or slow online updates when encountering O. O. D. payloads. To enable adaptation of quadruped locomotion to O. O. D. payloads, we propose the novel Rapid Introspective Neural Adaptation (RINA) method that rapidly compensates for differences between expected and actual joint torques caused by O. O. D. payloads. RINA introduces an adaptive residual dynamics representation that decouples the learning model's parameters from those used for adaptation. A new neural operator network is introduced to learn a set of basis functions as the learning model, which are combined using linear coefficients to predict residual dynamics. Then, these residual dynamics are used to adjust the locomotion controller's output, compensating for additional torques induced by the O. O. D. payload. During execution, the mixing coefficients can be rapidly and introspectively adapted on-the-go to generate joint torque compensations for O. O. D. payloads, while keeping the learned basis functions unchanged. Experimental results have demonstrated that our RINA approach well addresses on-the-go O. O. D. payload adaptation on varied natural terrains without collecting and retraining on additional data and outperforms baseline methods. More details of this work are provided on the project website: https://hcrlab.gitlab.io/project/rina.

NeurIPS Conference 2025 Conference Paper

SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models

• Ye Sun
• Hao Zhang
• Henghui Ding
• Tiehua Zhang
• Xingjun Ma
• Yu-Gang Jiang

Achieving fine-grained spatio-temporal understanding in videos remains a major challenge for current Video Large Multimodal Models (Video LMMs). Addressing this challenge requires mastering two core capabilities: video referring understanding, which captures the semantics of video regions, and video grounding, which segments object regions based on natural language descriptions. However, most existing approaches tackle these tasks in isolation, limiting progress toward unified, referentially grounded video interaction. We identify a key bottleneck in the lack of high-quality, unified video instruction data and a comprehensive benchmark for evaluating referentially grounded video chat. To address these challenges, we contribute in three core aspects: dataset, model, and benchmark. First, we introduce SAMA-239K, a large-scale dataset comprising 15K videos specifically curated to enable joint learning of video referring understanding, grounding, and multi-turn video chat. Second, we propose the SAMA model, which incorporates a versatile spatio-temporal context aggregator and a Segment Anything Model to jointly enhance fine-grained video comprehension and precise grounding capabilities. Finally, we establish SAMA-Bench, a meticulously designed benchmark consisting of 5, 067 questions from 522 videos, to comprehensively evaluate the integrated capabilities of Video LMMs in multi-turn, spatio-temporal referring understanding and grounded dialogue. Extensive experiments and benchmarking results show that SAMA not only achieves strong performance on SAMA-Bench but also sets a new state-of-the-art on general grounding benchmarks, while maintaining highly competitive performance on standard visual understanding benchmarks.

AAAI Conference 2025 Conference Paper

Scalable Trajectory-User Linking with Dual-Stream Representation Networks

• Hao Zhang
• Wei Chen
• Xingyu Zhao
• Jianpeng Qi
• Guiyuan Jiang
• Yanwei Yu

Trajectory-user linking (TUL) aims to match anonymous trajectories to the most likely users who generated them, offering benefits for a wide range of real-world spatio-temporal applications. However, existing TUL methods are limited by high model complexity and poor learning of the effective representations of trajectories, rendering them ineffective in handling large-scale user trajectory data.In this work, we propose a novel Scalable Trajectory-User Linking with dual-stream representation networks for large-scale TUL problem, named ScaleTUL Specifically, ScaleTUL generates two views using temporal and spatial augmentations to exploit supervised contrastive learning framework to effectively capture the irregularities of trajectories. In each view, a dual-stream trajectory encoder consisting of a long-term encoder and a short-term encoder is designed to learn the unified representations of trajectories that fuses different temporal-spatial dependencies. Then, a TUL layer is used to associate the trajectories with the corresponding users in the representation space using a two-stage training model.Experimental results on check-in mobility datasets from three real-world cities and the nationwide U.S. demonstrate the superiority of ScaleTUL over state-of-the-art baselines for large-scale TUL tasks.

NeurIPS Conference 2025 Conference Paper

Scaling Language-centric Omnimodal Representation Learning

• Chenghao Xiao
• Hou Pong (Ken) Chan
• Hao Zhang
• Weiwen Xu
• Mahani Aljunied
• Yu Rong

Recent multimodal embedding approaches leveraging multimodal large language models (MLLMs) fine-tuned with contrastive learning (CL) have shown promising results, yet the underlying reasons behind their superiority remain underexplored. This work argues that a crucial advantage of MLLM-based approaches stems from implicit cross-modal alignment achieved during generative pretraining, where the language decoder learns to exploit multimodal signals within a shared representation space for generating unimodal outputs. Through analysis of anisotropy and kernel similarity structure, we empirically confirm that latent alignment emerges within MLLM representations, allowing CL to serve as a lightweight refinement stage. Leveraging this insight, we propose a Language-Centric Omnimodal Embedding framework, termed LCO-Embed. Extensive experiments across diverse backbones and benchmarks demonstrate its effectiveness, achieving state-of-the-art performance across modalities. Furthermore, we identify a Generation-Representation Scaling Law (GRSL), showing that the representational capabilities gained through contrastive refinement scale positively with the MLLM's generative capabilities. This suggests that improving generative abilities evolves as an effective paradigm for enhancing representation quality. We provide a theoretical explanation of GRSL, which formally links the MLLM's generative quality to the upper bound on its representation performance, and validate it on a challenging, low-resource visual-document retrieval task, showing that continual generative pretraining before CL can further enhance the potential of a model's embedding capabilities. Codes, models, and resources are available at https: //github. com/LCO-Embedding/LCO-Embedding.

NeurIPS Conference 2025 Conference Paper

Scaling Speculative Decoding with Lookahead Reasoning

• Yichao Fu
• Rui Ge
• Zelei Shao
• Zhijie Deng
• Hao Zhang

Reasoning models excel by generating long chain-of-thoughts, but decoding the resulting thousands of tokens is slow. Token-level specualtive decoding (SD) helps, but its benefit is capped, because the chance that an entire $\gamma$-token guess is correct falls exponentially as $\gamma$ grows. This means allocating more compute for longer token drafts faces an algorithmic ceiling -- making the speedup modest and hardware-agnostic. We raise this ceiling with lookahead reasoning, which exploits a second, step-level layer of parallelism. Our key insight is that reasoning models generate step-by-step, and each step needs only to be semantically correct, not exact token matching. In lookahead reasoning, a lightweight draft model proposes several future steps; the target model expands each proposal in one batched pass, and a verifier keeps semantically correct steps while letting the target regenerate any that fail. Token-level SD still operates within each reasoning step, so the two layers of parallelism multiply. We show lookahead reasoning lifts the peak speedup of SD both theoretically and empirically. Across GSM8K, AIME, and other benchmarks, lookahead reasoning improves the speedup of SD from 1. 4x to 2. 1x while preserving answer quality, and its speedup scales better with additional GPU throughput. Our code is available at https: //github. com/hao-ai-lab/LookaheadReasoning

IROS Conference 2025 Conference Paper

Spatial-Temporal Graph Contrastive Learning with Decreasing Masks for Traffic Flow Forecasting

• Bin Ren
• Yongfa Zhang
• Yamin Wen
• Haocheng Luo
• Hao Zhang
• Chunhong He

In recent years, Contrastive learning has shown great potential in traffic flow prediction tasks. However, existing contrastive learning methods have difficulties in dealing with missing data and noise, and it is difficult to fully capture local and global correlations by relying on a single contrast method. In this paper, a Decreasing Mask Spatio-Temporal Graph Comparison Learning Model (DMSTGCL) is proposed. The model dynamically adjusts the mask ratio through the adaptive mask reduction technique to effectively deal with the problem of missing data and noise. Meanwhile, the projection head is further combined with the TripleAttention mechanism in the spatio-temporal contrast learning process, which overcomes the limitations of a single contrast method and captures the complex relationships in local and global space more effectively. Experiments on three real-world datasets demonstrate that DMSTGCL achieves significantly higher prediction accuracy than existing methods.

NeurIPS Conference 2025 Conference Paper

Stable Part Diffusion 4D: Multi-View RGB and Kinematic Parts Video Generation

• Hao Zhang
• Chun-Han Yao
• Simon Donné
• Narendra Ahuja
• Varun Jampani

We present Stable Part Diffusion 4D (SP4D), a framework for generating paired RGB and kinematic part videos from monocular inputs. Unlike conventional part segmentation methods that rely on appearance-based semantic cues, SP4D learns to produce kinematic parts --- structural components aligned with object articulation and consistent across views and time. SP4D adopts a dual-branch diffusion model that jointly synthesizes RGB frames and corresponding part segmentation maps. To simplify architecture and flexibly enable different part counts, we introduce a spatial color encoding scheme that maps part masks to continuous RGB-like images. This encoding allows the segmentation branch to share the latents VAE from the RGB branch, while enabling part segmentation to be recovered via straightforward post-processing. A Bidirectional Diffusion Fusion (BiDiFuse) module enhances cross-branch consistency, supported by a contrastive part consistency loss to promote spatial and temporal alignment of part predictions. We demonstrate that the generated 2D part maps can be lifted to 3D to derive skeletal structures and harmonic skinning weights with few manual adjustments. To train and evaluate SP4D, we construct KinematicParts20K, a curated dataset of over 20K rigged objects selected and processed from Objaverse XL, each paired with multi-view RGB and part video sequences. Experiments show that SP4D generalizes strongly to diverse scenarios, including real-world videos, novel generated objects, and rare articulated poses, producing kinematic-aware outputs suitable for downstream animation and motion-related tasks.

IJCAI Conference 2025 Conference Paper

Towards Automatic Sampling of User Behaviors for Sequential Recommender Systems

• Hao Zhang
• Mingyue Cheng
• Zhiding Liu
• Junzhe Jiang

Sequential recommender systems (SRS) have gained increasing popularity due to their remarkable proficiency in capturing dynamic user preferences. In the current setup of SRS, a common configuration is to uniformly consider each historical behavior as a positive interaction. However, this setting has the potential to yield sub-optimal performance as each individual item often have a different impact on shaping the user's interests. Hence, in this paper, we propose a novel automatic sampling framework for sequential recommendation, named AutoSAM, to non-uniformly treat historical behaviors. Specifically, AutoSAM extends the conventional SRS framework by integrating an extra sampler to intelligently discern the skew distribution of the raw input, and then sample informative sub-sets to build more generalizable SRS. To tackle the challenges posed by non-differentiable sampling actions and to introduce multiple decision factors for sampling, we further design a novel reinforcement learning based method to guide the training of the sampler. Furthermore, we theoretically devise multi-objective sampling rewards including Future Prediction and Sequence Perplexity, and then optimize the whole framework in an end-to-end manner by combining the policy gradient. We conduct extensive experiments on benchmark recommendation models and four real-world datasets. The experimental results demonstrate the effectiveness of the proposed AutoSAM.

EAAI Journal 2025 Journal Article

TSSA-Net: Transposed Sparse Self-Attention-based network for image super-resolution

• Guanhao Chen
• Dan Xu
• Kangjian He
• Hongzhen Shi
• Hao Zhang

Image super-resolution (SR) aims to reconstruct high-resolution images from low-resolution ones. Transformer-based methods have recently demonstrated remarkable results, but the conventional dense self-attention mechanism fails to capture local relationships between patches, leading to suboptimal performance. Moreover, the recovery of high-frequency information, crucial for edge reconstruction, is often insufficient. To address these challenges, we propose the Transposed Sparse Self-Attention (TSSA) mechanism, which improves local feature attention by restructuring the self-attention computation, without using convolutions. Additionally, we introduce a Segmented Convolutional Feed-Forward Network (SCFFN) to enhance high-frequency detail recovery and local feature acquisition, while maintaining a low parameter count. We combine TSSA, SCFFN, and a channel attention mechanism to develop TSSA-Net, an innovative network for image super-resolution. Comprehensive evaluations on classical, lightweight, and real-world SR tasks show that TSSA-Net outperforms recent methods on the Set14, Urban100, and Manga109 datasets, with improvements of 0. 01–0. 04 decibel (dB), 0. 11–0. 12 dB, and 0. 01–0. 08 dB, respectively. TSSA-Net achieves notable results in both metric-based and visual-based evaluations. The code is available at https: //github. com/VMC-Lab-Chen/TSSA-Net.

EAAI Journal 2025 Journal Article

Tunnel-YOLO: An improved You Only Look Once algorithm for real-time shield tunnel lining leakage detection

• Ruijun Yang
• Hao Zhang

Detecting crack leakages in shield tunnels is crucial for ensuring structural safety and extending service life, as traditional detection methods are limited by high subjectivity and low accuracy. To address these limitations, this paper proposes Tunnel-YOLO, an improved object detection algorithm based on You Only Look Once version 8 (YOLOv8). This algorithm replaces standard convolutional blocks with a novel Receptive Field Channel Attention Convolution (RFCAConv) module, which leverages dynamic receptive fields to enhance feature capture at different scales. We also introduce a C2f_SGE module, integrating the Spatial Group-wise Enhance (SGE) attention mechanism into the C2f (CSPNet with 2 convolutions) block to significantly improve feature extraction while suppressing background interference. Furthermore, an Edge Feature Enhancement Detection Head (EFE-Head) incorporates deconvolution layers to enhance fine-grained details for more precise boundary localization. To better accommodate the shape-sensitive detection task, our LeShape-IoU (Intersection over Union) loss function is designed to focus on the shape and scale characteristics of target bounding boxes. Experimental results on a public, real-world dataset demonstrate that Tunnel-YOLO significantly outperforms the baseline, increasing Recall, Precision, and mean Average Precision at 0. 5 IoU (mAP50) by 15. 7%, 10. 3%, and 14. 8%, respectively. Comparative analysis with other mainstream algorithms further validates the effectiveness and superiority of the proposed Tunnel-YOLO.

IROS Conference 2025 Conference Paper

UAV-DETR: Efficient End-to-End Object Detection for Unmanned Aerial Vehicle Imagery

• Huaxiang Zhang 0002
• Hao Zhang
• Kai Liu
• Zhongxue Gan
• Guo-Niu Zhu

Unmanned aerial vehicle object detection (UAV-OD) has been widely used in various scenarios. However, most existing UAV-OD algorithms rely on manually designed components, which require extensive tuning. End-to-end models that do not depend on such manually designed components are mainly designed for natural images, which are less effective for UAV imagery. To address such challenges, this paper proposes an efficient detection transformer (DETR) framework tailored for UAV imagery, i. e. , UAV-DETR. The framework includes a multi-scale feature fusion with frequency enhancement module, which captures both spatial and frequency information at different scales. In addition, a frequency-focused downsampling module is presented to retain critical spatial details during downsampling. A semantic alignment and calibration module is developed to align and fuse features from different fusion paths. Experimental results demonstrate the effectiveness and generalization of our approach across various UAV imagery datasets. On the VisDrone dataset, our method improves AP by 3. 1% and AP50 by 4. 2% over the baseline. Similar enhancements are observed on the UAVVaste dataset. The project page is available at https://github.com/ValiantDiligent/UAV-DETR.

EAAI Journal 2024 Journal Article

A block-based heuristic search algorithm for the two-dimensional guillotine strip packing problem

• Hao Zhang
• Shaowen Yao
• Shenghui Zhang
• Jiewu Leng
• Lijun Wei
• Qiang Liu

Introduction: This paper addresses the two-dimensional strip-packing (2DSP) problem of placing a set of rectangular pieces onto a fixed-width rectangular sheet to minimize the total length used. Methodology: We propose a Block-Based Heuristic Search Algorithm (BBHSA) to solve 2DSP problems with guillotine cut constraints. Initially, it converts the 2DSP problem into a series of 2D rectangular packing problems (2DRP), where the size of the sheet is fixed. In the BBHSA, rectangular pieces are aggregated into blocks, which are partial solutions without residual space. These blocks provide the ingredients for a good layout and are basic components in a tree-based constructive search process. Two basic operations, called placing & splitting and approximate binary search are used in the search process. Furthermore, several block-based placement rules are explored to speed up the search process and improve solution quality. Result discussion: To verify the performance of our proposed algorithm, we conducted extensive experiments using the zero-waste benchmark and non-zero-waste benchmark instances. Final results: The results show that BBHSA demonstrates computational effectiveness, particularly in zero-waste cases, achieving optimal solutions for almost all zero-waste benchmark instances reported in the existing literature.

EAAI Journal 2024 Journal Article

A commercial vehicle weight prediction method based on driving simulation data

• Yu Liu
• Hao Zhang
• Xianglei Zhu
• Jingyuan Li
• Hang Xu
• Bo Zhang
• Hanzhengnan Yu
• Yu Wang

Neural network is beneficial for obtaining weight, which is a crucial parameter for vehicle in operation. However, the high demand of neural network for real data, and the lack of in-depth research on multi-vehicle weight prediction scenarios, have limited the development of weight prediction. In this paper, relying on the genetic algorithm-optimized BP neural network (GA-BP neural network), a commercial vehicle weight prediction method based on driving simulation data was proposed. The driving data of multiple commercial vehicle types were collected and used to study the weight prediction effect of the proposed method in single-vehicle and multi-vehicle scenarios. The fuzzy processing method of driving simulation data was proposed to process simulation data instead of real data to train the model, which effectively solves the problem of high demand for real data in neural network method. The study shows that proposed method can achieve weight prediction deviation of 2. 87% for single-vehicle model and 4. 63% for multi-vehicle model.

AAAI Conference 2024 Conference Paper

A Robust Mutual-Reinforcing Framework for 3D Multi-Modal Medical Image Fusion Based on Visual-Semantic Consistency

• Hao Zhang
• Xuhui Zuo
• Huabing Zhou
• Tao Lu
• Jiayi Ma

This work proposes a robust 3D medical image fusion framework to establish a mutual-reinforcing mechanism between visual fusion and lesion segmentation, achieving their double improvement. Specifically, we explore the consistency between vision and semantics by sharing feature fusion modules. Through the coupled optimization of the visual fusion loss and the lesion segmentation loss, visual-related and semantic-related features will be pulled into the same domain, effectively promoting accuracy improvement in a mutual-reinforcing manner. Further, we establish the robustness guarantees by constructing a two-level refinement constraint in the process of feature extraction and reconstruction. Benefiting from full consideration for common degradations in medical images, our framework can not only provide clear visual fusion results for doctor's observation, but also enhance the defense ability of lesion segmentation against these negatives. Extensive evaluations of visual fusion and lesion segmentation scenarios demonstrate the advantages of our method in terms of accuracy and robustness. Moreover, our proposed framework is generic, which can be well-compatible with existing lesion segmentation algorithms and improve their performance. The code is publicly available at https://github.com/HaoZhang1018/RMR-Fusion.

IROS Conference 2024 Conference Paper

Accurate and Efficient Loop Closure Detection With Deep Binary Image Descriptor and Augmented Point Cloud Registration

• Jialiang Wang
• Zhi Gao 0005
• Zhipeng Lin
• Zhiyu Zhou
• Xiaonan Wang
• Jianhua Cheng
• Hao Zhang
• Xinyi Liu 0002

Loop Closure Detection (LCD) is an essential component of Simultaneous Localization and Mapping (SLAM), helping to correct drift errors, facilitate map merging, or both by identifying previously observed scenes. Despite its importance, traditional LCD algorithms based on single sensor such as camera or LiDAR exhibit degraded performance in challenging scenarios due to their inherent limitations. To address this issue, we propose a novel LCD method based on camera-LiDAR fusion, exploiting the rich textural information from cameras and the accurate geometric data from LiDAR to ensure robustness and speed in challenging environments. Specifically, we first employ deep hashing learning to encode deep image features into binary image descriptors for extremely fast loop candidate (LC) retrieval. Then, LiDAR points are augmented with image color for accurate geometric verification. Finally, we incorporate a spatial-temporal consistency check that mandates an LC to have consistently matched neighbors to be accepted as true. Our method is extensively verified and compared with the state-of-the-art methods on various datasets encompassing both indoor and outdoor environments. Experimental results demonstrate that our method obtains the best performance, increasing the maximum recall rate at 100% precision by a significant margin of 20% while operating in real-time at an average speed of 30 fps.

ICLR Conference 2024 Conference Paper

Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding

• Zilong Wang 0002
• Hao Zhang
• Chun-Liang Li
• Julian Martin Eisenschlos
• Vincent Perot
• Zifeng Wang 0002
• Lesly Miculicich
• Yasuhisa Fujii

Table-based reasoning with large language models (LLMs) is a promising direction to tackle many table understanding tasks, such as table-based question answering and fact verification. Compared with generic reasoning, table-based reasoning requires the extraction of underlying semantics from both free-form questions and semi-structured tabular data. Chain-of-Thought and its similar approaches incorporate the reasoning chain in the form of textual context, but it is still an open question how to effectively leverage tabular data in the reasoning chain. We propose the Chain-of-Table framework, where tabular data is explicitly used in the reasoning chain as a proxy for intermediate thoughts. Specifically, we guide LLMs using in-context learning to iteratively generate operations and update the table to represent a tabular reasoning chain. LLMs can therefore dynamically plan the next operation based on the results of the previous ones. This continuous evolution of the table forms a chain, showing the reasoning process for a given tabular problem. The chain carries structured information of the intermediate results, enabling more accurate and reliable predictions. Chain-of-Table achieves new state-of-the-art performance on WikiTQ, FeTaQA, and TabFact benchmarks across multiple LLM choices.

AAAI Conference 2024 Conference Paper

Clarifying the Behavior and the Difficulty of Adversarial Training

• Xu Cheng
• Hao Zhang
• Yue Xin
• Wen Shen
• Quanshi Zhang

Adversarial training is usually difficult to optimize. This paper provides conceptual and analytic insights into the difficulty of adversarial training via a simple theoretical study, where we derive an approximate dynamics of a recursive multi-step attack in a simple setting. Despite the simplicity of our theory, it still reveals verifiable predictions about various phenomena in adversarial training under real-world settings. First, compared to vanilla training, adversarial training is more likely to boost the influence of input samples with large gradient norms in an exponential manner. Besides, adversarial training also strengthens the influence of the Hessian matrix of the loss w.r.t. network parameters, which is more likely to make network parameters oscillate and boosts the difficulty of adversarial training.

NeurIPS Conference 2024 Conference Paper

ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Ablation Capability for Large Vision-Language Models

• Shuo Liu
• Kaining Ying
• Hao Zhang
• Yue Yang
• Yuqi Lin
• Tianle Zhang
• Chuanhao Li
• Yu Qiao

Multi-turn visual conversation is an important ability of real-world AI assistants. However, the related evaluation benchmark is missed. This paper presents ConvBench, a multi-turn conversation benchmark with hierarchical capabilities ablation evaluation for Large Vision-Language Models (LVLMs). ConvBench comprises 577 curated multi-turn conversations, encompassing 215 tasks. These tasks are broad and open-ended, which resemble real-world user behaviors. ConvBench progressively examines the LVLMs' perception, reasoning, and creativity capabilities in each conversation and can decouple these capabilities in evaluations and thus perform reliable error attribution. Besides, considering the diversity of open-ended questions, we introduce an efficient and reliable automatic evaluation framework. Experimental results reveal that ConvBench is a significant challenge for current LVLMs, even for GPT4V, which achieves only a 39. 51% score. Besides, we have some insightful findings, such as the weak perception of LVLMs inhibits authentic strengths in reasoning and creation. We believe our design of hierarchical capabilities, decoupling capabilities evaluation, and multi-turn conversation can blaze a new trail in LVLMs evaluation. Code and benchmark are released at https: //github. com/shirlyliu64/ConvBench.

NeurIPS Conference 2024 Conference Paper

CRAYM: Neural Field Optimization via Camera RAY Matching

• Liqiang Lin
• Wenpeng Wu
• Chi-Wing Fu
• Hao Zhang
• Hui Huang

We introduce camera ray matching (CRAYM) into the joint optimization of camera poses and neural fields from multi-view images. The optimized field, referred to as a feature volume, can be “probed” by the camera rays for novel view synthesis (NVS) and 3D geometry reconstruction. One key reason for matching camera rays, instead of pixels as in prior works, is that the camera rays can be parameterized by the feature volume to carry both geometric and photometric information. Multi-view consistencies involving the camera rays and scene rendering can be naturally integrated into the joint optimization and network training, to impose physically meaningful constraints to improve the final quality of both the geometric reconstruction and photorealistic rendering. We formulate our per-ray optimization and matched ray coherence by focusing on camera rays passing through keypoints in the input images to elevate both the efficiency and accuracy of scene correspondences. Accumulated ray features along the feature volume provide a means to discount the coherence constraint amid erroneous ray matching. We demonstrate the effectiveness of CRAYM for both NVS and geometry reconstruction, over dense- or sparse-view settings, with qualitative and quantitative comparisons to state-of-the-art alternatives.

IJCAI Conference 2024 Conference Paper

Cross-Scale Domain Adaptation with Comprehensive Information for Pansharpening

• Meiqi Gong
• Hao Zhang
• Hebaixu Wang
• Jun Chen
• Jun Huang
• Xin Tian
• Jiayi Ma

Deep learning-based pansharpening methods typically use simulated data at the reduced-resolution scale for training. It limits their performance when generalizing the trained model to the full-resolution scale due to incomprehensive information utilization of panchromatic (PAN) images at the full-resolution scale and low generalization ability. In this paper, we adopt two targeted strategies to address the above two problems. On the one hand, we introduce a cross-scale comprehensive information capture module, which improves the information utilization of the original PAN image through fully-supervised reconstruction. On the other hand, we pioneer a domain adaptation strategy to tackle the problem of low generalization across different scales. Considering the instinct domain gap between different scales, we leverage the maximum mean discrepancy loss and the inherent pixel-level correlations between features at different scales to reduce the scale variance, thus boosting the generalization ability of our model. Experiments on various satellites demonstrate the superiority of our method over the state-of-the-arts in terms of information retention. Our code is publicly available at https: //github. com/Meiqi-Gong/SDIPS.

AAAI Conference 2024 Conference Paper

CSL: Class-Agnostic Structure-Constrained Learning for Segmentation Including the Unseen

• Hao Zhang
• Fang Li
• Lu Qi
• Ming-Hsuan Yang
• Narendra Ahuja

Addressing Out-Of-Distribution (OOD) Segmentation and Zero-Shot Semantic Segmentation (ZS3) is challenging, necessitating segmenting unseen classes. Existing strategies adapt the class-agnostic Mask2Former (CA-M2F) tailored to specific tasks. However, these methods cater to singular tasks, demand training from scratch, and we demonstrate certain deficiencies in CA-M2F, which affect performance. We propose the Class-Agnostic Structure-Constrained Learning (CSL), a plug-in framework that can integrate with existing methods, thereby embedding structural constraints and achieving performance gain, including the unseen, specifically OOD, ZS3, and domain adaptation (DA) tasks. There are two schemes for CSL to integrate with existing methods (1) by distilling knowledge from a base teacher network, enforcing constraints across training and inference phrases, or (2) by leveraging established models to obtain per-pixel distributions without retraining, appending constraints during the inference phase. Our soft assignment and mask split methodologies enhance OOD object segmentation. Empirical evaluations demonstrate CSL's prowess in boosting the performance of existing algorithms spanning OOD segmentation, ZS3, and DA segmentation, consistently transcending the state-of-art across all three tasks.

AAAI Conference 2024 Conference Paper

Data Adaptive Traceback for Vision-Language Foundation Models in Image Classification

• Wenshuo Peng
• Kaipeng Zhang
• Yue Yang
• Hao Zhang
• Yu Qiao

Vision-language foundation models have been incredibly successful in a wide range of downstream computer vision tasks using adaptation methods. However, due to the high cost of obtaining pre-training datasets, pairs with weak image-text correlation in the data exist in large numbers. We call them weak-paired samples. Due to the limitations of these weak-paired samples, the pre-training model are unable to mine all the knowledge from pre-training data. The existing adaptation methods do not consider the missing knowledge, which may lead to crucial task-related knowledge for the downstream tasks being ignored. To address this issue, we propose a new adaptation framework called Data Adaptive Traceback (DAT). Specifically, we utilize a zero-shot-based method to extract the most downstream task-related subset of the pre-training data to enable the downstream tasks. Furthermore, we adopt a pseudo-label-based semi-supervised technique to reuse the pre-training images and a vision-language contrastive learning method to address the confirmation bias issue in semi-supervised learning. We conduct extensive experiments that show our proposed DAT approach meaningfully improves various benchmark datasets’ performance over traditional adaptation methods by simply.

AAAI Conference 2024 Conference Paper

Deep Unfolded Network with Intrinsic Supervision for Pan-Sharpening

• Hebaixu Wang
• Meiqi Gong
• Xiaoguang Mei
• Hao Zhang
• Jiayi Ma

Existing deep pan-sharpening methods lack the learning of complementary information between PAN and MS modalities in the intermediate layers, and exhibit low interpretability due to their black-box designs. To this end, an interpretable deep unfolded network with intrinsic supervision for pan-sharpening is proposed. Building upon the observation degradation process, it formulates the pan-sharpening task as a variational model minimization with spatial consistency prior and spectral projection prior. The former prior requires a joint component decomposition of PAN and MS images to extract intrinsic features. By being supervised in the intermediate layers, it can selectively provide high-frequency information for spatial enhancement. The latter prior constrains the intensity correlation between MS and PAN images derived from physical observations, so as to improve spectral fidelity. To further enhance the transparency of network design, we develop an iterative solution algorithm following the half-quadratic splitting to unfold the deep model. It rigorously adheres to the variational model, significantly enhancing the interpretability behind network design and efficiently alternating the optimization of the network. Extensive experiments demonstrate the advantages of our method compared to state-of-the-arts, showcasing its remarkable generalization capability to real-world scenes. Our code is publicly available at https://github.com/Baixuzx7/DISPNet.

NeurIPS Conference 2024 Conference Paper

Efficient LLM Scheduling by Learning to Rank

• Yichao Fu
• Siqi Zhu
• Runlong Su
• Aurick Qiao
• Ion Stoica
• Hao Zhang

In Large Language Model (LLM) inference, the output length of an LLM request is typically regarded as not known a priori. Consequently, most LLM serving systems employ a simple First-come-first-serve (FCFS) scheduling strategy, leading to Head-Of-Line (HOL) blocking and reduced throughput and service quality. In this paper, we reexamine this assumption -- we show that, although predicting the exact generation length of each request is infeasible, it is possible to predict the relative ranks of output lengths in a batch of requests, using learning to rank. The ranking information offers valuable guidance for scheduling requests. Building on this insight, we develop a novel scheduler for LLM inference and serving that can approximate the shortest-job-first (SJF) schedule better than existing approaches. We integrate this scheduler with the state-of-the-art LLM serving system and show significant performance improvement in several important applications: 2. 8x lower latency in chatbot serving and 6. 5x higher throughput in synthetic data generation. Our code is available at https: //github. com/hao-ai-lab/vllm-ltr. git

NeurIPS Conference 2024 Conference Paper

Efficiently Learning Significant Fourier Feature Pairs for Statistical Independence Testing

• Yixin Ren
• Yewei Xia
• Hao Zhang
• Jihong Guan
• Shuigeng Zhou

We propose a novel method to efficiently learn significant Fourier feature pairs for maximizing the power of Hilbert-Schmidt Independence Criterion~(HSIC) based independence tests. We first reinterpret HSIC in the frequency domain, which reveals its limited discriminative power due to the inability to adapt to specific frequency-domain features under the current inflexible configuration. To remedy this shortcoming, we introduce a module of learnable Fourier features, thereby developing a new criterion. We then derive a finite sample estimate of the test power by modeling the behavior of the criterion, thus formulating an optimization objective for significant Fourier feature pairs learning. We show that this optimization objective can be computed in linear time (with respect to the sample size $n$), which ensures fast independence tests. We also prove the convergence property of the optimization objective and establish the consistency of the independence tests. Extensive empirical evaluation on both synthetic and real datasets validates our method's superiority in effectiveness and efficiency, particularly in handling high-dimensional data and dealing with large-scale scenarios.

AAAI Conference 2024 Conference Paper

Explaining Generalization Power of a DNN Using Interactive Concepts

• Huilin Zhou
• Hao Zhang
• Huiqi Deng
• Dongrui Liu
• Wen Shen
• Shih-Han Chan
• Quanshi Zhang

This paper explains the generalization power of a deep neural network (DNN) from the perspective of interactions. Although there is no universally accepted definition of the concepts encoded by a DNN, the sparsity of interactions in a DNN has been proved, i.e., the output score of a DNN can be well explained by a small number of interactions between input variables. In this way, to some extent, we can consider such interactions as interactive concepts encoded by the DNN. Therefore, in this paper, we derive an analytic explanation of inconsistency of concepts of different complexities. This may shed new lights on using the generalization power of concepts to explain the generalization power of the entire DNN. Besides, we discover that the DNN with stronger generalization power usually learns simple concepts more quickly and encodes fewer complex concepts. We also discover the detouring dynamics of learning complex concepts, which explains both the high learning difficulty and the low generalization power of complex concepts. The code will be released when the paper is accepted.

EAAI Journal 2024 Journal Article

HIWANet: A high imperceptibility watermarking attack network

• Chunpeng Wang
• Xinying Li
• Zhiqiu Xia
• Qi Li
• Hao Zhang
• Jian Li
• Bing Han
• Bin Ma

Digital image watermarking technology has made a significant contribution to the copyright protection of digital images. In recent years, researchers have focused on designing various watermarking algorithms to enhance resistance against different forms of attacks. However, the evolution of watermarking attack technology has been sluggish, thus impeding possible advancements in digital copyright protection. Existing watermarking attack methods exhibit a notable drawback, causing substantial deterioration in visual quality and undermining the practical utility of attacked images. In this paper, we propose a high imperceptibility watermarking attack network, named HIWANet, based on deep neural networks. To enhance the watermarking attack ability, a feature extraction module (FEM) is aimed to better capture watermark information features, and a watermarking attack module (WAM) is constructed to learn high-level abstract features of the images. In addition, to ensure the imperceptibility of the watermarking attack, an asymmetric loss function is designed to maintain the quality of the attacked watermarked image. In the experiments, we randomly select 2000 color images from the PASCAL VOC2012 database as the dataset, with the training and test sets containing 1000 distinct images. Compared to traditional watermarking attack methods, our HIWANet achieves a significant increase in bit error rate (improved by 242%), indicating a higher attack ability. Meanwhile, it brings more than 26% improvement in the attack imperceptibility. Furthermore, our HIWANet also offers significant advantages compared to deep learning-based watermarking attack methods.

NeurIPS Conference 2024 Conference Paper

Improving Generalization in Federated Learning with Model-Data Mutual Information Regularization: A Posterior Inference Approach

• Hao Zhang
• Chenglin Li
• Nuowen Kan
• Ziyang Zheng
• Wenrui Dai
• Junni Zou
• Hongkai Xiong

Most of existing federated learning (FL) formulation is treated as a point-estimate of models, inherently prone to overfitting on scarce client-side data with overconfident decisions. Though Bayesian inference can alleviate this issue, a direct posterior inference at clients may result in biased local posterior estimates due to data heterogeneity, leading to a sub-optimal global posterior. From an information-theoretic perspective, we propose FedMDMI, a federated posterior inference framework based on model-data mutual information (MI). Specifically, a global model-data MI term is introduced as regularization to enforce the global model to learn essential information from the heterogeneous local data, alleviating the bias caused by data heterogeneity and hence enhancing generalization. To make this global MI tractable, we decompose it into local MI terms at the clients, converting the global objective with MI regularization into several locally optimizable objectives based on local data. For these local objectives, we further show that the optimal local posterior is a Gibbs posterior, which can be efficiently sampled with stochastic gradient Langevin dynamics methods. Finally, at the server, we approximate sampling from the global Gibbs posterior by simply averaging samples from the local posteriors. Theoretical analysis provides a generalization bound for FL w. r. t. the model-data MI, which, at different levels of regularization, represents a federated version of the bias-variance trade-off. Experimental results demonstrate a better generalization behavior with better calibrated uncertainty estimates of FedMDMI.

ICML Conference 2024 Conference Paper

InferCept: Efficient Intercept Support for Augmented Large Language Model Inference

• Reyna Abhyankar
• Zijian He
• Vikranth Srivatsa
• Hao Zhang
• Yiying Zhang 0005

Large language models are increasingly integrated with external environments, tools, and agents like ChatGPT plugins to extend their capability beyond language-centric tasks. However, today’s LLM inference systems are designed for standalone LLMs. They treat each external interaction as the end of LLM generation and form a new request when the interaction finishes, causing unnecessary recomputation of already computed contexts, which accounts for 37-40% of total model forwarding time. This paper presents InferCept, the first LLM inference framework targeting augmented LLMs and supporting the efficient interception of LLM generation. InferCept minimizes the GPU resource waste caused by LLM interceptions and dedicates saved memory for serving more requests. InferCept improves the overall serving throughput by 1. 6x-2x and completes 2x more requests per second compared to the state-of-the-art LLM inference systems.

NeurIPS Conference 2024 Conference Paper

Interfacing Foundation Models' Embeddings

• Xueyan Zou
• Linjie Li
• Jianfeng Wang
• Jianwei Yang
• Mingyu Ding
• Junyi Wei
• Zhengyuan Yang
• Feng Li

Foundation models possess strong capabilities in reasoning and memorizing across modalities. To further unleash the power of foundation models, we present FIND, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity. As shown in Fig. 1, a lightweight transformer interface without tuning any foundation model weights is enough for segmentation, grounding, and retrieval in an interleaved manner. The proposed interface has the following favorable attributes: (1) Generalizable. It applies to various tasks spanning retrieval, segmentation, etc. , under the same architecture and weights. (2) Interleavable. With the benefit of multi-task multi-modal training, the proposed interface creates an interleaved shared embedding space. (3) Extendable. The proposed interface is adaptive to new tasks, and new models. In light of the interleaved embedding space, we introduce FIND-Bench, which introduces new training and evaluation annotations to the COCO dataset for interleaved segmentation and retrieval. We are the first work aligning foundations models' embeddings for interleave understanding. Meanwhile, our approach achieves state-of-the-art performance on FIND-Bench and competitive performance on standard retrieval and segmentation settings.

EAAI Journal 2024 Journal Article

Leak detection for natural gas gathering pipeline using spatio-temporal fusion of practical operation data

• Jing Liang
• Shan Liang
• Li Ma
• Hao Zhang
• Juan Dai
• Hongyu Zhou

Gathering pipelines are one of the key upstream infrastructures in the gas industry that link production well to the processing plant. Leak detection is critical for ensuring the safety of pipeline transmission. The detection of small leakage in gathering pipelines consistently poses a formidable challenge. In this paper, a process model is built based on health data of supervisory control and data acquisition system from the actual operating pipeline. In the model structure, the convolutional neural network is used to extract the spatial features, the bi-directional long short-term memory is used to extract the temporal features, and the attention mechanism is employed to allocate the model’s attention resources reasonably. Next, the residual between the entity pipeline’s output data and the process model’s output data is used as a monitoring indicator of the operating state of the pipeline. A clustering-based boundary determination method is proposed to recognize the centroid of normal and small leak conditions, and pipeline leak detection is performed by the Euclidean distance between the monitoring indicator and the centroid. This paper explores the feasibility of fast modeling and leak detection with limited hardware. Field tests for the validation of the proposed methods were implemented in two in-service natural gas gathering pipeline. The experimental results demonstrate that the proposed method significantly enhances the detection performance of small-size leak. The leak detection rates of 94. 06% and 92. 16% evinces the potency of the proposed method applied in the leak detection of gathering pipelines across diverse real-world scenarios.

ICRA Conference 2024 Conference Paper

Learning for Dynamic Subteaming and Voluntary Waiting in Heterogeneous Multi-Robot Collaborative Scheduling

• Williard Joshua Jose
• Hao Zhang

Coordinating heterogeneous robots is essential for autonomous multi-robot teaming. To execute a set of dependent tasks as quickly as possible, and to complete tasks that cannot be addressed by individual robots, it is necessary to form subteams that can collaboratively finish the tasks. It is also advantageous for robots to wait for teammates and tasks to become available in order to form better subteams or reduce the overall completion time. To enable both abilities, we introduce a new graph learning approach that formulates heterogeneous collaborative scheduling as a bipartite matching problem that maximizes a reward matrix learned via imitation learning. We design a novel graph attention transformer network (GATN) that represents the problem of collaborative scheduling as a bipartite graph, and integrates both local and global graph information to estimate the reward matrix using graph attention networks and transformers. By relaxing the constraint of one-to-one correspondence in bipartite matching, our approach allows multiple robots to address the same task as a subteam. Our approach also enables voluntary waiting by introducing an idle task that the robots can select to wait. Experimental results have shown that our approach well addresses heterogeneous collaborative scheduling with dynamic subteam formation and voluntary waiting, and outperforms the previous and baseline methods.

EAAI Journal 2024 Journal Article

Medical hyperspectral image classification based weakly supervised single-image global learning network

• Chenglong Zhang
• Lichao Mou
• Shihao Shan
• Hao Zhang
• Yafei Qi
• Dexin Yu
• Xiao Xiang Zhu
• Nianzheng Sun

Medical hyperspectral imaging provides new possibilities for non-invasive detection and characterization of diseases, and the processing of images can be accelerated and rationalized by using deep learning technology to classify pixels as one tissue or another, or as lesion or healthy tissue. However, most current methods for intelligently identifying pixels are not robust to large variations in pixel intensity within an image, particularly local learning approaches that rely on pixel or patch input. In this paper, we propose a network being able to learn to classify all pixels on an image by training with only a small number of manually labeled pixels in the same image. The network contains a hard band attention module (HBAM) to eliminate noisy bands and a dual-kernel spatial–spectral fusion attention module (DK-SSFAM) which uses two convolution kernels to weight spatial and spectral features and integrates them accordingly. We demonstrate that our proposed weakly supervised single-image global learning (SiGL) network classifies pixels in hyperspectral images of human brain in vivo better than traditional deep learning methods, suggesting potential for the clinic.

NeurIPS Conference 2024 Conference Paper

Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length

• Xuezhe Ma
• Xiaomeng Yang
• Wenhan Xiong
• Beidi Chen
• Lili Yu
• Hao Zhang
• Jonathan May
• Luke Zettlemoyer

The quadratic complexity and weak length extrapolation of Transformers limits their ability to scale to long sequences, and while sub-quadratic solutions like linear attention and state space models exist, they empirically underperform Transformers in pretraining efficiency and downstream task accuracy. We introduce MEGALODON, an neural architecture for efficient sequence modeling with unlimited context length. MEGALODON inherits the architecture of MEGA (exponential moving average with gated attention), and further introduces multiple technical components to improve its capability and stability, including complex exponential moving average (CEMA), timestep normalization layer, normalized attention mechanism and pre-norm with two-hop residual configuration. In a controlled head-to-head comparison with LLAMA2, MEGALODON achieves better efficiency than Transformer in the scale of 7 billion parameters and 2 trillion training tokens. MEGALODON reaches a training loss of 1. 70, landing mid-way between LLAMA2-7B (1. 75) and LLAMA2-13B (1. 67). This result is robust throughout a wide range of benchmarks, where MEGALODON consistently outperforms Transformers across different tasks, domains, and modalities.

AAMAS Conference 2024 Conference Paper

Minimizing Negative Side Effects in Cooperative Multi-Agent Systems using Distributed Coordination

• Moumita Choudhury
• Sandhya Saisubramanian
• Hao Zhang
• Shlomo Zilberstein

Autonomous agents in real-world environments may encounter undesirable outcomes or negative side effects (NSEs) when working collaboratively alongside other agents. We frame the challenge of minimizing NSEs in a multi-agent setting as a lexicographic decentralized Markov decision process in which we assume independence of rewards and transitions with respect to the primary assigned tasks, but allowing negative side effects to create a form of dependence among the agents. We present a lexicographic Q-learning approach to mitigate the NSEs using human feedback models while maintaining near-optimality with respect to the assigned tasks—up to some given slack. Our empirical evaluation across two domains demonstrates that our collaborative approach effectively mitigates NSEs, outperforming non-collaborative methods.

NeurIPS Conference 2024 Conference Paper

MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs

• Zhongshen Zeng
• Yinhong Liu
• Yingjia Wan
• Jingyao Li
• Pengguang Chen
• Jianbo Dai
• Yuxuan Yao
• Rongwu Xu

Large language models (LLMs) have shown increasing capability in problem-solving and decision-making, largely based on the step-by-step chain-of-thought reasoning processes. However, evaluating these reasoning abilities has become increasingly challenging. Existing outcome-based benchmarks are beginning to saturate, becoming less effective in tracking meaningful progress. To address this, we present a process-based benchmark MR-Ben that demands a meta-reasoning skill, where LMs are asked to locate and analyse potential errors in automatically generated reasoning steps. Our meta-reasoning paradigm is especially suited for system-2 slow thinking, mirroring the human cognitive process of carefully examining assumptions, conditions, calculations, and logic to identify mistakes. MR-Ben comprises 5, 975 questions curated by human experts across a wide range of subjects, including physics, chemistry, logic, coding, and more. Through our designed metrics for assessing meta-reasoning on this benchmark, we identify interesting limitations and weaknesses of current LLMs (open-source and closed-source models). For example, with models like the o1 series from OpenAI demonstrating strong performance by effectively scrutinizing the solution space, many other state-of-the-art models fall significantly behind on MR-Ben, exposing potential shortcomings in their training strategies and inference methodologies.

AAMAS Conference 2024 Conference Paper

PADDLE: Logic Program Guided Policy Reuse in Deep Reinforcement Learning

• Hao Zhang
• Tianpei Yang
• Yan Zheng
• Jianye Hao
• Matthew E. Taylor

Learning new skills through previous experience is regular in human life, which is the core idea of Transfer Reinforcement Learning (TRL). TRL requires the agent to learn when and which source policy is the best to reuse as the target task’s policy and how to reuse the source policy. Most TRL methods learn, transfer, and reuse blackbox policies, which is hard to explain: 1) when to reuse, 2) which source policy is effective, and reduces transfer efficiency. In this paper, we propose a novel TRL method called ProgrAm guiDeD poLicy rEuse (PADDLE). PADDLE can measure the logic similarities between tasks and transfer knowledge which reflects the logic behind the target task. To achieve this, we propose a hybrid decision model that synthesizes high-level logic programs and learns low-level DRL policy to learn source tasks. Second, we propose a transferability metric that can measure the logic similarity between the target task and source tasks. Last, we combine it with the lowlevel policy similarity to select the appropriate source policy as the guiding policy for the target task. Experimental results show that PADDLE can effectively select the appropriate source tasks to guide learning on the target task, outperforming black-box TRL methods.

IJCAI Conference 2024 Conference Paper

PointTFA: Training-Free Clustering Adaption for Large 3D Point Cloud Models

• Jinmeng Wu
• Chong Cao
• Hao Zhang
• Basura Fernando
• Yanbin Hao
• Hanyu Hong

The success of contrastive learning models like CLIP, known for aligning 2D image-text pairs, has inspired the development of triplet alignment for Large 3D Point Cloud Models (3D-PCM). Examples like ULIP integrate images, text, and point clouds into a unified semantic space. However, despite showing impressive zero-shot capabilities, frozen 3D-PCM still falls short compared to fine-tuned methods, especially when downstream 3D datasets are significantly different from upstream data. Addressing this, we propose a Data-Efficient, Training-Free 3D Adaptation method named PointTFA that adjusts ULIP outputs with representative samples. PointTFA comprises the Representative Memory Cache (RMC) for selecting a representative support set, Cloud Query Refactor (CQR) for reconstructing a query cloud using the support set, and Training-Free 3D Adapter (3D-TFA) for inferring query categories from the support set. A key advantage of PointTFA is that it introduces no extra training parameters, yet outperforms vanilla frozen ULIP, closely approaching few-shot fine-tuning training methods in downstream cloud classification tasks like ModelNet10 & 40 and ScanObjectNN. The code is available at: https: //github. com/CaoChong-git/PointTFA.

JBHI Journal 2024 Journal Article

RegFSC-Net: Medical Image Registration via Fourier Transform With Spatial Reorganization and Channel Refinement Network

• Chenou Liu
• Kangjian He
• Dan Xu
• Hongzhen Shi
• Hao Zhang
• Kunyuan Zhao

Medical image registration is crucial in medical image analysis applications. Recently, U-Net-style networks have been commonly used for unsupervised image registration, predicting dense displacement fields in full-resolution space. However, this process is resource-intensive and time-consuming for high-resolution volumetric image data. To address this challenge, this paper proposes a novel model named RegFSC-Net, which utilizes Fourier transform with spatial reorganization (SR) and channel refinement (CR) network for registration. We embed efficient feature extraction modules SR and CR modules into the encoder, and adopt a parameter-free model to drive the decoder to improve the U-shaped network. Precisely, RegFSC-Net does not directly predict the full-resolution displacement field in space but learns the low-dimensional representation of the displacement field in the bandlimited Fourier domain, which is beneficial in reducing network parameters, memory usage, and computational costs. Experimental results show that RegFSC-Net outperforms various state-of-the-art methods. Specifically, in comparison to the widely recognized Transformer-based method TransMorph, RegFSC-Net utilizes only around 8. 2% of its parameters, resulting in a 1. 95% higher Dice score and significantly faster inference speeds of 126. 67% and 419. 99% on GPU and CPU, respectively. Furthermore, we also designed three variants of RegFSC-Net and demonstrated their potential applications in computer-aided diagnosis.

IROS Conference 2024 Conference Paper

Segmented Safety Docking Control for Mobile Self-Reconfigurable Robots

• Zhi Zheng
• Tao Jiang 0018
• Senqi Tan
• Hao Zhang
• Jianchuan Ye

Mobile self-reconfigurable robots (MSRRs), as a novel multi-robot system with flexible configurations and task adaptability, hold promising applications in unstructured task environments. However, existing autonomous docking strategies are primarily applied in laboratory settings and face numerous challenges and limitations in actual applications, including differences in sensor characteristics, safety threats, and saturation constraints. To address these issues, this paper proposes a segmented secure docking control framework based on global localization and local perception to achieve stable and reliable reconfiguration of MSRRs in practical applications. Specific contributions include the implementation of a dual-layer constraint framework for safeness of units in the long-distance phase against velocity and acceleration nested windups, and the integration of active line-of-sight (LOS) correction and adaptive windup driving mobile units to achieve precise and rapid locking of docked positions within the LOS in the close-range phase. Finally, the validity of the proposed method is verified via physical experiments, offering an innovative approach to deploying MSRRs in complex scenarios.

IROS Conference 2024 Conference Paper

SiCP: Simultaneous Individual and Cooperative Perception for 3D Object Detection in Connected and Automated Vehicles

• Deyuan Qu
• Qi Chen 0018
• Tianyu Bai
• Hongsheng Lu
• Heng Fan 0001
• Hao Zhang
• Song Fu
• Qing Yang 0003

Cooperative perception for connected and automated vehicles is traditionally achieved through the fusion of feature maps from two or more vehicles. However, the absence of feature maps shared from other vehicles can lead to a significant decline in 3D object detection performance for cooperative perception models compared to standalone 3D detection models. This drawback impedes the adoption of cooperative perception as vehicle resources are often insufficient to concurrently employ two perception models. To tackle this issue, we present Simultaneous Individual and Cooperative Perception (SiCP), a generic framework that supports a wide range of the state-of-the-art standalone perception backbones and enhances them with a novel Dual-Perception Network (DP-Net) designed to facilitate both individual and cooperative perception. In addition to its lightweight nature with only 0. 13M parameters, DP-Net is robust and retains crucial gradient information during feature map fusion. As demonstrated in a comprehensive evaluation on the V2V4Real and OPV2V datasets, thanks to DP-Net, SiCP surpasses state-of-the-art cooperative perception solutions while preserving the performance of standalone perception solutions. The source code can be found at https://github.com/DarrenQu/SiCP.

NeurIPS Conference 2024 Conference Paper

TAPTRv2: Attention-based Position Update Improves Tracking Any Point

• Hongyang Li
• Hao Zhang
• Shilong Liu
• Zhaoyang Zeng
• Feng Li
• Tianhe Ren
• Bohan Li
• Lei Zhang

In this paper, we present TAPTRv2, a Transformer-based approach built upon TAPTR for solving the Tracking Any Point (TAP) task. TAPTR borrows designs from DEtection TRansformer (DETR) and formulates each tracking point as a point query, making it possible to leverage well-studied operations in DETR-like algorithms. TAPTRv2 improves TAPTR by addressing a critical issue regarding its reliance on cost-volume, which contaminates the point query’s content feature and negatively impacts both visibility prediction and cost-volume computation. In TAPTRv2, we propose a novel attention-based position update (APU) operation and use key-aware deformable attention to realize. For each query, this operation uses key-aware attention weights to combine their corresponding deformable sampling positions to predict a new query position. This design is based on the observation that local attention is essentially the same as cost-volume, both of which are computed by dot-production between a query and its surrounding features. By introducing this new operation, TAPTRv2 not only removes the extra burden of cost-volume computation, but also leads to a substantial performance improvement. TAPTRv2 surpasses TAPTR and achieves state-of-the-art performance on many challenging datasets, demonstrating the effectiveness of our approach.

NeurIPS Conference 2024 Conference Paper

Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model

• Hao Zhang
• Lei Cao
• Jaiyi Ma

Existing multi-modal image fusion methods fail to address the compound degradations presented in source images, resulting in fusion images plagued by noise, color bias, improper exposure, etc. Additionally, these methods often overlook the specificity of foreground objects, weakening the salience of the objects of interest within the fused images. To address these challenges, this study proposes a novel interactive multi-modal image fusion framework based on the text-modulated diffusion model, called Text-DiFuse. First, this framework integrates feature-level information integration into the diffusion process, allowing adaptive degradation removal and multi-modal information fusion. This is the first attempt to deeply and explicitly embed information fusion within the diffusion process, effectively addressing compound degradation in image fusion. Second, by embedding the combination of the text and zero-shot location model into the diffusion fusion process, a text-controlled fusion re-modulation strategy is developed. This enables user-customized text control to improve fusion performance and highlight foreground objects in the fused images. Extensive experiments on diverse public datasets show that our Text-DiFuse achieves state-of-the-art fusion performance across various scenarios with complex degradation. Moreover, the semantic segmentation experiment validates the significant enhancement in semantic performance achieved by our text-controlled fusion re-modulation strategy. The code is publicly available at https: //github. com/Leiii-Cao/Text-DiFuse.

IROS Conference 2024 Conference Paper

Tracking Control with Uncertainty Smoothing Estimation under Aggressive Maneuvers of Aerial Vehicles

• Hao Zhang
• Tao Jiang 0018
• Jianchuan Ye
• Senqi Tan
• Zhi Zheng

Aggressive maneuvering is crucial for aerial vehicles to execute adversarial and penetration missions. However, this challenges the accurate tracking control of drones due to uncertainties induced by high-speed flight. Therefore, firstly, a highly dynamic tracking control framework is proposed to actualize the accurate tracking of aggressive trajectories with velocities up to 15 m/s (i. e. , 54 km/h) and acceleration of 2 g. Secondly, in order to mitigate the impact of conjoint effects on uncertainty estimation during aggressive flights and to ensure that uncertainty is smoothly compensated, a novel adaptive nonlinear extended state observer (ANESO) with noise suppression and peak attenuation capabilities is designed. Finally, extensive comparative simulation and real-world practical experimental results certify the superiority of the proposed control strategy in tracking aggressive trajectories.

JBHI Journal 2024 Journal Article

TSGET: Two-Stage Global Enhanced Transformer for Automatic Radiology Report Generation

• Xiulong Yi
• You Fu
• Ruiqing Liu
• Hao Zhang
• Rong Hua

Recently, automatic radiology report generation, which targets to generate multiple sentences that can accurately describe medical observations for given X-ray images, has gained increasing attention. Existing methods commonly employ the attention mechanism for accurate word generation. However, such attention-based methods fail to leverage useful image-level global features, thereby limiting the model's reasoning ability. To tackle this challenge, we propose two-stage global enhancement layers to facilitate the Transformer to generate more reliable reports from a global perspective. Specifically, the 1st Global Enhancement Layer (1st GEL) is designed to capture the global visual context features by establishing the relationships between image-level global features and previously generated words. The 2nd Global Enhancement Layer (2nd GEL) is devised to capture the region-global level features by building the relationships between image-level global features and region-level information. The experiments demonstrate that by integrating the aforementioned two-stage global enhancement layers into the Transformer model, our proposal achieves state-of-the-art (SOTA) performance on various Natural Language Generation (NLG) evaluation metrics. Further Clinical Efficacy (CE) evaluations also validate that our proposal is able to predict more critical information.

NeurIPS Conference 2024 Conference Paper

UnSeg: One Universal Unlearnable Example Generator is Enough against All Image Segmentation

• Ye Sun
• Hao Zhang
• Tiehua Zhang
• Xingjun Ma
• Yu-Gang Jiang

Image segmentation is a crucial vision task that groups pixels within an image into semantically meaningful segments, which is pivotal in obtaining a fine-grained understanding of real-world scenes. However, an increasing privacy concern exists regarding training large-scale image segmentation models on unauthorized private data. In this work, we exploit the concept of unlearnable examples to make images unusable to model training by generating and adding unlearnable noise into the original images. Particularly, we propose a novel Unlearnable Segmentation (UnSeg) framework to train a universal unlearnable noise generator that is capable of transforming any downstream images into their unlearnable version. The unlearnable noise generator is finetuned from the Segment Anything Model (SAM) via bilevel optimization on an interactive segmentation dataset towards minimizing the training error of a surrogate model that shares the same architecture with SAM (but trains from scratch). We empirically verify the effectiveness of UnSeg across 6 mainstream image segmentation tasks, 10 widely used datasets, and 7 different network architectures, and show that the unlearnable images can reduce the segmentation performance by a large margin. Our work provides useful insights into how to leverage foundation models in a data-efficient and computationally affordable manner to protect images against image segmentation models.

NeurIPS Conference 2024 Conference Paper

Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation

• Ruihao Xia
• Yu Liang
• Peng-tao Jiang
• Hao Zhang
• Bo Li
• Yang Tang
• Pan Zhou

Despite their success, unsupervised domain adaptation methods for semantic segmentation primarily focus on adaptation between image domains and do not utilize other abundant visual modalities like depth, infrared and event. This limitation hinders their performance and restricts their application in real-world multimodal scenarios. To address this issue, we propose Modality Adaptation with text-to-image Diffusion Models (MADM) for semantic segmentation task which utilizes text-to-image diffusion models pre-trained on extensive image-text pairs to enhance the model's cross-modality capabilities. Specifically, MADM comprises two key complementary components to tackle major challenges. First, due to the large modality gap, using one modal data to generate pseudo labels for another modality suffers from a significant drop in accuracy. To address this, MADM designs diffusion-based pseudo-label generation which adds latent noise to stabilize pseudo-labels and enhance label accuracy. Second, to overcome the limitations of latent low-resolution features in diffusion models, MADM introduces the label palette and latent regression which converts one-hot encoded labels into the RGB form by palette and regresses them in the latent space, thus ensuring the pre-trained decoder for up-sampling to obtain fine-grained features. Extensive experimental results demonstrate that MADM achieves state-of-the-art adaptation performance across various modality tasks, including images to depth, infrared, and event modalities. We open-source our code and models at https: //github. com/XiaRho/MADM.

TMLR Journal 2023 Journal Article

Contrastive Attraction and Contrastive Repulsion for Representation Learning

• Huangjie Zheng
• Xu Chen
• Jiangchao Yao
• Hongxia Yang
• Chunyuan Li
• Ya Zhang
• Hao Zhang
• Ivor Tsang

Contrastive learning (CL) methods effectively learn data representations in a self-supervision manner, where the encoder contrasts each positive sample over multiple negative samples via a one-vs-many softmax cross-entropy loss. By leveraging large amounts of unlabeled image data, recent CL methods have achieved promising results when pretrained on large-scale datasets, such as ImageNet. However, most of them consider the augmented views from the same instance are positive pairs, while views from other instances are negative ones. Such binary partition insufficiently considers the relation between samples and tends to yield worse performance when generalized on images in the wild. In this paper, to further improve the performance of CL and enhance its robustness on various datasets, we propose a doubly CL strategy that contrasts positive samples and negative ones within themselves separately. We realize this strategy with contrastive attraction and contrastive repulsion (CACR), which makes the query not only exert a greater force to attract more distant positive samples but also do so to repel closer negative samples. Theoretical analysis reveals that CACR generalizes CL's behavior by positive attraction and negative repulsion. It further considers the intra-contrastive relation within the positive and negative pairs to narrow the gap between the sampled and true distribution, which is important when datasets are less curated. Extensive large-scale experiments on standard vision tasks show that CACR not only consistently outperforms existing CL methods on benchmark datasets, but also shows better robustness when generalized on imbalanced image datasets.

NeurIPS Conference 2023 Conference Paper

D$^2$CSG: Unsupervised Learning of Compact CSG Trees with Dual Complements and Dropouts

• Fenggen Yu
• Qimin Chen
• Maham Tanveer
• Ali Mahdavi Amiri
• Hao Zhang

We present D$^2$CSG, a neural model composed of two dual and complementary network branches, with dropouts, for unsupervised learning of compact constructive solid geometry (CSG) representations of 3D CAD shapes. Our network is trained to reconstruct a 3D shape by a fixed-order assembly of quadric primitives, with both branches producing a union of primitive intersections or inverses. A key difference between D$^2$CSG and all prior neural CSG models is its dedicated residual branch to assemble the potentially complex shape complement, which is subtracted from an overall shape modeled by the cover branch. With the shape complements, our network is provably general, while the weight dropout further improves compactness of the CSG tree by removing redundant primitives. We demonstrate both quantitatively and qualitatively that D$^2$CSG produces compact CSG reconstructions with superior quality and more natural primitives than all existing alternatives, especially over complex and high-genus CAD shapes.

NeurIPS Conference 2023 Conference Paper

De novo Drug Design using Reinforcement Learning with Multiple GPT Agents

• Xiuyuan Hu
• Guoqing Liu
• Yang Zhao
• Hao Zhang

De novo drug design is a pivotal issue in pharmacology and a new area of focus in AI for science research. A central challenge in this field is to generate molecules with specific properties while also producing a wide range of diverse candidates. Although advanced technologies such as transformer models and reinforcement learning have been applied in drug design, their potential has not been fully realized. Therefore, we propose MolRL-MGPT, a reinforcement learning algorithm with multiple GPT agents for drug molecular generation. To promote molecular diversity, we encourage the agents to collaborate in searching for desirable molecules in diverse directions. Our algorithm has shown promising results on the GuacaMol benchmark and exhibits efficacy in designing inhibitors against SARS-CoV-2 protein targets. The codes are available at: https: //github. com/HXYfighter/MolRL-MGPT.

EAAI Journal 2023 Journal Article

Deep learning framework for solving Fokker–Planck equations with low-rank separation representation

• Hao Zhang
• Yong Xu
• Qi Liu
• Yongge Li

An insightful deep learning framework is proposed to solve the well-known Fokker–Planck (FP) equations that quantify the evolution of the probability density function. It efficiently reduces the demand of training data in acquiring precise integrations of special normalization conditions via neural network (NN). Instead of all hypercubic discrete points, the inputs of each NN only require one-dimensional discrete data, and this also avoids the exponential increase in training data as the dimension increase. Without loss of generality, to solve a d -dimensional FP equation, d NNs are employed and assembled into a low-rank separation representation. The FP equation, boundary conditions, and integral operators are then re-expressed in the sense of the separation representation. It enables the constructed loss function to perform simple vector operations, in that complicated d -dimensional operators are replaced by a set of one dimensional operators. A tractable strategy is presented for the selection of separation rank inspired by the potential function of the given system, although selecting an appropriate separation rank is still an open issue. Typical numerical examples reveal that the proposed algorithm is effective and superior for solving FP equations. The suggested framework could be applied and extended in various areas of engineering and applied sciences.

AAAI Conference 2023 Conference Paper

Differentially Private Nonlinear Causal Discovery from Numerical Data

• Hao Zhang
• Yewei Xia
• Yixin Ren
• Jihong Guan
• Shuigeng Zhou

Recently, several methods such as private ANM, EM-PC and Priv-PC have been proposed to perform differentially private causal discovery in various scenarios including bivariate, multivariate Gaussian and categorical cases. However, there is little effort on how to conduct private nonlinear causal discovery from numerical data. This work tries to challenge this problem. To this end, we propose a method to infer nonlinear causal relations from observed numerical data by using regression-based conditional independence test (RCIT) that consists of kernel ridge regression (KRR) and Hilbert-Schmidt independence criterion (HSIC) with permutation approximation. Sensitivity analysis for RCIT is given and a private constraint-based causal discovery framework with differential privacy guarantee is developed. Extensive simulations and real-world experiments for both conditional independence test and causal discovery are conducted, which show that our method is effective in handling nonlinear numerical cases and easy to implement. The source code of our method and data are available at https://github.com/Causality-Inference/PCD.

NeurIPS Conference 2023 Conference Paper

DiViNeT: 3D Reconstruction from Disparate Views using Neural Template Regularization

• Aditya Vora
• Akshay Gadi Patil
• Hao Zhang

We present a volume rendering-based neural surface reconstruction method that takes as few as three disparate RGB images as input. Our key idea is to regularize the reconstruction, which is severely ill-posed and leaving significant gaps between the sparse views, by learning a set of neural templates that act as surface priors. Our method, coined DiViNet, operates in two stages. The first stage learns the templates, in the form of 3D Gaussian functions, across different scenes, without 3D supervision. In the reconstruction stage, our predicted templates serve as anchors to help “stitch” the surfaces over sparse regions. We demonstrate that our approach is not only able to complete the surface geometry but also reconstructs surface details to a reasonable extent from few disparate input views. On the DTU and BlendedMVS datasets, our approach achieves the best reconstruction quality among existing methods in the presence of such sparse views and performs on par, if not better, with competing methods when dense views are employed as inputs.

AAAI Conference 2023 Conference Paper

DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding

• Shilong Liu
• Shijia Huang
• Feng Li
• Hao Zhang
• Yaoyuan Liang
• Hang Su
• Jun Zhu
• Lei Zhang

In this paper, we study the problem of visual grounding by considering both phrase extraction and grounding (PEG). In contrast to the previous phrase-known-at-test setting, PEG requires a model to extract phrases from text and locate objects from image simultaneously, which is a more practical setting in real applications. As phrase extraction can be regarded as a 1D text segmentation problem, we formulate PEG as a dual detection problem and propose a novel DQ-DETR model, which introduces dual queries to probe different features from image and text for object prediction and phrase mask prediction. Each pair of dual queries are designed to have shared positional parts but different content parts. Such a design effectively alleviates the difficulty of modality alignment between image and text (in contrast to a single query design) and empowers Transformer decoder to leverage phrase mask-guided attention to improve the performance. To evaluate the performance of PEG, we also propose a new metric CMAP (cross-modal average precision), analogous to the AP metric in object detection. The new metric overcomes the ambiguity of Recall@1 in many-box-to-one-phrase cases in phrase grounding. As a result, our PEG pre-trained DQ-DETR establishes new state-of-the-art results on all visual grounding benchmarks with a ResNet-101 backbone. For example, it achieves 91.04% and 83.51% in terms of recall rate on RefCOCO testA and testB with a ResNet-101 backbone.

NeurIPS Conference 2023 Conference Paper

FaceDNeRF: Semantics-Driven Face Reconstruction, Prompt Editing and Relighting with Diffusion Models

• Hao Zhang
• Tianyuan DAI
• Yanbo Xu
• Yu-Wing Tai
• Chi-Keung Tang

The ability to create high-quality 3D faces from a single image has become increasingly important with wide applications in video conferencing, AR/VR, and advanced video editing in movie industries. In this paper, we propose Face Diffusion NeRF (FaceDNeRF), a new generative method to reconstruct high-quality Face NeRFs from single images, complete with semantic editing and relighting capabilities. FaceDNeRF utilizes high-resolution 3D GAN inversion and expertly trained 2D latent-diffusion model, allowing users to manipulate and construct Face NeRFs in zero-shot learning without the need for explicit 3D data. With carefully designed illumination and identity preserving loss, as well as multi-modal pre-training, FaceDNeRF offers users unparalleled control over the editing process enabling them to create and edit face NeRFs using just single-view images, text prompts, and explicit target lighting. The advanced features of FaceDNeRF have been designed to produce more impressive results than existing 2D editing approaches that rely on 2D segmentation maps for editable attributes. Experiments show that our FaceDNeRF achieves exceptionally realistic results and unprecedented flexibility in editing compared with state-of-the-art 3D face reconstruction and editing methods. Our code will be available at https: //github. com/BillyXYB/FaceDNeRF.

ICML Conference 2023 Conference Paper

FedCR: Personalized Federated Learning Based on Across-Client Common Representation with Conditional Mutual Information Regularization

• Hao Zhang
• Chenglin Li
• Wenrui Dai
• Junni Zou
• Hongkai Xiong

In personalized federated learning (PFL), multiple clients train customized models to fulfill their personal objectives, which, however, are prone to overfitting to local data due to the heterogeneity and scarcity of local data. To address this, we propose from the information-theoretic perspective a personalized federated learning framework based on the common representation learned across clients, named FedCR. Specifically, we introduce to the local client update a regularizer that aims at minimizing the discrepancy between local and global conditional mutual information (CMI), such that clients are encouraged to learn and exploit the common representation. Upon this, each client learns individually a customized predictor (head), while the extractor (body) remains to be aggregated by the server. Our CMI regularizer leads to a theoretically sound alignment between the local and global stochastic feature distributions in terms of their Kullback-Leibler (KL) divergence. More importantly, by modeling the global joint feature distribution as a product of multiple local feature distributions, clients can efficiently extract diverse information from the global data but without need of the raw data from other clients. We further show that noise injection via feature alignment and ensemble of local predictors in FedCR would help enhance its generalization capability. Experiments on benchmark datasets demonstrate a consistent performance gain and better generalization behavior of FedCR.

NeurIPS Conference 2023 Conference Paper

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

• Lianmin Zheng
• Wei-Lin Chiang
• Ying Sheng
• Siyuan Zhuang
• Zhanghao Wu
• Yonghao Zhuang
• Zi Lin
• Zhuohan Li

Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions. We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and propose solutions to mitigate some of them. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform. Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80\% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain. Additionally, we show our benchmark and traditional benchmarks complement each other by evaluating several variants of LLaMA and Vicuna. The MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available at https: //github. com/lm-sys/FastChat/tree/main/fastchat/llm_judge.

EAAI Journal 2023 Journal Article

mmSignature: Semi-supervised human identification system based on millimeter wave radar

• Yicheng Yao
• Hao Zhang
• Pan Xia
• Changyu Liu
• Fanglin Geng
• Zhongrui Bai
• Lidong Du
• Xianxiang Chen

Human identification is vital in health monitoring, human-computer interaction, safety detection, and other fields. Compared with traditional vision-based methods, millimeter wave radar sensors can protect users' privacy and work in dark environments, which has a wide range of application prospects in iot fields such as smart homes and smart medical care. Previous studies need to manually collect labeled data, which makes the data collection work need substantial human resources and is unsuitable for popularization and application. We automatically collect multi-modal radar signals in users ' daily lives without requiring researchers to label data manually. Based on the proposed data collection method, we established the first semi-supervised data set for human identification, which includes synchronous radar point cloud data and range-velocity map data. The dataset contains four experiments, including ten monitoring users and ten other users. We propose a semi-supervised co-training framework based on multi-modal data fusion for human identification. The framework guides the models to learn from unlabeled data using the complementary characteristics of point cloud data and range-velocity map data. In addition, we propose an information fusion method to fuse the radar data of two modes to further improve the model's performance. The experimental results show that the proposed method achieves 93. 7% human identification accuracy, showing radar-based human identification technology's application and promotion potential.

AAAI Conference 2023 Conference Paper

Multi-Level Wavelet Mapping Correlation for Statistical Dependence Measurement: Methodology and Performance

• Yixin Ren
• Hao Zhang
• Yewei Xia
• Jihong Guan
• Shuigeng Zhou

We propose a new criterion for measuring dependence between two real variables, namely, Multi-level Wavelet Mapping Correlation (MWMC). MWMC can capture the nonlinear dependencies between variables by measuring their correlation under different levels of wavelet mappings. We show that the empirical estimate of MWMC converges exponentially to its population quantity. To support independence test better with MWMC, we further design a permutation test based on MWMC and prove that our test can not only control the type I error rate (the rate of false positives) well but also ensure that the type II error rate (the rate of false negatives) is upper bounded by O(1/n) (n is the sample size) with finite permutations. By extensive experiments on (conditional) independence tests and causal discovery, we show that our method outperforms existing independence test methods.

ICRA Conference 2023 Conference Paper

Reinforcement Learning Based Pushing and Grasping Objects from Ungraspable Poses

• Hao Zhang
• Hongzhuo Liang
• Lin Cong
• Jianzhi Lyu
• Long Zeng 0001
• Pingfa Feng
• Jianwei Zhang 0001

Grasping an object when it is in an ungraspable pose is a challenging task, such as books or other large flat objects placed horizontally on a table. Inspired by human manipulation, we address this problem by pushing the object to the edge of the table and then grasping it from the hanging part. In this paper, we develop a model-free Deep Reinforcement Learning framework to synergize pushing and grasping actions. We first pre-train a Variational Autoencoder to extract high-dimensional features of input scenario images. One Proximal Policy Optimization algorithm with the common reward and sharing layers of Actor-Critic is employed to learn both pushing and grasping actions with high data efficiency. Experiments show that our one network policy can converge 2. 5 times faster than the policy using two parallel networks. Moreover, the experiments on unseen objects show that our policy can generalize to the challenging case of objects with curved surfaces and off-center irregularly shaped objects. Lastly, our policy can be transferred to a real robot without fine-tuning by using CycleGAN for domain adaption and outperforms the push-to-wall baseline.

NeurIPS Conference 2023 Conference Paper

Segment Everything Everywhere All at Once

• Xueyan Zou
• Jianwei Yang
• Hao Zhang
• Feng Li
• Linjie Li
• Jianfeng Wang
• Lijuan Wang
• Jianfeng Gao

In this work, we present SEEM, a promotable and interactive model for segmenting everything everywhere all at once in an image. In SEEM, we propose a novel and versatile decoding mechanism that enables diverse prompting for all types of segmentation tasks, aiming at a universal interface that behaves like large language models (LLMs). More specifically, SEEM is designed with four desiderata: i) Versatility. We introduce a new visual prompt to unify different spatial queries including points, boxes, scribbles, and masks, which can further generalize to a different referring image; ii) Compositionality. We learn a joint visual-semantic space between text and visual prompts, which facilitates the dynamic composition of two prompt types required for various segmentation tasks, as shown in Fig. 1; iii) Interactivity. We further incorporate learnable memory prompts into the decoder to retain segmentation history through mask-guided cross-attention from the decoder to image features; iv) Semantic awareness. We use a text encoder to encode text queries and mask labels into the same semantic space for open-vocabulary segmentation. We conduct a comprehensive empirical study to validate the effectiveness of SEEM across diverse segmentation tasks. The results demonstrate that SEEM exhibits robust generalizing to unseen user intents as it learns to compose prompts of different types in a unified representation space. Our approach achieves competitive performance on interactive segmentation, generic segmentation, referring segmentation, and video object segmentation on 9 datasets with minimum 1/100 supervision in a single set of weights.

ICLR Conference 2023 Conference Paper

SoftZoo: A Soft Robot Co-design Benchmark For Locomotion In Diverse Environments

• Tsun-Hsuan Wang
• Pingchuan Ma 0002
• Andrew Spielberg
• Zhou Xian
• Hao Zhang
• Joshua B. Tenenbaum
• Daniela Rus
• Chuang Gan 0001

While significant research progress has been made in robot learning for control, unique challenges arise when simultaneously co-optimizing morphology. Existing work has typically been tailored for particular environments or representations. In order to more fully understand inherent design and performance tradeoffs and accelerate the development of new breeds of soft robots, a comprehensive virtual platform — with well-established tasks, environments, and evaluation metrics — is needed. In this work, we introduce SoftZoo, a soft robot co-design platform for locomotion in diverse environments. SoftZoo supports an extensive, naturally-inspired material set, including the ability to simulate environments such as flat ground, desert, wetland, clay, ice, snow, shallow water, and ocean. Further, it provides a variety of tasks relevant for soft robotics, including fast locomotion, agile turning, and path following, as well as differentiable design representations for morphology and control. Combined, these elements form a feature-rich platform for analysis and development of soft robot co-design algorithms. We benchmark prevalent representations and co-design algorithms, and shed light on 1) the interplay between environment, morphology, and behavior (2) the importance of design space representations 3) the ambiguity in muscle formation and controller synthesis and 4) the value of differentiable physics. We envision that SoftZoo will serve as a standard platform and template an approach toward the development of novel representations and algorithms for co-designing soft robots’ behavioral and morphological intelligence. Demos are available on our project page.

EAAI Journal 2023 Journal Article

SPSNet: Boosting 3D point-based object detectors with stable point sampling

• Ao Liang
• Hao Zhang
• Haiyang Hua
• Whenyu Chen
• Huaici Zhao

Currently, the primary focus of 3D point-based detectors is to enhance their performance by primarily increasing the recall rate of foreground points at the semantic level. However, this approach is suboptimal. On one hand, ground-truth annotations of 3D bounding boxes are inherently ambiguous due to occlusions, signal distortion, or manual annotation errors, which can obscure genuine semantic information. On the other hand, the saliency of each point varies. In this paper, we propose SPSNet based on Bayes theorem to explore the mapping relationship between the sampled point and the corresponding bounding box, which we call point stability. This is a plug-and-play module and used as the basis for downsampling instead of relying on semantic information obtained through hard supervision of 3D bounding boxes. We incorporate the proposed method into the most popular point-based detector, IA-SSD, called SPSNet-IA. On the challenging KITTI validation set, with only changes to the downsampling strategy, it obtains 1. 18, 2. 78 and 2. 96 gains for the metric of A P 3 D on the easy, moderate and hard levels, respectively. In addition, SPSNet-IA outperforms all published point-based approaches by a large margin and ranks 1st among single-modal methods. We also design qualitative and quantitative experiments to explain the meaning of stability learned from SPSNet. These additional experiments demonstrate that SPSNet is not only a performance-enhancing model but also a saliency analysis model. The code is available at https: //github. com/AlanLiangC/SPSNet. git.

NeurIPS Conference 2022 Conference Paper

A Variational Edge Partition Model for Supervised Graph Representation Learning

• Yilin He
• Chaojie Wang
• Hao Zhang
• Bo Chen
• Mingyuan Zhou

Graph neural networks (GNNs), which propagate the node features through the edges and learn how to transform the aggregated features under label supervision, have achieved great success in supervised feature extraction for both node-level and graph-level classification tasks. However, GNNs typically treat the graph structure as given and ignore how the edges are formed. This paper introduces a graph generative process to model how the observed edges are generated by aggregating the node interactions over a set of overlapping node communities, each of which contributes to the edges via a logical OR mechanism. Based on this generative model, we partition each edge into the summation of multiple community-specific weighted edges and use them to define community-specific GNNs. A variational inference framework is proposed to jointly learn a GNN-based inference network that partitions the edges into different communities, these community-specific GNNs, and a GNN-based predictor that combines community-specific GNNs for the end classification task. Extensive evaluations on real-world graph datasets have verified the effectiveness of the proposed method in learning discriminative representations for both node-level and graph-level classification tasks.

NeurIPS Conference 2022 Conference Paper

AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness

• Dacheng Li
• Hongyi Wang
• Eric Xing
• Hao Zhang

Scaling up model sizes can lead to fundamentally new capabilities in many machine learning (ML) tasks. However, training big models requires strong distributed system expertise to carefully design model-parallel execution strategies that suit the model architectures and cluster setups. In this paper, we develop AMP, a framework that automatically derives such strategies. AMP identifies a valid space of model parallelism strategies and efficiently searches the space for high-performed strategies, by leveraging a cost model designed to capture the heterogeneity of the model and cluster specifications. Unlike existing methods, AMP is specifically tailored to support complex models composed of uneven layers and cluster setups with more heterogeneous accelerators and bandwidth. We evaluate AMP on popular modelsand cluster setups from public clouds and show that AMP returns parallel strategies that match the expert-tuned strategies on typical cluster setups. On heterogeneous clusters or models with heterogeneous architectures, AMP finds strategies with 1. 54$\times$ and 1. 77$\times$ higher throughput than state-of-the-art model-parallel systems, respectively.

IROS Conference 2022 Conference Paper

Behaviour Learning with Adaptive Motif Discovery and Interacting Multiple Model

• Hanqing Zhao
• Travis Manderson
• Hao Zhang
• Xue Liu 0004
• Gregory Dudek

We propose an approach that enables simultaneous interpretable learning of a high-level discrete behaviour and its low-level rhythmic sub-behaviour. We do this though a unified reward function, where a reward function that only describes low-level behaviour, with less impact on learning of other behaviours is recovered from few-shot motion demonstrations. To this end, we first extract local behaviour motifs from state-only human demonstrations and random driving samples using an adaptive motif discovery approach derived from the Matrix Profile algorithm. We then optimize parameters for motif discovery by maximizing the sum and entropy over motif sizes. Interacting Multiple Model (IMM) estimators are constructed on top of linear-Gaussian dynamics of discovered motifs, the cumulative distributions over motifs estimated by IMMs serve as the basis of the reward function. By combining the recovered reward with the terrain type signal gathered from the environment, we are able to train a dual-objective off-road vehicle controller that demonstrates both terrain selection and human-like driving behaviours. Compared with related approaches across 10 people, our rhythmic behaviour reward recovery approach enables the controller to produce higher preference over human driving demonstrations. In addition to performing more stable across different people with 87% less variance than the best baseline in rhythmic behaviour indicator, our method reduces the negative effects on higher-level behaviour learning while maintaining high interpretability at all stages of the algorithm.

NeurIPS Conference 2022 Conference Paper

GALOIS: Boosting Deep Reinforcement Learning via Generalizable Logic Synthesis

• Yushi Cao
• Zhiming Li
• Tianpei Yang
• Hao Zhang
• Yan Zheng
• Yi Li
• Jianye Hao
• Yang Liu

Despite achieving superior performance in human-level control problems, unlike humans, deep reinforcement learning (DRL) lacks high-order intelligence (e. g. , logic deduction and reuse), thus it behaves ineffectively than humans regarding learning and generalization in complex problems. Previous works attempt to directly synthesize a white-box logic program as the DRL policy, manifesting logic-driven behaviors. However, most synthesis methods are built on imperative or declarative programming, and each has a distinct limitation, respectively. The former ignores the cause-effect logic during synthesis, resulting in low generalizability across tasks. The latter is strictly proof-based, thus failing to synthesize programs with complex hierarchical logic. In this paper, we combine the above two paradigms together and propose a novel Generalizable Logic Synthesis (GALOIS) framework to synthesize hierarchical and strict cause-effect logic programs. GALOIS leverages the program sketch and defines a new sketch-based hybrid program language for guiding the synthesis. Based on that, GALOIS proposes a sketch-based program synthesis method to automatically generate white-box programs with generalizable and interpretable cause-effect logic. Extensive evaluations on various decision-making tasks with complex logic demonstrate the superiority of GALOIS over mainstream baselines regarding the asymptotic performance, generalizability, and great knowledge reusability across different environments.

AAAI Conference 2022 Conference Paper

Hybrid Neural Networks for On-Device Directional Hearing

• Anran Wang
• Maruchi Kim
• Hao Zhang
• Shyamnath Gollakota

On-device directional hearing requires audio source separation from a given direction while achieving stringent humanimperceptible latency requirements. While neural nets can achieve significantly better performance than traditional beamformers, all existing models fall short of supporting low-latency causal inference on computationally-constrained wearables. We present HybridBeam, a hybrid model that combines traditional beamformers with a custom lightweight neural net. The former reduces the computational burden of the latter and also improves its generalizability, while the latter is designed to further reduce the memory and computational overhead to enable real-time and low-latency operations. Our evaluation shows comparable performance to stateof-the-art causal inference models on synthetic data while achieving a 5x reduction of model size, 4x reduction of computation per second, 5x reduction in processing time and generalizing better to real hardware data. Further, our real-time hybrid model runs in 8 ms on mobile CPUs designed for lowpower wearable devices and achieves an end-to-end latency of 17. 5 ms.

AAAI Conference 2022 Conference Paper

Residual Similarity Based Conditional Independence Test and Its Application in Causal Discovery

• Hao Zhang
• Shuigeng Zhou
• Kun Zhang
• Jihong Guan

Recently, many regression based conditional independence (CI) test methods have been proposed to solve the problem of causal discovery. These methods provide alternatives to test CI by first removing the information of the controlling set from the two target variables, and then testing the independence between the corresponding residuals Res1 and Res2. When the residuals are linearly uncorrelated, the independence test between them is nontrivial. With the ability to calculate inner product in high-dimensional space, kernel-based methods are usually used to achieve this goal, but still consume considerable time. In this paper, we investigate the independence between two linear combinations under linear non-Gaussian structural equation model. We show that the dependence between the two residuals can be captured by the difference between the similarity of (Res1, Res2) and that of (Res1, Res3) (Res3 is generated by random permutation) in highdimensional space. With this result, we design a new method called SCIT for CI test, where permutation test is performed to control Type I error rate. The proposed method is simpler yet more efficient and effective than the existing ones. When applied to causal discovery, the proposed method outperforms the counterparts in terms of both speed and Type II error rate, especially in the case of small sample size, which is validated by our extensive experiments on various datasets.

NeurIPS Conference 2021 Conference Paper

A Prototype-Oriented Framework for Unsupervised Domain Adaptation

• Korawat Tanwisuth
• Xinjie Fan
• Huangjie Zheng
• Shujian Zhang
• Hao Zhang
• Bo Chen
• Mingyuan Zhou

Existing methods for unsupervised domain adaptation often rely on minimizing some statistical distance between the source and target samples in the latent space. To avoid the sampling variability, class imbalance, and data-privacy concerns that often plague these methods, we instead provide a memory and computation-efficient probabilistic framework to extract class prototypes and align the target features with them. We demonstrate the general applicability of our method on a wide range of scenarios, including single-source, multi-source, class-imbalance, and source-private domain adaptation. Requiring no additional model parameters and having a moderate increase in computation over the source model alone, the proposed method achieves competitive performance with state-of-the-art methods.

AAAI Conference 2021 Conference Paper

Ada-Segment: Automated Multi-loss Adaptation for Panoptic Segmentation

• Gengwei Zhang
• Yiming Gao
• Hang Xu
• Hao Zhang
• Zhenguo Li
• Xiaodan Liang

Panoptic segmentation that unifies instance segmentation and semantic segmentation has recently attracted increasing attention. While most existing methods focus on designing novel architectures, we steer toward a different perspective: performing automated multi-loss adaptation (named Ada- Segment) on the fly to flexibly adjust multiple training losses over the course of training using a controller trained to capture the learning dynamics. This offers a few advantages: it bypasses manual tuning of the sensitive loss combination, a decisive factor for panoptic segmentation; it allows to explicitly model the learning dynamics, and reconcile the learning of multiple objectives (up to ten in our experiments); with an end-to-end architecture, it generalizes to different datasets without the need of re-tuning hyperparameters or readjusting the training process laboriously. Our Ada-Segment brings 2. 7% panoptic quality (PQ) improvement on COCO val split from the vanilla baseline, achieving the state-of-theart 48. 5% PQ on COCO test-dev split and 32. 9% PQ on ADE20K dataset. The extensive ablation studies reveal the ever-changing dynamics throughout the training process, necessitating the incorporation of an automated and adaptive learning strategy as presented in this paper.

AAAI Conference 2021 Conference Paper

Appearance-Motion Memory Consistency Network for Video Anomaly Detection

• Ruichu Cai
• Hao Zhang
• Wen Liu
• Shenghua Gao
• Zhifeng Hao

Abnormal event detection in the surveillance video is an essential but the challenging task and many methods have been proposed to deal with this problem. The previous methods either only considers the appearance information or directly integrate the results of appearance and motion information without considering their endogenous consistency semantic explicitly. Inspired by the rule that humans identify the abnormal frames from multi-modality signals, we propose an Appearance-Motion Memory Consistency Network (AMMC-Net). Our method first makes full use of the prior knowledge of appearance and motion signals to capture the correspondence between them in the high-level feature space explicitly. Then, it combines the multi-view features to obtain a more essential and robust feature representation of regular events, which can significantly increase the gap between an abnormal and a regular event. In the anomaly detection phase, we further introduce a commit error in the latent space joint with the prediction error in pixel space to enhance the detection accuracy. Solid experimental results on various standard datasets validate the effectiveness of our approach.

AAAI Conference 2021 Conference Paper

Building Interpretable Interaction Trees for Deep NLP Models

• Die Zhang
• Hao Zhang
• Huilin Zhou
• Xiaoyi Bao
• Da Huo
• Ruizhao Chen
• Xu Cheng
• Mengyue Wu

This paper proposes a method to disentangle and quantify interactions among words that are encoded inside a DNN for natural language processing. We construct a tree to encode salient interactions extracted by the DNN. Six metrics are proposed to analyze properties of interactions between constituents in a sentence. The interaction is defined based on Shapley values of words, which are considered as an unbiased estimation of word contributions to the network prediction. Our method is used to quantify word interactions encoded inside the BERT, ELMo, LSTM, CNN, and Transformer networks. Experimental results have provided a new perspective to understand these DNNs, and have demonstrated the effectiveness of our method.

AAAI Conference 2021 Conference Paper

GDPNet: Refining Latent Multi-View Graph for Relation Extraction

• Fuzhao Xue
• Aixin Sun
• Hao Zhang
• Eng Siong Chng

Relation Extraction (RE) is to predict the relation type of two entities that are mentioned in a piece of text, e. g. , a sentence or a dialogue. When the given text is long, it is challenging to identify indicative words for the relation prediction. Recent advances on RE task are from BERT-based sequence modeling and graph-based modeling of relationships among the tokens in the sequence. In this paper, we propose to construct a latent multi-view graph to capture various possible relationships among tokens. We then refine this graph to select important words for relation prediction. Finally, the representation of the refined graph and the BERT-based sequence representation are concatenated for relation extraction. Specifically, in our proposed GDPNet (Gaussian Dynamic Time Warping Pooling Net), we utilize Gaussian Graph Generator (GGG) to generate edges of the multi-view graph. The graph is then refined by Dynamic Time Warping Pooling (DTWPool). On DialogRE and TACRED, we show that GDPNet achieves the best performance on dialogue-level RE, and comparable performance with the state-of-the-arts on sentence-level RE. Our code is available at https: //github. com/XueFuzhao/GDPNet.

AAAI Conference 2021 Conference Paper

Interpreting Multivariate Shapley Interactions in DNNs

• Hao Zhang
• Yichen Xie
• Longjie Zheng
• Die Zhang
• Quanshi Zhang

This paper aims to explain deep neural networks (DNNs) from the perspective of multivariate interactions. In this paper, we define and quantify the significance of interactions among multiple input variables of the DNN. Input variables with strong interactions usually form a coalition and reflect prototype features, which are memorized and used by the DNN for inference. We define the significance of interactions based on the Shapley value, which is designed to assign the attribution value of each input variable to the inference. We have conducted experiments with various DNNs. Experimental results have demonstrated the effectiveness of the proposed method.

AAAI Conference 2021 Conference Paper

Testing Independence Between Linear Combinations for Causal Discovery

• Hao Zhang
• Kun Zhang
• Shuigeng Zhou
• Jihong Guan
• Ji Zhang

Recently, regression based conditional independence (CI) tests have been employed to solve the problem of causal discovery. These methods provide an alternative way to test for CI by transforming CI to independence between residuals. Generally, it is nontrivial to check for independence when these residuals are linearly uncorrelated. With the ability to represent high-order moments, kernel-based methods are usually used to achieve this goal, but at a cost of considerable time. In this paper, we investigate the independence between two linear combinations under linear non-Gaussian structural equation model (SEM). We show that generally the 1-st to 4-th moments of the two linear combinations contain enough information to infer whether or not they are independent. The proposed method provides a simpler but more effective way to measure CIs, with only calculating the 1-st to 4-th moments of the input variables. When applied to causal discovery, the proposed method outperforms kernel-based methods in terms of both speed and accuracy. which is validated by extensive experiments.

NeurIPS Conference 2020 Conference Paper

AutoSync: Learning to Synchronize for Data-Parallel Distributed Deep Learning

• Hao Zhang
• Yuan Li
• Zhijie Deng
• Xiaodan Liang
• Lawrence Carin
• Eric Xing

Synchronization is a key step in data-parallel distributed machine learning (ML). Different synchronization systems and strategies perform differently, and to achieve optimal parallel training throughput requires synchronization strategies that adapt to model structures and cluster configurations. Existing synchronization systems often only consider a single or a few synchronization aspects, and the burden of deciding the right synchronization strategy is then placed on the ML practitioners, who may lack the required expertise. In this paper, we develop a model- and resource-dependent representation for synchronization, which unifies multiple synchronization aspects ranging from architecture, message partitioning, placement scheme, to communication topology. Based on this representation, we build an end-to-end pipeline, AutoSync, to automatically optimize synchronization strategies given model structures and resource specifications, lowering the bar for data-parallel distributed ML. By learning from low-shot data collected in only 200 trial runs, AutoSync can discover synchronization strategies up to 1. 6x better than manually optimized ones. We develop transfer-learning mechanisms to further reduce the auto-optimization cost -- the simulators can transfer among similar model architectures, among similar cluster configurations, or both. We also present a dataset that contains over 10000 synchronization strategies and run-time pairs on a diverse set of models and cluster specifications.

NeurIPS Conference 2020 Conference Paper

Bidirectional Convolutional Poisson Gamma Dynamical Systems

• Wenchao Chen
• Chaojie Wang
• Bo Chen
• Yicheng Liu
• Hao Zhang
• Mingyuan Zhou

Incorporating the natural document-sentence-word structure into hierarchical Bayesian modeling, we propose convolutional Poisson gamma dynamical systems (PGDS) that introduce not only word-level probabilistic convolutions, but also sentence-level stochastic temporal transitions. With word-level convolutions capturing phrase-level topics and sentence-level transitions capturing how the topic usages evolve over consecutive sentences, we aggregate the topic proportions of all sentences of a document as its feature representation. To consider not only forward but also backward sentence-level information transmissions, we further develop a bidirectional convolutional PGDS to incorporate the full contextual information to represent each sentence. For efficient inference, we construct a convolutional-recurrent inference network, which provides both sentence-level and document-level representations, and introduce a hybrid Bayesian inference scheme combining stochastic-gradient MCMC and amortized variational inference. Experimental results on a variety of document corpora demonstrate that the proposed models can extract expressive multi-level latent representations, including interpretable phrase-level topics and sentence-level temporal transitions as well as discriminative document-level features, achieving state-of-the-art document categorization performance while being memory and computation efficient.

NeurIPS Conference 2020 Conference Paper

Deep Relational Topic Modeling via Graph Poisson Gamma Belief Network

• Chaojie Wang
• Hao Zhang
• Bo Chen
• Dongsheng Wang
• Zhengjue Wang
• Mingyuan Zhou

To analyze a collection of interconnected documents, relational topic models (RTMs) have been developed to describe both the link structure and document content, exploring their underlying relationships via a single-layer latent representation with limited expressive capability. To better utilize the document network, we first propose graph Poisson factor analysis (GPFA) that constructs a probabilistic model for interconnected documents and also provides closed-form Gibbs sampling update equations, moving beyond sophisticated approximate assumptions of existing RTMs. Extending GPFA, we develop a novel hierarchical RTM named graph Poisson gamma belief network (GPGBN), and further introduce two different Weibull distribution based variational graph auto-encoders for efficient model inference and effective network information aggregation. Experimental results demonstrate that our models extract high-quality hierarchical latent document representations, leading to improved performance over baselines on various graph analytic tasks.

AAAI Conference 2020 System Paper

Embedding High-Level Knowledge into DQNs to Learn Faster and More Safely

• Zihang Gao
• Fangzhen Lin
• Yi Zhou
• Hao Zhang
• Kaishun Wu
• Haodi Zhang

Deep reinforcement learning has been successfully applied in many decision making scenarios. However, the slow training process and difficulty in explaining limit its application. In this paper, we attempt to address some of these problems by proposing a framework of Rule-interposing Learning (RIL) that embeds knowledge into deep reinforcement learning. In this framework, the rules dynamically effect the training progress, and accelerate the learning. The embedded knowledge in form of rule not only improves learning efficiency, but also prevents unnecessary or disastrous explorations at early stage of training. Moreover, the modularity of the framework makes it straightforward to transfer high-level knowledge among similar tasks.

AAAI Conference 2020 Conference Paper

FDN: Feature Decoupling Network for Head Pose Estimation

• Hao Zhang
• Mengmeng Wang
• Yong Liu
• Yi Yuan

Head pose estimation from RGB images without depth information is a challenging task due to the loss of spatial information as well as large head pose variations in the wild. The performance of existing landmark-free methods remains unsatisfactory as the quality of estimated pose is inferior. In this paper, we propose a novel three-branch network architecture, termed as Feature Decoupling Network (FDN), a more powerful architecture for landmark-free head pose estimation from a single RGB image. In FDN, we first propose a feature decoupling (FD) module to explicitly learn the discriminative features for each pose angle by adaptively recalibrating its channel-wise responses. Besides, we introduce a crosscategory center (CCC) loss to constrain the distribution of the latent variable subspaces and thus we can obtain more compact and distinct subspaces. Extensive experiments on both in-the-wild and controlled environment datasets demonstrate that the proposed method outperforms other state-of-the-art methods based on a single RGB image and behaves on par with approaches based on multimodal input resources.

AAAI Conference 2020 Conference Paper

Long-Term Loop Closure Detection through Visual-Spatial Information Preserving Multi-Order Graph Matching

• Peng Gao
• Hao Zhang

Loop closure detection is a fundamental problem for simultaneous localization and mapping (SLAM) in robotics. Most of the previous methods only consider one type of information, based on either visual appearances or spatial relationships of landmarks. In this paper, we introduce a novel visual-spatial information preserving multi-order graph matching approach for long-term loop closure detection. Our approach constructs a graph representation of a place from an input image to integrate visual-spatial information, including visual appearances of the landmarks and the background environment, as well as the second and third-order spatial relationships between two and three landmarks, respectively. Furthermore, we introduce a new formulation that formulates loop closure detection as a multi-order graph matching problem to compute a similarity score directly from the graph representations of the query and template images, instead of performing conventional vectorbased image matching. We evaluate the proposed multi-order graph matching approach based on two public long-term loop closure detection benchmark datasets, including the St. Lucia and CMU-VL datasets. Experimental results have shown that our approach is effective for long-term loop closure detection and it outperforms the previous state-of-the-art methods.

AAAI Conference 2020 Conference Paper

Mastering Complex Control in MOBA Games with Deep Reinforcement Learning

• Deheng Ye
• Zhao Liu
• Mingfei Sun
• Bei Shi
• Peilin Zhao
• Hao Wu
• Hongsheng Yu
• Shaojie Yang

We study the reinforcement learning problem of complex action control in the Multi-player Online Battle Arena (MOBA) 1v1 games. This problem involves far more complicated state and action spaces than those of traditional 1v1 games, such as Go and Atari series, which makes it very difficult to search any policies with human-level performance. In this paper, we present a deep reinforcement learning framework to tackle this problem from the perspectives of both system and algorithm. Our system is of low coupling and high scalability, which enables efficient explorations at large scale. Our algorithm includes several novel strategies, including control dependency decoupling, action mask, target attention, and dualclip PPO, with which our proposed actor-critic network can be effectively trained in our system. Tested on the MOBA game Honor of Kings, the trained AI agents can defeat top professional human players in full 1v1 games.

NeurIPS Conference 2020 Conference Paper

PIE-NET: Parametric Inference of Point Cloud Edges

• Xiaogang Wang
• Yuelang Xu
• Kai Xu
• Andrea Tagliasacchi
• Bin Zhou
• Ali Mahdavi-Amiri
• Hao Zhang

We introduce an end-to-end learnable technique to robustly identify feature edges in 3D point cloud data. We represent these edges as a collection of parametric curves (i. e. ,~lines, circles, and B-splines). Accordingly, our deep neural network, coined PIE-NET, is trained for parametric inference of edges. The network relies on a "region proposal" architecture, where a first module proposes an over-complete collection of edge and corner points, and a second module ranks each proposal to decide whether it should be considered. We train and evaluate our method on the ABC dataset, a large dataset of CAD models, and compare our results to those produced by traditional (non-learning) processing pipelines, as well as a recent deep learning based edge detector (EC-NET). Our results significantly improve over the state-of-the-art from both a quantitative and qualitative standpoint.

AAAI Conference 2020 Conference Paper

Rethinking the Image Fusion: A Fast Unified Image Fusion Network based on Proportional Maintenance of Gradient and Intensity

• Hao Zhang
• Han Xu
• Yang Xiao
• Xiaojie Guo
• Jiayi Ma

In this paper, we propose a fast unified image fusion network based on proportional maintenance of gradient and intensity (PMGI), which can end-to-end realize a variety of image fusion tasks, including infrared and visible image fusion, multiexposure image fusion, medical image fusion, multi-focus image fusion and pan-sharpening. We unify the image fusion problem into the texture and intensity proportional maintenance problem of the source images. On the one hand, the network is divided into gradient path and intensity path for information extraction. We perform feature reuse in the same path to avoid loss of information due to convolution. At the same time, we introduce the pathwise transfer block to exchange information between different paths, which can not only pre-fuse the gradient information and intensity information, but also enhance the information to be processed later. On the other hand, we define a uniform form of loss function based on these two kinds of information, which can adapt to different fusion tasks. Experiments on publicly available datasets demonstrate the superiority of our PMGI over the state-of-the-art in terms of both visual effect and quantitative metric in a variety of fusion tasks. In addition, our method is faster compared with the state-of-the-art.

AAAI Conference 2020 Conference Paper

RoboCoDraw: Robotic Avatar Drawing with GAN-Based Style Transfer and Time-Efficient Path Optimization

• Tianying Wang
• Wei Qi Toh
• Hao Zhang
• Xiuchao Sui
• Shaohua Li
• Yong Liu
• Wei Jing

Robotic drawing has become increasingly popular as an entertainment and interactive tool. In this paper we present RoboCoDraw, a real-time collaborative robot-based drawing system that draws stylized human face sketches interactively in front of human users, by using the Generative Adversarial Network (GAN)-based style transfer and a Random-Key Genetic Algorithm (RKGA)-based path optimization. The proposed RoboCoDraw system takes a real human face image as input, converts it to a stylized avatar, then draws it with a robotic arm. A core component in this system is the Avatar- GAN proposed by us, which generates a cartoon avatar face image from a real human face. AvatarGAN is trained with unpaired face and avatar images only and can generate avatar images of much better likeness with human face images in comparison with the vanilla CycleGAN. After the avatar image is generated, it is fed to a line extraction algorithm and converted to sketches. An RKGA-based path optimization algorithm is applied to find a time-efficient robotic drawing path to be executed by the robotic arm. We demonstrate the capability of RoboCoDraw on various face images using a lightweight, safe collaborative robot UR5.

IJCAI Conference 2020 Conference Paper

Speeding up Very Fast Decision Tree with Low Computational Cost

• Jian Sun
• Hongyu Jia
• Bo Hu
• Xiao Huang
• Hao Zhang
• Hai Wan
• Xibin Zhao

Very Fast Decision Tree (VFDT) is one of the most widely used online decision tree induction algorithms, and it provides high classification accuracy with theoretical guarantees. In VFDT, the split-attempt operation is essential for leaf-split. It is computation-intensive since it computes the heuristic measure of all attributes of a leaf. To reduce split-attempts, VFDT tries to split at constant intervals (for example, every 200 examples). However, this mechanism introduces split-delay for split can only happen at fixed intervals, which slows down the growth of VFDT and finally lowers accuracy. To address this problem, we first devise an online incremental algorithm that computes the heuristic measure of an attribute with a much lower computational cost. Then a subset of attributes is carefully selected to find a potential split timing using this algorithm. A split-attempt will be carried out once the timing is verified. By the whole process, computational cost and split-delay are lowered significantly. Comprehensive experiments are conducted using multiple synthetic and real datasets. Compared with state-of-the-art algorithms, our method reduces split-attempts by about 5 to 10 times on average with much lower split-delay, which makes our algorithm run faster and more accurate.

TIST Journal 2019 Journal Article

Measuring Conditional Independence by Independent Residuals for Causal Discovery

• Hao Zhang
• Shuigeng Zhou
• Jihong Guan
• Jun (Luke) Huan

We investigate the relationship between conditional independence (CI) x ⫫ y | Z and the independence of two residuals x −E( x | Z )⫫ y −E( y | Z ), where x and y are two random variables and Z is a set of random variables. We show that if x, y, and Z are generated by following linear structural equation models and all external influences follow joint Gaussian distribution, then x ⫫ y | Z if and only if x −E( x | Z )⫫ y −E( y | Z ). That is, the test of x ⫫ y | Z can be relaxed to a simpler unconditional independence test of x −E( x | Z )⫫ y −E( y | Z ). Furthermore, testing x −E( x | Z )⫫ y −E( y | Z ) can be simplified by testing x −E( x | Z )⫫ y or y −E( y | Z )⫫ x. On the other side, if all these external influences follow non-Gaussian distributions and the model satisfies structural faithfulness condition, then we have x ⫫ y | Z ⇔ x −E( x | Z )⫫ y −E( y | Z ). We apply the results above to the causal discovery problem, where the causal directions are generally determined by a set of V -structures and their consistent propagations, so CI test-based methods can return a set of Markov equivalence classes. We show that in the linear non-Gaussian context, in many cases x −E( x | Z )⫫ z or y −E( y | Z )⫫ z (∀ z ∈ Z and Z is a minimal d -separator) is satisfied when x −E( x | Z )⫫ y −E( y | Z ), which implies z causes x (or y ) if z directly connects to x (or y ). Therefore, we conclude that CIs have useful information for distinguishing Markov equivalence classes. In summary, comparing with the existing discretization-based and kernel-based CI testing methods, the proposed method provides a simpler way to measure CI, which needs only one unconditional independence test and two regression operations. When being applied to causal discovery, it can find more causal relationships, which is extensively validated by experiments.

IJCAI Conference 2019 Conference Paper

Multiple Noisy Label Distribution Propagation for Crowdsourcing

• Hao Zhang
• Liangxiao Jiang
• Wenqiang Xu

Crowdsourcing services provide a fast, efficient, and cost-effective means of obtaining large labeled data for supervised learning. Ground truth inference, also called label integration, designs proper aggregation strategies to infer the unknown true label of each instance from the multiple noisy label set provided by ordinary crowd workers. However, to the best of our knowledge, nearly all existing label integration methods focus solely on the multiple noisy label set itself of the individual instance while totally ignoring the intercorrelation among multiple noisy label sets of different instances. To solve this problem, a multiple noisy label distribution propagation (MNLDP) method is proposed in this study. MNLDP first transforms the multiple noisy label set of each instance into its multiple noisy label distribution and then propagates its multiple noisy label distribution to its nearest neighbors. Consequently, each instance absorbs a fraction of the multiple noisy label distributions from its nearest neighbors and yet simultaneously maintains a fraction of its own original multiple noisy label distribution. Promising experimental results on simulated and real-world datasets validate the effectiveness of our proposed method.

AAAI Conference 2019 Conference Paper

Recursively Learning Causal Structures Using Regression-Based Conditional Independence Test

• Hao Zhang
• Shuigeng Zhou
• Chuanxu Yan
• Jihong Guan
• Xin Wang

This paper addresses two important issues in causality inference. One is how to reduce redundant conditional independence (CI) tests, which heavily impact the efficiency and accuracy of existing constraint-based methods. Another is how to construct the true causal graph from a set of Markov equivalence classes returned by these methods. For the first issue, we design a recursive decomposition approach where the original data (a set of variables) is first decomposed into three small subsets, each of which is then recursively decomposed into three smaller subsets until none of subsets can be decomposed further. Consequently, redundant CI tests can be reduced by inferring causality from these subsets. Advantage of this decomposition scheme lies in two aspects: 1) it requires only low-order CI tests, and 2) it does not violate d-separation. Thus, the complete causality can be reconstructed by merging all the partial results of the subsets. For the second issue, we employ regression-based conditional independence test to check CIs in linear non-Gaussian additive noise cases, which can identify more causal directions by x−E(x|Z)⊥z (or y−E(y|Z)⊥z). Therefore, causal direction learning is no longer limited by the number of returned Vstructures and the consistent propagation. Extensive experiments show that the proposed method can not only substantially reduce redundant CI tests but also effectively distinguish the equivalence classes, thus is superior to the state of the art constraint-based methods in causality inference.

AAAI Conference 2019 Conference Paper

Visual Place Recognition via Robust ℓ2-Norm Distance Based Holism and Landmark Integration

• Kai Liu
• Hua Wang
• Fei Han
• Hao Zhang

Visual place recognition is essential for large-scale simultaneous localization and mapping (SLAM). Long-term robot operations across different time of the days, months, and seasons introduce new challenges from significant environment appearance variations. In this paper, we propose a novel method to learn a location representation that can integrate the semantic landmarks of a place with its holistic representation. To promote the robustness of our new model against the drastic appearance variations due to long-term visual changes, we formulate our objective to use non-squared ℓ2-norm distances, which leads to a difficult optimization problem that minimizes the ratio of the ℓ2,1-norms of matrices. To solve our objective, we derive a new efficient iterative algorithm, whose convergence is rigorously guaranteed by theory. In addition, because our solution is strictly orthogonal, the learned location representations can have better place recognition capabilities. We evaluate the proposed method using two large-scale benchmark data sets, the CMU-VL and Nordland data sets. Experimental results have validated the effectiveness of our new method in long-term visual place recognition applications.

NeurIPS Conference 2018 Conference Paper

Deep Poisson gamma dynamical systems

• DanDan Guo
• Bo Chen
• Hao Zhang
• Mingyuan Zhou

We develop deep Poisson-gamma dynamical systems (DPGDS) to model sequentially observed multivariate count data, improving previously proposed models by not only mining deep hierarchical latent structure from the data, but also capturing both first-order and long-range temporal dependencies. Using sophisticated but simple-to-implement data augmentation techniques, we derived closed-form Gibbs sampling update equations by first backward and upward propagating auxiliary latent counts, and then forward and downward sampling latent variables. Moreover, we develop stochastic gradient MCMC inference that is scalable to very long multivariate count time series. Experiments on both synthetic and a variety of real-world data demonstrate that the proposed model not only has excellent predictive performance, but also provides highly interpretable multilayer latent structure to represent hierarchical and temporal information propagation.

JBHI Journal 2018 Journal Article

Estimating SpO 2 via Time-Efficient High-Resolution Harmonics Analysis and Maximum Likelihood Tracking

• Feiyi Fan
• Yuepeng Yan
• Kun Zhao
• Fei Long
• Hao Zhang

The accuracy of noninvasive oxygen saturation (SpO 2 ), which is defined by the measurements based on photoplethysmographic (PPG) signals, is intensively affected by motion artifacts (MAs) and low perfusion. This study introduces a novel approach called ESPRIT-MLT to measure SpO 2 when such interferences are present. In contrast to previous studies, the work focuses on the harmonic model of the PPG signal and the probability model of results from harmonic analysis. The optimized parametric ESPRIT method is applied to improve the accuracy of harmonic power estimation, and the maximum likelihood SpO 2 tracking (MLT) technique is proposed to track the most probable uncontaminated harmonic of heart rate frequency. We construct an evaluation platform for testing the proposed method via generated signals and subject tests. Compared with the nonparametric periodogram method, the probability of correct harmonics being found is improved by 18. 7% or 19. 7%, when the signal is contaminated by motion artifacts or affected by low perfusion, respectively. In comparison with the reference methods, the proposed ESPRIT-MLT method exhibits a lower average root mean square error (RMSE) (1. 17%) in the simulation using an MA-contaminated PPG signal, and a lower RMSE (2. 70%) in the simulation using an extremely low (0. 05%) perfusion index. A comprehensive subject test that consists of 4 activities and 20 subjects shows an average RMSE of 0. 84% ( $\pm$ 0. 44%). Furthermore, the time-efficiency is optimized to be adaptable with wearable devices. Therefore, the proposed method has potential in enhancing the performance of clinical pulse oximetry and wearable SpO 2 measurement devices for daily use.

AAAI Conference 2018 Conference Paper

Learning Integrated Holism-Landmark Representations for Long-Term Loop Closure Detection

• Fei Han
• Hua Wang
• Hao Zhang

Loop closure detection is a critical component of large-scale simultaneous localization and mapping (SLAM) in loopy environments. This capability is challenging to achieve in longterm SLAM, when the environment appearance exhibits significant long-term variations across various time of the day, months, and even seasons. In this paper, we introduce a novel formulation to learn an integrated long-term representation based upon both holistic and landmark information, which integrates two previous insights under a unified framework: (1) holistic representations outperform keypoint-based representations, and (2) landmarks as an intermediate representation provide informative cues to detect challenging locations. Our new approach learns the representation by projecting input visual data into a low-dimensional space, which preserves both the global consistency (to minimize representation error) and the local consistency (to preserve landmarks’ pairwise relationship) of the input data. To solve the formulated optimization problem, a new algorithm is developed with theoretically guaranteed convergence. Extensive experiments have been conducted using two large-scale public benchmark data sets, in which the promising performances have demonstrated the effectiveness of the proposed approach.

AAAI Conference 2018 Conference Paper

Measuring Conditional Independence by Independent Residuals: Theoretical Results and Application in Causal Discovery

• Hao Zhang
• Shuigeng Zhou
• Jihong Guan

We investigate the relationship between conditional independence (CI) x ⊥ y|Z and the independence of two residuals x − E(x|Z) ⊥ y − E(y|Z), where x and y are two random variables, and Z is a set of random variables. We show that if x, y and Z are generated by following linear structural equation model and all external influences follow Gaussian distributions, then x ⊥ y|Z if and only if x − E(x|Z) ⊥ y − E(y|Z). That is, the test of x ⊥ y|Z can be relaxed to a simpler unconditional independence test of x − E(x|Z) ⊥ y − E(y|Z). Furthermore, if all these external influences follow non-Gaussian distributions and the model satisfies structural faithfulness condition, then we have x ⊥ y|Z ⇔ x − E(x|Z) ⊥ y − E(y|Z). We apply the results above to the causal discovery problem, where the causal directions are generally determined by a set of V -structures and their consistent propagations, so CI test-based methods can return a set of Markov equivalence classes. We show that in linear non-Gaussian context, x − E(x|Z) ⊥ y − E(y|Z) ⇒ x − E(x|Z) ⊥ z or y − E(y|Z) ⊥ z (∀z ∈ Z) if Z is a minimal d-separator, which implies z causes x (or y) if z directly connects to x (or y). Therefore, we conclude that CIs have useful information for distinguishing Markov equivalence classes. In summary, compared with the existing discretization-based and kernel-based CI testing methods, the proposed method provides a simpler way to measure CI, which needs only one unconditional independence test and two regression operations. When being applied to causal discovery, it can find more causal relationships, which is experimentally validated.

NeurIPS Conference 2018 Conference Paper

Symbolic Graph Reasoning Meets Convolutions

• Xiaodan Liang
• Zhiting Hu
• Hao Zhang
• Liang Lin
• Eric Xing

Beyond local convolution networks, we explore how to harness various external human knowledge for endowing the networks with the capability of semantic global reasoning. Rather than using separate graphical models (e. g. CRF) or constraints for modeling broader dependencies, we propose a new Symbolic Graph Reasoning (SGR) layer, which performs reasoning over a group of symbolic nodes whose outputs explicitly represent different properties of each semantic in a prior knowledge graph. To cooperate with local convolutions, each SGR is constituted by three modules: a) a primal local-to-semantic voting module where the features of all symbolic nodes are generated by voting from local representations; b) a graph reasoning module propagates information over knowledge graph to achieve global semantic coherency; c) a dual semantic-to-local mapping module learns new associations of the evolved symbolic nodes with local representations, and accordingly enhances local features. The SGR layer can be injected between any convolution layers and instantiated with distinct prior graphs. Extensive experiments show incorporating SGR significantly improves plain ConvNets on three semantic segmentation tasks and one image classification task. More analyses show the SGR layer learns shared symbolic representations for domains/datasets with the different label set given a universal knowledge graph, demonstrating its superior generalization capability.

AAAI Conference 2017 Short Paper

A Computational Assessment Model for the Adaptive Level of Rehabilitation Exergames for the Elderly

• Hao Zhang
• Chunyan Miao
• Han Yu
• Cyril Leung

Rehabilitation exergames can engage the elderly in physical activities and help them recover part of their deteriorating capabilities. However, most existing exergames lack measures of how suitable they are to specific individuals. In this paper, we propose the Computational Person-Environment Fit model to evaluate the adaptability of the exergames to each individual elderly user.

AAAI Conference 2017 Conference Paper

Causal Discovery Using Regression-Based Conditional Independence Tests

• Hao Zhang
• Shuigeng Zhou
• Kun Zhang
• Jihong Guan

Conditional independence (CI) testing is an important tool in causal discovery. Generally, by using CI tests, a set of Markov equivalence classes w. r. t. the observed data can be estimated by checking whether each pair of variables x and y is d-separated, given a set of variables Z. Due to the curse of dimensionality, CI testing is often difficult to return a reliable result for high-dimensional Z. In this paper, we propose a regression-based CI test to relax the test of x ⊥ y|Z to simpler unconditional independence tests of x − f(Z) ⊥ y − g(Z), and x − f(Z) ⊥ Z or y − g(Z) ⊥ Z under the assumption that the data-generating procedure follows additive noise models (ANMs). When the ANM is identifiable, we prove that x − f(Z) ⊥ y − g(Z) ⇒ x ⊥ y|Z. We also show that 1) f and g can be easily estimated by regression, 2) our test is more powerful than the state-of-the-art kernel CI tests, and 3) existing causal learning algorithms can infer much more causal directions by using the proposed method.

ICRA Conference 2017 Conference Paper

Grasp quality evaluation and planning for objects with negative curvature

• Shuo Liu 0006
• Zhe Hu
• Hao Zhang
• Mingu Kwon
• Zhikang Wang
• Yi Xu
• Stefano Carpin

We consider the problem of grasping concave objects, i. e. , objects whose surface includes regions with negative curvature. When a multifingered hand is used to restrain these objects, these areas can be advantageously used to determine grasps capable of more robustly resisting to external disturbance wrenches. We propose a new grasp quality metric specifically suited for this case, and we use it to inform a grasp planner searching the space of possible grasps. Our findings are validated both in simulation and on a real robot system executing a bin picking task. Experimental validation shows that our method is more effective than those not explicitly considering negative curvature.

NeurIPS Conference 2017 Conference Paper

Structured Generative Adversarial Networks

• Zhijie Deng
• Hao Zhang
• Xiaodan Liang
• Luona Yang
• Shizhen Xu
• Jun Zhu
• Eric Xing

We study the problem of conditional generative modeling based on designated semantics or structures. Existing models that build conditional generators either require massive labeled instances as supervision or are unable to accurately control the semantics of generated samples. We propose structured generative adversarial networks (SGANs) for semi-supervised conditional generative modeling. SGAN assumes the data x is generated conditioned on two independent latent variables: y that encodes the designated semantics, and z that contains other factors of variation. To ensure disentangled semantics in y and z, SGAN builds two collaborative games in the hidden space to minimize the reconstruction error of y and z, respectively. Training SGAN also involves solving two adversarial games that have their equilibrium concentrating at the true joint data distributions p(x, z) and p(x, y), avoiding distributing the probability mass diffusely over data space that MLE-based methods may suffer. We assess SGAN by evaluating its trained networks, and its performance on downstream tasks. We show that SGAN delivers a highly controllable generator, and disentangled representations; it also establishes start-of-the-art results across multiple datasets when applied for semi-supervised image classification (1. 27%, 5. 73%, 17. 26% error rates on MNIST, SVHN and CIFAR-10 using 50, 1000 and 4000 labels, respectively). Benefiting from the separate modeling of y and z, SGAN can generate images with high visual quality and strictly following the designated semantic, and can be extended to a wide spectrum of applications, such as style transfer.

AAAI Conference 2016 Conference Paper

Drosophila Gene Expression Pattern Annotations via Multi-Instance Biological Relevance Learning

• Hua Wang
• Cheng Deng
• Hao Zhang
• Xinbo Gao
• Heng Huang

IJCAI Conference 2016 Conference Paper

Enforcing Template Representability and Temporal Consistency for Adaptive Sparse Tracking

• Xue Yang
• Fei Han
• Hua Wang
• Hao Zhang

Sparse representation has been widely studied in visual tracking, which has shown promising tracking performance. Despite a lot of progress, the visual tracking problem is still a challenging task due to appearance variations over time. In this paper, we propose a novel sparse tracking algorithm that well addresses temporal appearance changes, by enforcing template representability and temporal consistency (TRAC). By modeling temporal consistency, our algorithm addresses the issue of drifting away from a tracking target. By exploring the templates' long-term-short-term representability, the proposed method adaptively updates the dictionary using the most descriptive templates, which significantly improves the robustness to target appearance changes. We compare our TRAC algorithm against the state-of-the-art approaches on 12 challenging benchmark image sequences. Both qualitative and quantitative results demonstrate that our algorithm significantly outperforms previous state-of-the-art trackers.

AIIM Journal 2016 Journal Article

Web-video-mining-supported workflow modeling for laparoscopic surgeries

• Rui Liu
• Xiaoli Zhang
• Hao Zhang

Motivation As quality assurance is of strong concern in advanced surgeries, intelligent surgical systems are expected to have knowledge such as the knowledge of the surgical workflow model (SWM) to support their intuitive cooperation with surgeons. For generating a robust and reliable SWM, a large amount of training data is required. However, training data collected by physically recording surgery operations is often limited and data collection is time-consuming and labor-intensive, severely influencing knowledge scalability of the surgical systems. Objective The objective of this research is to solve the knowledge scalability problem in surgical workflow modeling with a low cost and labor efficient way. Methods A novel web-video-mining-supported surgical workflow modeling (webSWM) method is developed. A novel video quality analysis method based on topic analysis and sentiment analysis techniques is developed to select high-quality videos from abundant and noisy web videos. A statistical learning method is then used to build the workflow model based on the selected videos. To test the effectiveness of the webSWM method, 250 web videos were mined to generate a surgical workflow for the robotic cholecystectomy surgery. The generated workflow was evaluated by 4 web-retrieved videos and 4 operation-room-recorded videos, respectively. Results The evaluation results (video selection consistency n-index ≥0. 60; surgical workflow matching degree ≥0. 84) proved the effectiveness of the webSWM method in generating robust and reliable SWM knowledge by mining web videos. Conclusion With the webSWM method, abundant web videos were selected and a reliable SWM was modeled in a short time with low labor cost. Satisfied performances in mining web videos and learning surgery-related knowledge show that the webSWM method is promising in scaling knowledge for intelligent surgical systems.

YNIMG Journal 2010 Journal Article

Addiction related alteration in resting-state brain connectivity

• Ning Ma
• Ying Liu
• Nan Li
• Chang-Xin Wang
• Hao Zhang
• Xiao-Feng Jiang
• Hu-Sheng Xu
• Xian-Ming Fu

It is widely accepted that addictive drug use is related to abnormal functional organization in the user's brain. The present study aimed to identify this type of abnormality within the brain networks implicated in addiction by resting-state functional connectivity measured with functional magnetic resonance imaging (fMRI). With fMRI data acquired during resting state from 14 chronic heroin users (12 of whom were being treated with methadone) and 13 non-addicted controls, we investigated the addiction related alteration in functional connectivity between the regions in the circuits implicated in addiction with seed-based correlation analysis. Compared with controls, chronic heroin users showed increased functional connectivity between nucleus accumbens and ventral/rostral anterior cingulate cortex (ACC), between nucleus accumbens and orbital frontal cortex (OFC), and between amygdala and OFC and reduced functional connectivity between prefrontal cortex and OFC and between prefrontal cortex and ACC. These observations of altered resting-state functional connectivity suggested abnormal functional organization in the addicted brain and may provide additional evidence supporting the theory of addiction that emphasizes enhanced salience value of a drug and its related cues but weakened cognitive control in the addictive state.

YNIMG Journal 2005 Journal Article

An fMRI study of the role of suprapontine brain structures in the voluntary voiding control induced by pelvic floor contraction

• Hao Zhang
• Andre Reitz
• Spyros Kollias
• Paul Summers
• Armin Curt
• Brigitte Schurch

We have learned that micturition is comprised of two basic phases: storage and emptying; during bladder emptying, the pontine and periaqueductal gray (PAG) micturition center ensures coordinated inhibition of striated sphincter and pelvic floor muscles and relaxation of the internal urethral sphincter while the detrusor muscle contracts. Due to several disorders of the brain and spinal cord, the achieved voluntary control of bladder function can be impaired, and involuntary mechanisms of bladder activation again become evident. However, little has been discovered so far how higher brain centers strictly regulate the intricate process of micturition. The present functional magnetic resonance imaging (fMRI) study attempted to identify brain areas involved in such voluntary control of the micturition reflex by performing functional magnetic resonance imaging during a block design experiment in 12 healthy subjects. The protocol consisted of alternating periods of rest and pelvic muscle contraction during empty-bladder condition (EBC) and full-bladder condition (FBC). Repeated pelvic floor muscle contractions were performed during full bladder to induce a stronger contrast of bladder sensation, desire to void and inhibition of the micturition reflex triggering, since the subjects were asked not to urinate. Empty-bladder conditions were applied as control groups. Activation maps calculated by contrast of subtracting the two different conditions were purposed to disclose these brain areas that are involved during the inhibition of the micturition reflex, in which contrast, the SMA, bilateral putamen, right parietal cortex, right limbic system, and right cerebellum were found activated. The combined activation of basal ganglia, parietal cortex, limbic system, and cerebellum might support the assumption that a complex visceral sensory–motor program is involved during the inhibitory control of the micturition reflex.

ICRA Conference 1999 Conference Paper

Force-Responsive Robotic Assembly of Transmission Components

• Wyatt S. Newman
• Michael S. Branicky
• Andy Podgurski
• Siddharth R. Chhatpar
• Ling Huang
• Jayendran Swaminathan
• Hao Zhang

Assembly tasks involving large position uncertainties are unsuitable for use of position-controlled robots. To automate such tasks, the assembly system must be responsive to contact forces. Issues in addressing force-responsive automated assembly include contact stability, the degree of force responsiveness required for success, the speed of a successful implementation, and the means to program a force-responsive system to perform a given assembly task. We examine these issues for robotic assembly in the context of automotive transmission components. We report on an impedance-based low-level algorithm and its interface to higher-level strategies that exhibits gentle, fast and reliable assembly of our example components.