Arrow Research search

Author name cluster

Hao Zhu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

36 papers
2 author rows

Possible papers

36

EAAI Journal 2026 Journal Article

FEAN: A Fragments Embedding and Aligning Network for image-text matching

  • Xianlun Tang
  • Lin Jiang
  • Yu Xia
  • Wuquan Deng
  • Bo Tang
  • Tianyu Xiang
  • Hao Zhu

Most recent image-text matching methods use fragments alignment structures to achieve cross modal interaction. Although they compensate for the lack of cross modal interaction caused by overall embedding, the accuracy of image-text matching is compromised by the cross modal semantic gap and redundant alignment. To address these issues, we propose a Fragments Embedding and Aligning Network (FEAN) to achieve accurate image-text matching, which focuses on the representation of global and local image-text features and their similarity. Specifically, we embed local image and textual features into fragments to obtain global image and textual features representations and global similarity scores, and use cross modal weight calculation to obtain fragments aligned local image and textual features and local similarity matrices, in order to mitigate the cross modal semantic gap. To avoid redundant alignment in image and textual fragments, a Similarity Pooling (SP) strategy is proposed to aggregate global similarity scores and local similarity matrices. In addition, to compensate for the missing contextual semantic information on the image region features, we add the Position Weight Feature Reinforcement (PWFR) module before embedding alignment to achieve consistency with the text semantic information. Extensive experiments on two publicly available datasets for image-text matching, Flickr30K and MSCOCO, have shown that our fragments embedding and aligning network achieve the best R@1 and RSUM values in bidirectional matching with optimal accuracy.

Details DOI

AAAI Conference 2026 Conference Paper

Hierarchical Direction Perception via Atomic Dot-Product Operators for Rotation-Invariant Point Clouds Learning

  • Chenyu Hu
  • Xiaotong Li
  • Hao Zhu
  • Biao Hou

Point cloud processing has become a cornerstone technology in many 3D vision tasks. However, arbitrary rotations introduce variations in point cloud orientations, posing a long-standing challenge for effective representation learning. The core of this issue is the disruption of the point cloud's intrinsic directional characteristics caused by rotational perturbations. Recent methods attempt to implicitly model rotational equivariance and invariance, preserving directional information and propagating it into deep semantic spaces. Yet, they often fall short of fully exploiting the multiscale directional nature of point clouds to enhance feature representations. To address this, we propose the Direction-Perceptive Vector Network (DiPVNet). At its core is an atomic dot-product operator that simultaneously encodes directional selectivity and rotation invariance--endowing the network with both rotational symmetry modeling and adaptive directional perception. At the local level, we introduce a Learnable Local Dot-Product (L2DP) Operator, which enables interactions between a center point and its neighbors to adaptively capture the non-uniform local structures of point clouds. At the global level, we leverage generalized harmonic analysis to prove that the dot-product between point clouds and spherical sampling vectors is equivalent to a direction-aware spherical Fourier transform (DASFT). This leads to the construction of a global directional response spectrum for modeling holistic directional structures. We rigorously prove the rotation invariance of both operators. Extensive experiments on challenging scenarios involving noise and large-angle rotations demonstrate that DiPVNet achieves state-of-the-art performance on point cloud classification and segmentation tasks.

PDF Details DOI

YNIMG Journal 2026 Journal Article

Motor-based prediction during preparation of hand movement modulates auditory processing in two distinct directions

  • Xinjing Li
  • Xiaodan Zheng
  • Yuchunzi Wu
  • Hao Zhu
  • Yunying Shu
  • Ruiqi Tong
  • Xing Tian

Differentiating self-generated from externally induced sounds is crucial for survival. Predictions can be generated based on action-outcome contingency and suppress neural responses to sensory reafference for distinguishing the origin of stimuli. The action-outcome contingency can be flexible or relatively fixed (e.g., keypress could trigger various sounds vs. articulatory gestures generate corresponding speech sounds) and can be available during the entire course of action (including stages of intention, preparation and execution). Are motor-based predictions created equally based on different types of action-outcome contingency and during distinct stages of action? We conducted three EEG experiments to determine how motor preparation modulates auditory processing using a delayed keypress paradigm in which participants prepared to press a key to trigger a sound without knowing what key to press. In Experiment 1, keypress preparation showed overall enhanced N1 responses (∼100 ms), largest for syllables, but did not reveal any suppression effects. Experiment 2 replicated N1 enhancement and showed significant P2 suppression (∼200 ms) in response to auditory syllables, when participants were pianists who had extensive keypress-sound mapping experience. Experiment 3, when pianists were in their unfamiliar pairing of lab keys and familiar piano tones, again showed N1 enhancement, but the P2 suppression was absent. Together, these results suggest that preparatory motor prediction in an optional mapping can influence auditory processing in multiple directions and motivate a two-stage gain-to-attenuation hypothesis that may depend on the precision (reliability) of action-outcome associations.

Details DOI

AAAI Conference 2026 Conference Paper

RcAE: Recursive Reconstruction Framework for Unsupervised Industrial Anomaly Detection

  • Rongcheng Wu
  • Hao Zhu
  • Shiying Zhang
  • Mingzhe Wang
  • Zhidong Li
  • Hui Li
  • Jianlong Zhou
  • Jiangtao Cui

Unsupervised industrial anomaly detection requires accurately identifying defects without labeled data. Traditional autoencoder-based methods often struggle with incomplete anomaly suppression and loss of fine details, as their single-pass decoding fails to effectively handle anomalies with varying severity and scale. We propose a recursive architecture for autoencoder (RcAE), which performs reconstruction iteratively to progressively suppress anomalies while refining normal structures. Unlike traditional single-pass models, this recursive design naturally produces a sequence of reconstructions, progressively exposing suppressed abnormal patterns. To leverage this reconstruction dynamics, we introduce a Cross Recursion Detection (CRD) module that tracks inconsistencies across recursion steps, enhancing detection of both subtle and large-scale anomalies. Additionally, we incorporate a Detail Preservation Network (DPN) to recover high-frequency textures typically lost during reconstruction. Extensive experiments demonstrate that our method significantly outperforms existing non-diffusion methods, and achieves performance on par with recent diffusion models with only 10% of their parameters and offering substantially faster inference. These results highlight the practicality and efficiency of our approach for real-world applications.

PDF Details DOI

CLeaR Conference 2026 Conference Paper

Smoothing the Landscape: Causal Structure Learning via Diffusion Denoising Objectives

  • Hao Zhu
  • Di Zhou
  • Donna Slonim

Understanding causal dependencies in observational data is critical for informing decision-making. These relationships are often modeled as Bayesian Networks (BNs) and Directed Acyclic Graphs (DAGs). Existing methods, such as NOTEARS and DAG-GNN, often face issues with scalability and stability in high-dimensional data, especially when there is a feature-sample imbalance. Here, we show that the denoising score matching objective of diffusion models could smooth the gradients for faster, more stable convergence. We also propose an adaptive k-hop acyclicity constraint that improves runtime over existing solutions that require matrix inversion. We name this framework Denoising Diffusion Causal Discovery (DDCD). Unlike generative diffusion models, DDCD utilizes the reverse denoising process to infer a parameterized causal structure rather than to generate data. We demonstrate the competitive performance of DDCDs on synthetic benchmarking data. We also show that our methods are practically useful by conducting qualitative analyses on two real-world examples. Code is available at this url: https: //github. com/haozhu233/ddcd.

PDF Details

AAAI Conference 2026 Conference Paper

Split-Layer: Enhancing Implicit Neural Representation by Maximizing the Dimensionality of Feature Space

  • Zhicheng Cai
  • Hao Zhu
  • Linsen Chen
  • Qiu Shen
  • Xun Cao

Implicit neural representation (INR) models signals as continuous functions using neural networks, offering efficient and differentiable optimization for inverse problems across diverse disciplines. However, the representational capacity of INR—defined by the range of functions the neural network can characterize—is inherently limited by the low-dimensional feature space in conventional multilayer perceptron (MLP) architectures. While widening the MLP can linearly increase feature space dimensionality, it also leads to a quadratic growth in computational and memory costs. To address this limitation, we propose the split-layer, a novel reformulation of MLP construction. The split-layer divides each layer into multiple parallel branches and integrates their outputs via Hadamard product, effectively constructing a high-degree polynomial space. This approach significantly enhances INR’s representational capacity by expanding the feature space dimensionality without incurring prohibitive computational overhead. Extensive experiments demonstrate that the split-layer substantially improves INR performance, surpassing existing methods across multiple tasks, including 2D image fitting, 2D CT reconstruction, 3D shape representation, and 5D novel view synthesis.

PDF Details DOI

EAAI Journal 2025 Journal Article

Advances in physics-informed neural networks for solving complex partial differential equations and their engineering applications: A systematic review

  • Jiangtao Guo
  • Hao Zhu
  • Yujie Yang
  • Chenrui Guo

Most physical and engineering problems can be described by partial differential equations (PDEs), which are typically solved using numerical methods such as the finite difference method and the finite element method. However, conventional numerical discretization approaches face significant challenges in terms of computational efficiency and convergence speed when dealing with complex nonlinear PDEs, such as high-dimensional nonlinear PDEs, stiff PDEs, PDEs with complex boundary conditions or irregular geometries, and multi-scale PDEs. Recently, physics-informed neural networks (PINNs) have emerged as a transformative methodology for solving complex PDEs by integrating physical laws intrinsically into deep learning architectures. While PINNs effectively overcome mesh dependency and dimensionality constraints inherent in traditional numerical methods, they still encounter persistent challenges related to training convergence and generalization robustness. This paper aims to present a comprehensive review of the state-of-the-art developments in PINNs for solving complex PDE problems. The core ideas, network architectures, and generic implementation frameworks, along with associated open-source Python libraries, are first introduced in detail. Furthermore, a systematic taxonomy of optimization techniques is provided, covering hyperparameter selection, adaptive sampling strategies, physics-constrained loss formulations, hybrid differentiation approaches, and architectural innovations. Subsequently, various coping strategies and research advancements of PINNs in addressing complex nonlinear PDE problems are thoroughly discussed. Real-world engineering applications are then reviewed across multiple domains, including cosmology and quantum mechanics, materials science and manufacturing, fluid mechanics, energy systems, biological and environmental sciences, and power and information technologies. Finally, this paper discusses the current challenges and limitations of PINNs in solving complex PDEs and outlines potential directions for future research. By addressing the current limitations and pursuing targeted improvements in architectures, training, interpretability and generalization, PINNs can become a powerful tool in engineering and scientific applications.

Details DOI

ICLR Conference 2025 Conference Paper

Beyond Random Masking: When Dropout meets Graph Convolutional Networks

  • Yuankai Luo
  • Xiao-Ming Wu 0003
  • Hao Zhu

Graph Convolutional Networks (GCNs) have emerged as powerful tools for learning on graph-structured data, yet the behavior of dropout in these models remains poorly understood. This paper presents a comprehensive theoretical analysis of dropout in GCNs, revealing that its primary role differs fundamentally from standard neural networks - preventing oversmoothing rather than co-adaptation. We demonstrate that dropout in GCNs creates dimension-specific stochastic sub-graphs, leading to a form of structural regularization not present in standard neural networks. Our analysis shows that dropout effects are inherently degree-dependent, resulting in adaptive regularization that considers the topological importance of nodes. We provide new insights into dropout's role in mitigating oversmoothing and derive novel generalization bounds that account for graph-specific dropout effects. Furthermore, we analyze the synergistic interaction between dropout and batch normalization in GCNs, uncovering a mechanism that enhances overall regularization. Our theoretical findings are validated through extensive experiments on both node-level and graph-level tasks across 14 datasets. Notably, GCN with dropout and batch normalization outperforms state-of-the-art methods on several benchmarks, demonstrating the practical impact of our theoretical insights.

Details

ICML Conference 2025 Conference Paper

Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time

  • Mohamad Fares El Hajj Chehade
  • Soumya Suvra Ghosal
  • Souradip Chakraborty
  • Avinash Reddy
  • Dinesh Manocha
  • Hao Zhu
  • Amrit Singh Bedi

Aligning large language models with humans is challenging due to the inherently multifaceted nature of preference feedback. While existing approaches typically frame this as a multi-objective optimization problem, they often overlook how humans actually make decisions. Research on bounded rationality suggests that human decision making follows satisficing strategies- optimizing primary objectives while ensuring others meet acceptable thresholds. To bridge this gap and operationalize the notion of satisficing alignment, we propose SITAlign: an inference-time framework that addresses the multifaceted nature of alignment by maximizing a primary objective while satisfying threshold-based constraints on secondary criteria. We provide theoretical insights by deriving sub-optimality bounds of our satisficing-based inference alignment approach. We empirically validate SITAlign’s performance through extensive experimentation on multiple benchmarks. For instance, on the PKU-SafeRLHF dataset with the primary objective of maximizing helpfulness while ensuring a threshold on harmlessness, SITAlign outperforms the state-of-the-art multi-objective decoding strategy by a margin of 22. 3% in terms of GPT-4 win-tie rate for helpfulness reward while adhering to the threshold on harmlessness.

Details

TMLR Journal 2025 Journal Article

Constrained Reinforcement Learning with Smoothed Log Barrier Function

  • Baohe Zhang
  • Yuan Zhang
  • Hao Zhu
  • Shengchao Yan
  • Thomas Brox
  • Joschka Boedecker

Deploying reinforcement learning (RL) in real-world systems often requires satisfying strict safety constraints during both training and deployment, which simple reward shaping typically fails to enforce. Existing constrained RL algorithms frequently face several major challenges, including instabilities during training and overly conservative policies. To overcome these limitations, we propose CSAC-LB (Constrained Soft Actor-Critic with Log Barrier), a model-free, sample-efficient, off-policy algorithm that requires no pre-training. CSAC-LB integrates a linear smoothed log barrier function into the actor’s objective, providing a numerically stable, non-vanishing gradient that enables the agent to quickly recover from unsafe states while avoiding the instability of traditional interior-point methods. To further enhance safety and mitigate the underestimation of constraint violations, we employ a pessimistic double-critic architecture for the cost function, taking the maximum of two cost Q-networks to conservatively guide the policy. Through extensive experiments on challenging constrained control tasks, we demonstrate that CSAC-LB significantly outperforms baselines by consistently achieving high returns while strictly adhering to safety constraints. Our results establish CSAC-LB as a robust and stable solution for applying RL to safety-critical domains.

PDF Details

NeurIPS Conference 2025 Conference Paper

CrossSpectra: Exploiting Cross-Layer Smoothness for Parameter-Efficient Fine-Tuning

  • Yifei Zhang
  • Hao Zhu
  • Junhao Dong
  • Haoran Shi
  • Ziqiao Meng
  • Piotr Koniusz
  • Han Yu

Parameter-efficient fine-tuning (PEFT) is essential for adapting large foundation models without excessive storage cost. However, current approaches such as LoRA treat each layer’s adaptation independently, overlooking correlations across layers. This independence causes the number of trainable parameters to grow linearly with model depth. We provide theoretical and empirical evidence that skip connections in transformers create smooth gradient propagation across layers. This smoothness leads to weight adaptations that concentrate most of their energy in low-frequency spectral components, especially along the layer dimension. Empirical analysis confirms this effect, showing that most of adaptation energy lies in low frequencies. Building on this insight, we propose CrossSpectra, which parameterizes all attention-weight adaptations $(Q, K, V)$ across layers as a single 3D tensor and represents them with sparse spectral coefficients ($\kappa_1, \kappa_2$). Using $\kappa_{1}$ non-zero coefficients within each layer’s frequency space and truncating to $\kappa_{2}$ frequencies across layers, CrossSpectra requires $\mathcal{O}(\kappa_{1}\kappa_{2})$ parameters instead of LoRA’s $\mathcal{O}(Lrd)$, where $L$ is the number of layers and $r$ the rank. Across natural-language and vision benchmarks, \methodname{} matches or surpasses baseline performance while using fewer parameters than LoRA, achieving only $0. 36\%$ of LoRA’s parameter count when fine-tuning LLaMA-7B on instruction-following tasks. These results show that exploiting the \textbf{architectural smoothness of transformers} through spectral analysis yields major efficiency gains in PEFT.

PDF Details

AAAI Conference 2025 Conference Paper

From 2D CAD Drawings to 3D Parametric Models: A Vision-Language Approach

  • Xilin Wang
  • Jia Zheng
  • Yuanchao Hu
  • Hao Zhu
  • Qian Yu
  • Zihan Zhou

In this paper, we present CAD2Program, a new method for reconstructing 3D parametric models from 2D CAD drawings. Our proposed method is inspired by recent successes in vision-language models (VLMs), and departs from traditional methods which rely on task-specific data representations and/or algorithms. Specifically, on the input side, we simply treat the 2D CAD drawing as a raster image, regardless of its original format, and encode the image with a standard ViT model. We show that such an encoding scheme achieves competitive performance against existing methods that operate on vector-graphics inputs, while imposing substantially fewer restrictions on the 2D drawings. On the output side, our method auto-regressively predicts a general-purpose language describing 3D parametric models in text form. Compared to other sequence modeling methods for CAD which use domain-specific sequence representations with fixed-size slots, our text-based representation is more flexible, and can be easily extended to arbitrary geometric entities and semantic or functional properties. Experimental results on a large-scale dataset of cabinet models demonstrate the effectiveness of our method.

PDF Details DOI

ICML Conference 2025 Conference Paper

LEVIS: Large Exact Verifiable Input Spaces for Neural Networks

  • Mohamad Fares El Hajj Chehade
  • Wenting Li
  • Brian Wesley Bell
  • Russell Bent
  • Saif R. Kazi
  • Hao Zhu

The robustness of neural networks is crucial in safety-critical applications, where identifying a reliable input space is essential for effective model selection, robustness evaluation, and the development of reliable control strategies. Most existing robustness verification methods assess the worst-case output under the assumption that the input space is known. However, precisely identifying a verifiable input space $ \mathcal{C} $, where no adversarial examples exist, is challenging due to the possible high dimensionality, discontinuity, and non-convex nature of the input space. To address this challenge, we propose a novel framework, LEVIS, comprising LEVIS-$\alpha$ and LEVIS-$\beta$. LEVIS-$\alpha$ identifies a single, large verifiable ball that intersects at least two boundaries of a bounded region $ \mathcal{C} $, while LEVIS-$\beta$ systematically captures the entirety of the verifiable space by integrating multiple verifiable balls. Our contributions are fourfold: we introduce a verification framework, LEVIS, incorporating two optimization techniques for computing nearest and directional adversarial points based on mixed-integer programming (MIP); to enhance scalability, we integrate complementary constrained (CC) optimization with a reduced MIP formulation, achieving up to a 17-fold reduction in runtime by approximating the verifiable region in a principled way; we provide a theoretical analysis characterizing the properties of the verifiable balls obtained through LEVIS-$\alpha$; and we validate our approach across diverse applications, including electrical power flow regression and image classification, demonstrating performance improvements and visualizing the geometric properties of the verifiable region.

Details

NeurIPS Conference 2025 Conference Paper

Machine Unlearning via Task Simplex Arithmetic

  • Junhao Dong
  • Hao Zhu
  • Yifei Zhang
  • Xinghua Qu
  • Yew Soon Ong
  • Piotr Koniusz

As foundation Vision-Language Models (VLMs) unlock fine-tuning on smaller datasets while leveraging large-scale pre-training data, machine unlearning becomes critical in addressing privacy concerns and regulatory compliance. Task vector, representing the difference between parameters of models fine-tuned with and without specific data, is a popular retraining-free unlearning strategy. However, we observe that task vectors exhibit substantial sensitivity to various fine-tuning configurations, resulting in unstable unlearning effectiveness that correlates negatively with the prediction-level variance. While aggregating multiple functions (e. g. , VLM with classifier) whose parameters are represented by different task vectors reduces function variance and improves unlearning, the computational cost of obtaining numerous task vectors and aggregating functions is computationally high. Thus, in order to capture the space of task vectors induced by diverse fine-tuning strategies, we propose modeling it within the convex hull of $(Q-1)$-simplex whose vertices represent $Q$ task vectors. Although a function ensemble can be formed by sampling numerous task vectors from such a simplex, we derive a closed-form ensemble of an infinite number of functions whose parameters are uniformly sampled from the simplex, enabling efficient function-level task vector ensembling with enhanced unlearning performance. Extensive experiments and analyses across diverse datasets and scenarios demonstrate the efficacy of our method.

PDF Details

NeurIPS Conference 2025 Conference Paper

Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modeling

  • Yihong Dong
  • Ge Li
  • Xue Jiang
  • Yongding Tao
  • Kechi Zhang
  • Lecheng Wang
  • Hao Zhu
  • Huanyu Liu

Periodicity, as one of the most important basic characteristics, lays the foundation for facilitating structured knowledge acquisition and systematic cognitive processes within human learning paradigms. However, the potential flaws of periodicity modeling in Transformer affect the learning efficiency and establishment of underlying principles from data for large language models (LLMs) built upon it. In this paper, we demonstrate that integrating effective periodicity modeling can improve the learning efficiency and performance of LLMs. We introduce FANformer, which adapts Fourier Analysis Network (FAN) into attention mechanism to achieve efficient periodicity modeling, by modifying the feature projection process of attention mechanism. Extensive experimental results on language modeling show that FANformer consistently outperforms Transformer when scaling up model size and training tokens, underscoring its superior learning efficiency. Our pretrained FANformer-1B exhibits marked improvements on downstream tasks compared to open-source LLMs with similar model parameters or training tokens. Moreover, we reveal that FANformer exhibits superior ability to learn and apply rules for reasoning compared to Transformer. The results position FANformer as an effective and promising architecture for advancing LLMs.

PDF Details

NeurIPS Conference 2025 Conference Paper

Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models

  • Wen-Tse Chen
  • Jiayu Chen
  • Fahim Tajwar
  • Hao Zhu
  • Xintong Duan
  • Ruslan Salakhutdinov
  • Jeff Schneider

Learning from self-sampled data and sparse environmental feedback remains a fundamental challenge in training self-evolving agents. Temporal credit assignment mitigates this issue by transforming sparse feedback into dense supervision signals. However, previous approaches typically depend on domain-specific value functions for credit assignment, which suffer from poor sample efficiency and limited generalization. In this work, we propose to leverage pre-trained knowledge from large language models (LLMs) to transform sparse rewards into dense training signals (i. e. , the advantage function) through retrospective in-context learning (RICL). We further propose an online learning framework, RICOL, which iteratively refines the policy based on the credit assignment results from RICL. We empirically demonstrate that RICL can accurately estimate the advantage function with limited samples and effectively identify critical states for temporal credit assignment. Extended evaluation on the BabyAI benchmark shows that RICOL significantly improves sample efficiency compared to traditional online RL algorithms while achieving performance comparable to imitation learning from expert demonstartions. Our findings highlight the potential of leveraging LLMs for temporal credit assignment, paving the way for more sample-efficient and generalizable RL paradigms.

PDF Details

NeurIPS Conference 2025 Conference Paper

SATURN: SAT-based Reinforcement Learning to Unleash LLMs Reasoning

  • Huanyu Liu
  • Ge Li
  • Jia Li
  • Hao Zhu
  • Kechi Zhang
  • Yihong Dong

How to design reinforcement learning (RL) tasks that effectively unleash the reasoning capability of large language models (LLMs) remains an open question. Existing RL tasks (e. g. , math, programming, and constructing reasoning tasks) suffer from three key limitations: (1) Scalability. They rely heavily on human annotation or expensive LLM synthesis to generate sufficient training data. (2) Verifiability. LLMs' outputs are hard to verify automatically and reliably. (3) Controllable Difficulty. Most tasks lack fine-grained difficulty control, making it hard to train LLMs to develop reasoning ability from easy to hard. To address these limitations, we propose Saturn, a SAT-based RL framework that uses Boolean Satisfiability (SAT) problems to train and evaluate LLMs reasoning. Saturn enables scalable task construction, rule-based verification, and precise difficulty control. Saturn designs a curriculum learning pipeline that continuously improves LLMs' reasoning capability by constructing SAT tasks of increasing difficulty and training LLMs from easy to hard. To ensure stable training, we design a principled mechanism to control difficulty transitions. We introduce Saturn-2. 6k, a dataset of 2, 660 SAT problems with varying difficulty. It supports the evaluation of how LLM reasoning changes with problem difficulty. We apply Saturn to DeepSeek-R1-Distill-Qwen and obtain Saturn-1. 5B and Saturn-7B. We achieve several notable results: (1) On SAT problems, Saturn-1. 5B and Saturn-7B achieve average pass@3 improvements of +14. 0 and +28. 1, respectively. (2) On math and programming tasks, Saturn-1. 5B and Saturn-7B improve average scores by +4. 9 and +1. 8 on benchmarks (e. g. , AIME, LiveCodeBench). (3) Compared to the state-of-the-art (SOTA) approach in constructing RL tasks, Saturn achieves further improvements of +8. 8\%. We release the source code, data, and models to support future research.

PDF Details

NeurIPS Conference 2025 Conference Paper

SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions

  • Xianzhe Fan
  • Xuhui Zhou
  • Chuanyang Jin
  • Kolby Nottingham
  • Hao Zhu
  • Maarten Sap

Humans continuously infer the states, goals, and behaviors of others by perceiving their surroundings in dynamic, real-world social interactions. However, most Theory of Mind (ToM) benchmarks only evaluate static, text-based scenarios, which have a significant gap compared to real interactions. We propose the SoMi-ToM benchmark, designed to evaluate multi-perspective ToM in embodied multi-agent complex social interactions. This benchmark is based on rich multimodal interaction data generated by the interaction environment SoMi, covering diverse crafting goals and social relationships. Our framework supports multi-level evaluation: (1) first-person evaluation provides multimodal (visual, dialogue, action, etc. ) input from a first-person perspective during a task for real-time state inference, (2) third-person evaluation provides complete third-person perspective video and text records after a task for goal and behavior inference. This evaluation method allows for a more comprehensive examination of a model's ToM capabilities from both the subjective immediate experience and the objective global observation. We constructed a challenging dataset containing 35 third-person perspective videos, 363 first-person perspective images, and 1225 expert-annotated multiple-choice questions (three options). On this dataset, we systematically evaluated the performance of human subjects and several state-of-the-art large vision-language models (LVLMs). The results show that LVLMs perform significantly worse than humans on SoMi-ToM: the average accuracy gap between humans and models is 40. 1% in first-person evaluation and 26. 4% in third-person evaluation. This indicates that future LVLMs need to further improve their ToM capabilities in embodied, complex social interactions.

PDF Details

NeurIPS Conference 2025 Conference Paper

SpatialLM: Training Large Language Models for Structured Indoor Modeling

  • Yongsen Mao
  • Junhao Zhong
  • Chuan Fang
  • Jia Zheng
  • Rui Tang
  • Hao Zhu
  • Ping Tan
  • Zihan Zhou

SpatialLM is a large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. These outputs include architectural elements like walls, doors, windows, and oriented object boxes with their semantic categories. Unlike previous methods which exploit task-specific network designs, our model adheres to the standard multimodal LLM architecture and is fine-tuned directly from open-source LLMs. To train SpatialLM, we collect a large-scale, high-quality synthetic dataset consisting of the point clouds of 12, 328 indoor scenes (54, 778 rooms) with ground-truth 3D annotations, and conduct a careful study on various modeling and training decisions. On public benchmarks, our model gives state-of-the-art performance in layout estimation and competitive results in 3D object detection. With that, we show a feasible path for enhancing the spatial understanding capabilities of modern LLMs for applications in augmented reality, embodied robotics, and more.

PDF Details

AAAI Conference 2024 Conference Paper

A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields

  • Yiyu Zhuang
  • Qi Zhang
  • Xuan Wang
  • Hao Zhu
  • Ying Feng
  • Xiaoyu Li
  • Ying Shan
  • Xun Cao

Recent advances in implicit neural representation have demonstrated the ability to recover detailed geometry and material from multi-view images. However, the use of simplified lighting models such as environment maps to represent non-distant illumination, or using a network to fit indirect light modeling without a solid basis, can lead to an undesirable decomposition between lighting and material. To address this, we propose a fully differentiable framework named Neural Illumination Fields (NeIF) that uses radiance fields as a lighting model to handle complex lighting in a physically based way. Together with integral lobe encoding for roughness-adaptive specular lobe and leveraging the pre-convolved background for accurate decomposition, the proposed method represents a significant step towards integrating physically based rendering into the NeRF representation. The experiments demonstrate the superior performance of novel-view rendering compared to previous works, and the capability to re-render objects under arbitrary NeRF-style environments opens up exciting possibilities for bridging the gap between virtual and real-world scenes.

PDF Details DOI

IJCAI Conference 2024 Conference Paper

MISA: MIning Saliency-Aware Semantic Prior for Box Supervised Instance Segmentation

  • Hao Zhu
  • Yan Zhu
  • Jiayu Xiao
  • Yike Ma
  • Yucheng Zhang
  • Jintao Li
  • Feng Dai

Box supervised instance segmentation (BSIS) aims to achieve an effective trade-off between annotation costs and model performance by solely relying on bounding box annotations during training process. However, we observe that BSIS model is bottlenecked by the intricate objective under limited guidance, and tends to sacrifice segmentation capability in order to effectively recognize multiple instances. To boost the BSIS model's perceptual ability for object shape and contour, we introduce MISA, that is, MIning Saliency-Aware semantic prior from a well-optimized box supervised semantic segmentation (BSSS) network, and incorporating cross-model guidance into the learning process of BSIS. Specifically, we first design a Frequency-Space Distillation (FSD) module to extract assorted salient prior knowledge from BSSS model, and perform cross-model alignment for transfering the prior to BSIS model. Furthermore, we introduce Semantic-Enhanced Pairwise Affinity (SEPA), which borrows the object perceptual ability of BSSS model to emphasize the contribution of salient objects for pairwise affinity, providing more accurate guidance for the BSIS network. Extensive experiments show that our proposed MISA consistently surpasses the existing state-of-the-art methods by a large margin in the BSIS scenario.

PDF Details DOI

TMLR Journal 2024 Journal Article

Multi-intention Inverse Q-learning for Interpretable Behavior Representation

  • Hao Zhu
  • Brice De La Crompe
  • Gabriel Kalweit
  • Artur Schneider
  • Maria Kalweit
  • Ilka Diester
  • Joschka Boedecker

In advancing the understanding of natural decision-making processes, inverse reinforcement learning (IRL) methods have proven instrumental in reconstructing animal's intentions underlying complex behaviors. Given the recent development of a continuous-time multi-intention IRL framework, there has been persistent inquiry into inferring discrete time-varying rewards with IRL. To address this challenge, we introduce the class of hierarchical inverse Q-learning (HIQL) algorithms. Through an unsupervised learning process, HIQL divides expert trajectories into multiple intention segments, and solves the IRL problem independently for each. Applying HIQL to simulated experiments and several real animal behavior datasets, our approach outperforms current benchmarks in behavior prediction and produces interpretable reward functions. Our results suggest that the intention transition dynamics underlying complex decision-making behavior is better modeled by a step function instead of a smoothly varying function. This advancement holds promise for neuroscience and cognitive science, contributing to a deeper understanding of decision-making and uncovering underlying brain mechanisms.

PDF Details

AAAI Conference 2024 Conference Paper

PaintHuman: Towards High-Fidelity Text-to-3D Human Texturing via Denoised Score Distillation

  • Jianhui Yu
  • Hao Zhu
  • Liming Jiang
  • Chen Change Loy
  • Weidong Cai
  • Wayne Wu

Recent advances in zero-shot text-to-3D human generation, which employ the human model prior (e.g., SMPL) or Score Distillation Sampling (SDS) with pre-trained text-to-image diffusion models, have been groundbreaking. However, SDS may provide inaccurate gradient directions under the weak diffusion guidance, as it tends to produce over-smoothed results and generate body textures that are inconsistent with the detailed mesh geometry. Therefore, directly leveraging existing strategies for high-fidelity text-to-3D human texturing is challenging. In this work, we propose a model called PaintHuman to addresses the challenges from two perspectives. We first propose a novel score function, Denoised Score Distillation (DSD), which directly modifies the SDS by introducing negative gradient components to iteratively correct the gradient direction and generate high-quality textures. In addition, we use the depth map as a geometric guide to ensure that the texture is semantically aligned to human mesh surfaces. To guarantee the quality of rendered results, we employ geometry-aware networks to predict surface materials and render realistic human textures. Extensive experiments, benchmarked against state-of-the-art (SoTA) methods, validate the efficacy of our approach.Project page: https://painthuman.github.io/.

PDF Details DOI

EAAI Journal 2023 Journal Article

Advances in dynamic load identification based on data-driven techniques

  • Daixin Fu
  • Lingyi Wang
  • Guanlin Lv
  • Zhengyu Shen
  • Hao Zhu
  • W.D. Zhu

Dynamic loads on engineering structures are often difficult to measure directly. Therefore, indirect identification methods based on dynamic responses are commonly used. However, this approach is often influenced by ill-conditioned matrices, noise interference, unknown structural and/or material parameters, and difficulty of constructing transfer functions when the traditional physics-based model is used. To address these issues, significant strides have been made in data-driven identification of dynamic loads, which are model-free and independent of structural characteristics. This paper tries to present a comprehensive review of dynamic load identification methods based on data-driven techniques, covering two aspects: load localization and load reconstruction. Features of the widely used data-driven techniques such as the geometric method, reference database method, machine learning methods including SVM-based methods and ANN-based methods and deep learning methods are discussed in detail. Additionally, this paper offers insight into the challenges and prospects of the data-driven techniques for dynamic load identification. The review aims to provide valuable insights for identifying dynamic loads in complex structures based on data-driven techniques and suggests future research directions.

Details DOI

NeurIPS Conference 2023 Conference Paper

Mitigating the Popularity Bias of Graph Collaborative Filtering: A Dimensional Collapse Perspective

  • Yifei Zhang
  • Hao Zhu
  • Yankai Chen
  • Zixing Song
  • Piotr Koniusz
  • Irwin King

Graph-based Collaborative Filtering (GCF) is widely used in personalized recommendation systems. However, GCF suffers from a fundamental problem where features tend to occupy the embedding space inefficiently (by spanning only a low-dimensional subspace). Such an effect is characterized in GCF by the embedding space being dominated by a few of popular items with the user embeddings highly concentrated around them. This enhances the so-called Matthew effect of the popularity bias where popular items are highly recommend whereas remaining items are ignored. In this paper, we analyze the above effect in GCF and reveal that the simplified graph convolution operation (typically used in GCF) shrinks the singular space of the feature matrix. As typical approaches (i. e. , optimizing the uniformity term) fail to prevent the embedding space degradation, we propose a decorrelation-enhanced GCF objective that promotes feature diversity by leveraging the so-called principle of redundancy reduction in embeddings. However, unlike conventional methods that use the Euclidean geometry to relax hard constraints for decorrelation, we exploit non-Euclidean geometry. Such a choice helps maintain the range space of the matrix and obtain small condition number, which prevents the embedding space degradation. Our method outperforms contrastive-based GCF models on several benchmark datasets and improves the performance for unpopular items.

PDF Details

AAAI Conference 2023 Conference Paper

RAFaRe: Learning Robust and Accurate Non-parametric 3D Face Reconstruction from Pseudo 2D&3D Pairs

  • Longwei Guo
  • Hao Zhu
  • Yuanxun Lu
  • Menghua Wu
  • Xun Cao

We propose a robust and accurate non-parametric method for single-view 3D face reconstruction (SVFR). While tremendous efforts have been devoted to parametric SVFR, a visible gap still lies between the result 3D shape and the ground truth. We believe there are two major obstacles: 1) the representation of the parametric model is limited to a certain face database; 2) 2D images and 3D shapes in the fitted datasets are distinctly misaligned. To resolve these issues, a large-scale pseudo 2D&3D dataset is created by first rendering the detailed 3D faces, then swapping the face in the wild images with the rendered face. These pseudo 2D&3D pairs are created from publicly available datasets which eliminate the gaps between 2D and 3D data while covering diverse appearances, poses, scenes, and illumination. We further propose a non-parametric scheme to learn a well-generalized SVFR model from the created dataset, and the proposed hierarchical signed distance function turns out to be effective in predicting middle-scale and small-scale 3D facial geometry. Our model outperforms previous methods on FaceScape-wild/lab and MICC benchmarks and is well generalized to various appearances, poses, expressions, and in-the-wild environments. The code is released at https://github.com/zhuhao-nju/rafare.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Spectral Feature Augmentation for Graph Contrastive Learning and Beyond

  • Yifei Zhang
  • Hao Zhu
  • Zixing Song
  • Piotr Koniusz
  • Irwin King

Although augmentations (e.g., perturbation of graph edges, image crops) boost the efficiency of Contrastive Learning (CL), feature level augmentation is another plausible, complementary yet not well researched strategy. Thus, we present a novel spectral feature argumentation for contrastive learning on graphs (and images). To this end, for each data view, we estimate a low-rank approximation per feature map and subtract that approximation from the map to obtain its complement. This is achieved by the proposed herein incomplete power iteration, a non-standard power iteration regime which enjoys two valuable byproducts (under mere one or two iterations): (i) it partially balances spectrum of the feature map, and (ii) it injects the noise into rebalanced singular values of the feature map (spectral augmentation). For two views, we align these rebalanced feature maps as such an improved alignment step can focus more on less dominant singular values of matrices of both views, whereas the spectral augmentation does not affect the spectral angle alignment (singular vectors are not perturbed). We derive the analytical form for: (i) the incomplete power iteration to capture its spectrum-balancing effect, and (ii) the variance of singular values augmented implicitly by the noise. We also show that the spectral augmentation improves the generalization bound. Experiments on graph/image datasets show that our spectral feature augmentation outperforms baselines, and is complementary with other augmentation strategies and compatible with various contrastive losses.

PDF Details DOI

AAAI Conference 2022 Conference Paper

Detailed Facial Geometry Recovery from Multi-View Images by Learning an Implicit Function

  • Yunze Xiao
  • Hao Zhu
  • Haotian Yang
  • Zhengyu Diao
  • Xiangju Lu
  • Xun Cao

Recovering detailed facial geometry from a set of calibrated multi-view images is valuable for its wide range of applications. Traditional multi-view stereo (MVS) methods adopt an optimization-based scheme to regularize the matching cost. Recently, learning-based methods integrate all these into an end-to-end neural network and show superiority of efficiency. In this paper, we propose a novel architecture to recover extremely detailed 3D faces within dozens of seconds. Unlike previous learning-based methods that regularize the cost volume via 3D CNN, we propose to learn an implicit function for regressing the matching cost. By fitting a 3D morphable model from multi-view images, the features of multiple images are extracted and aggregated in the mesh-attached UV space, which makes the implicit function more effective in recovering detailed facial shape. Our method outperforms SOTA learning-based MVS in accuracy by a large margin on the FaceScape dataset. The code and data are released in https: //github. com/zhuhao-nju/mvfr.

PDF Details

NeurIPS Conference 2022 Conference Paper

Generalized Laplacian Eigenmaps

  • Hao Zhu
  • Piotr Koniusz

Graph contrastive learning attracts/disperses node representations for similar/dissimilar node pairs under some notion of similarity. It may be combined with a low-dimensional embedding of nodes to preserve intrinsic and structural properties of a graph. COLES, a recent graph contrastive method combines traditional graph embedding and negative sampling into one framework. COLES in fact minimizes the trace difference between the within-class scatter matrix encapsulating the graph connectivity and the total scatter matrix encapsulating negative sampling. In this paper, we propose a more essential framework for graph embedding, called Generalized Laplacian EigeNmaps (GLEN), which learns a graph representation by maximizing the rank difference between the total scatter matrix and the within-class scatter matrix, resulting in the minimum class separation guarantee. However, the rank difference minimization is an NP-hard problem. Thus, we replace the trace difference that corresponds to the difference of nuclear norms by the difference of LogDet expressions, which we argue is a more accurate surrogate for the NP-hard rank difference than the trace difference. While enjoying a lesser computational cost, the difference of LogDet terms is lower-bounded by the Affine-invariant Riemannian metric (AIRM) and Jesen-Bregman the LogDet Divergence (JBLD), and upper-bounded by AIRM scaled by the factor of $\sqrt{m}$. We show that GLEN offers favourable accuracy/scalability compared to state-of-the-art baselines.

PDF Details

NeurIPS Conference 2021 Conference Paper

Contrastive Laplacian Eigenmaps

  • Hao Zhu
  • Ke Sun
  • Peter Koniusz

Graph contrastive learning attracts/disperses node representations for similar/dissimilar node pairs under some notion of similarity. It may be combined with a low-dimensional embedding of nodes to preserve intrinsic and structural properties of a graph. In this paper, we extend the celebrated Laplacian Eigenmaps with contrastive learning, and call them COntrastive Laplacian EigenmapS (COLES). Starting from a GAN-inspired contrastive formulation, we show that the Jensen-Shannon divergence underlying many contrastive graph embedding models fails under disjoint positive and negative distributions, which may naturally emerge during sampling in the contrastive setting. In contrast, we demonstrate analytically that COLES essentially minimizes a surrogate of Wasserstein distance, which is known to cope well under disjoint distributions. Moreover, we show that the loss of COLES belongs to the family of so-called block-contrastive losses, previously shown to be superior compared to pair-wise losses typically used by contrastive methods. We show on popular benchmarks/backbones that COLES offers favourable accuracy/scalability compared to DeepWalk, GCN, Graph2Gauss, DGI and GRACE baselines.

PDF Details

NeurIPS Conference 2020 Conference Paper

AOT: Appearance Optimal Transport Based Identity Swapping for Forgery Detection

  • Hao Zhu
  • Chaoyou Fu
  • Qianyi Wu
  • Wayne Wu
  • Chen Qian
  • Ran He

Recent studies have shown that the performance of forgery detection can be improved with diverse and challenging Deepfakes datasets. However, due to the lack of Deepfakes datasets with large variance in appearance, which can be hardly produced by recent identity swapping methods, the detection algorithm may fail in this situation. In this work, we provide a new identity swapping algorithm with large differences in appearance for face forgery detection. The appearance gaps mainly arise from the large discrepancies in illuminations and skin colors that widely exist in real-world scenarios. However, due to the difficulties of modeling the complex appearance mapping, it is challenging to transfer fine-grained appearances adaptively while preserving identity traits. This paper formulates appearance mapping as an optimal transport problem and proposes an Appearance Optimal Transport model (AOT) to formulate it in both latent and pixel space. Specifically, a relighting generator is designed to simulate the optimal transport plan. It is solved via minimizing the Wasserstein distance of the learned features in the latent space, enabling better performance and less computation than conventional optimization. To further refine the solution of the optimal transport plan, we develop a segmentation game to minimize the Wasserstein distance in the pixel space. A discriminator is introduced to distinguish the fake parts from a mix of real and fake image patches. Extensive experiments reveal that the superiority of our method when compared with state-of-the-art methods and the ability of our generated data to improve the performance of face forgery detection.

PDF Details

IJCAI Conference 2020 Conference Paper

Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning

  • Hao Zhu
  • Huaibo Huang
  • Yi Li
  • Aihua Zheng
  • Ran He

Talking face generation aims to synthesize a face video with precise lip synchronization as well as a smooth transition of facial motion over the entire video via the given speech clip and facial image. Most existing methods mainly focus on either disentangling the information in a single image or learning temporal information between frames. However, cross-modality coherence between audio and video information has not been well addressed during synthesis. In this paper, we propose a novel arbitrary talking face generation framework by discovering the audio-visual coherence via the proposed Asymmetric Mutual Information Estimator (AMIE). In addition, we propose a Dynamic Attention (DA) block by selectively focusing the lip area of the input image during the training stage, to further enhance lip synchronization. Experimental results on benchmark LRW dataset and GRID dataset transcend the state-of-the-art methods on prevalent metrics with robust high-resolution synthesizing on gender and pose variations.

PDF Details DOI

AAAI Conference 2020 Conference Paper

Generative Attention Networks for Multi-Agent Behavioral Modeling

  • Guangyu Li
  • Bo Jiang
  • Hao Zhu
  • Zhengping Che
  • Yan Liu

Understanding and modeling behavior of multi-agent systems is a central step for artificial intelligence. Here we present a deep generative model which captures behavior generating process of multi-agent systems, supports accurate predictions and inference, infers how agents interact in a complex system, as well as identifies agent groups and interaction types. Built upon advances in deep generative models and a novel attention mechanism, our model can learn interactions in highly heterogeneous systems with linear complexity in the number of agents. We apply this model to three multi-agent systems in different domains and evaluate performance on a diverse set of tasks including behavior prediction, interaction analysis and system identification. Experimental results demonstrate its ability to model multi-agent systems, yielding improved performance over competitive baselines. We also show the model can successfully identify agent groups and interaction types in these systems. Our model offers new opportunities to predict complex multi-agent behaviors and takes a step forward in understanding interactions in multi-agent systems.

PDF Details

NeurIPS Conference 2020 Conference Paper

Multi-agent Trajectory Prediction with Fuzzy Query Attention

  • Nitin Kamra
  • Hao Zhu
  • Dweep Kumarbhai Trivedi
  • Ming Zhang
  • Yan Liu

Trajectory prediction for scenes with multiple agents and entities is a challenging problem in numerous domains such as traffic prediction, pedestrian tracking and path planning. We present a general architecture to address this challenge which models the crucial inductive biases of motion, namely, inertia, relative motion, intents and interactions. Specifically, we propose a relational model to flexibly model interactions between agents in diverse environments. Since it is well-known that human decision making is fuzzy by nature, at the core of our model lies a novel attention mechanism which models interactions by making continuous-valued (fuzzy) decisions and learning the corresponding responses. Our architecture demonstrates significant performance gains over existing state-of-the-art predictive models in diverse domains such as human crowd trajectories, US freeway traffic, NBA sports data and physics datasets. We also present ablations and augmentations to understand the decision-making process and the source of gains in our model.

PDF Details

IJCAI Conference 2017 Conference Paper

Iterative Entity Alignment via Joint Knowledge Embeddings

  • Hao Zhu
  • Ruobing Xie
  • Zhiyuan Liu
  • Maosong Sun

Entity alignment aims to link entities and their counterparts among multiple knowledge graphs (KGs). Most existing methods typically rely on external information of entities such as Wikipedia links and require costly manual feature construction to complete alignment. In this paper, we present a novel approach for entity alignment via joint knowledge embeddings. Our method jointly encodes both entities and relations of various KGs into a unified low-dimensional semantic space according to a small seed set of aligned entities. During this process, we can align entities according to their semantic distance in this joint semantic space. More specifically, we present an iterative and parameter sharing method to improve alignment performance. Experiment results on real-world datasets show that, as compared to baselines, our method achieves significant improvements on entity alignment, and can further improve knowledge graph completion performance on various KGs with the favor of joint knowledge embeddings.

PDF Details

IJCAI Conference 2017 Conference Paper

Locality Constrained Deep Supervised Hashing for Image Retrieval

  • Hao Zhu
  • Shenghua Gao

Deep Convolutional Neural Network (DCNN) based deep hashing has shown its success for fast and accurate image retrieval, however directly minimizing the quantization error in deep hashing will change the distribution of DCNN features, and consequently change the similarity between the query and the retrieved images in hashing. In this paper, we propose a novel Locality-Constrained Deep Supervised Hashing. By simultaneously learning discriminative DCNN features and preserving the similarity between image pairs, the hash codes of our scheme preserves the distribution of DCNN features thus favors the accurate image retrieval. The contributions of this paper are two-fold: i) Our analysis shows that minimizing quantization error in deep hashing makes the features less discriminative which is not desirable for image retrieval; ii) We propose a Locality-Constrained Deep Supervised Hashing which preserves the similarity between image pairs in hashing. Extensive experiments on the CIFARA-10 and NUS-WIDE datasets show that our method significantly boosts the accuracy of image retrieval, especially on the CIFAR-10 dataset, the improvement is usually more than 6% in terms of the MAP measurement. Further, our method demonstrates 10 times faster than state-of-the-art methods in the training phase.

PDF Details
v2026.09.13