Author name cluster

Jiayu Wang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

14 papers

1 author row

AAAI Conference 2026 Conference Paper

Breaking Task Boundaries: A Unified Model for 3D Medical Image Fusion and Segmentation Guided by Manifold Perspective

Zeyu Wang
Jiayu Wang
Haiyu Song

3D medical image fusion (MIF) and segmentation (MIS) are critical and inherently synergistic tasks in medical image analysis. However, fundamentally integrating them remains highly challenging, since effective collaborative paradigms are still scarce and their optimization objectives fundamentally diverge. Moreover, existing continual learning techniques are unable to achieve truly advanced performance for both tasks using a shared weight. To address these challenges, we propose M²-CoFS, a unified model capable of jointly handling both tasks. Our core contribution is a “network-guided network learning” paradigm designed to break the task boundaries. We model the weight spaces of MIF and MIS as high-dimensional manifolds and innovatively use a lightweight neural network to implicitly construct a shared manifold. Interestingly, this network yields a unified weight for both tasks. To ensure the shared manifold retains the intrinsic geometry of both original manifolds, we embed manifold distances into the loss function of this network as a constraint. Additionally, we design a tailored three-stage training paradigm for our core contribution mentioned above. Stage I focuses on independent task optimization for high-quality weights; Stage II aims to reduce parameter-space distance between tasks via our cross-task weight adaptation strategy; Our core innovation serves as stage III. Experimental results show that M²-CoFS consistently outperforms state-of-the-art comparison models on both MlF and MIS.

PDF Details DOI

AAAI Conference 2026 Conference Paper

D3-RSMDE: 40× Faster and High-Fidelity Remote Sensing Monocular Depth Estimation

Ruizhi Wang
Weihan Li
Zunlei Feng
Haofei Zhang
Mingli Song
Jiayu Wang
Jie Song
Li Sun

Real-time, high-fidelity monocular depth estimation from remote sensing imagery is crucial for numerous applications, yet existing methods face a stark trade-off between accuracy and efficiency. Although using Vision Transformer (ViT) backbones for dense prediction is fast, they often exhibit poor perceptual quality. Conversely, diffusion models offer high fidelity but at a prohibitive computational cost. To overcome these limitations, we propose Depth Detail Diffusion for Remote Sensing Monocular Depth Estimation (D³-RSMDE), an efficient framework designed to achieve an optimal balance between speed and quality. Our framework first leverages a ViT-based module to rapidly generate a high-quality preliminary depth map construction, which serves as a structural prior, effectively replacing the time-consuming initial structure generation stage of diffusion models. Based on this prior, we propose a Progressive Linear Blending Refinement (PLBR) strategy, which uses a lightweight U-Net to refine the details in only a few iterations. The entire refinement step operates efficiently in a compact latent space supported by a Variational Autoencoder (VAE). Extensive experiments demonstrate that D³-RSMDE achieves a notable 11.85% reduction in the Learned Perceptual Image Patch Similarity (LPIPS) perceptual metric over leading models like Marigold, while also achieving over a 40× speedup in inference and maintaining VRAM usage comparable to lightweight ViT models.

PDF Details DOI

AAAI Conference 2026 Conference Paper

SD-PSFNet: Sequential and Dynamic Point Spread Function Network for Image Deraining

Jiayu Wang
Haoyu Bian
Haoran Sun
Shaoning Zeng

Image deraining is crucial for vision applications but is challenged by the complex multi-scale physics of rain and its coupling with scenes. To address this challenge, a novel approach inspired by multi-stage image restoration is proposed, incorporating Point Spread Function (PSF) mechanisms to reveal the image degradation process while combining dynamic physical modeling with sequential feature fusion transfer, named SD-PSFNet. Specifically, SD-PSFNet employs a sequential restoration architecture with three cascaded stages, allowing multiple dynamic evaluations and refinements of the degradation process estimation. The network utilizes components with learned PSF mechanisms to dynamically simulate rain streak optics, enabling effective rain-background separation while progressively enhancing outputs through novel PSF components at each stage. Additionally, SD-PSFNet incorporates adaptive gated fusion for optimal cross-stage feature integration, enabling sequential refinement from coarse rain removal to fine detail restoration. Our model achieves state-of-the-art PSNR/SSIM metrics on Rain100H (33.12dB/0.9371), RealRain-1k-L (42.28dB/0.9872), and RealRain-1k-H (41.08dB/0.9838). In summary, SD-PSFNet demonstrates excellent capability in complex scenes and dense rainfall conditions, providing a new physics-aware approach to image deraining.

PDF Details DOI

AAAI Conference 2026 Conference Paper

SigFusion: Unified Signal-Level Self-Supervised Learning Paradigm for Image Fusion

Zeyu Wang
Jiawei Feng
Jiayu Wang
Pengjie Wang
Haiyu Song

Image Fusion (IF) aims to integrate complementary features from multiple source images into a single image. However, a key challenge in this field is the lack of large-scale real-world training datasets. Existing models typically rely on either small datasets or synthetic, less realistic datasets. To address this, we propose SigFusion, a unified signal-level self-supervised learning paradigm for various IF tasks.The core idea is to use signal-level Pseudo-Label Generation Networks (PLGN) to automatically synthesize training sets and pseudo labels with real multi-source signal characteristics from vast unlabeled natural images.PLGN includes two critical components: learnable 1D Signal Modulators (SM) and SigFormer. SM learns implicit 1D signal patterns across various source images and embeds them into natural images, reducing the domain gap between synthetic and real datasets. SigFormer integrates Transformer with signal processing methods, establishing an appropriate signal representation space for SM. Its cascaded, multi-level design allows hierarchical feature learning from coarse to fine detail. Moreover, SigFormer can serve as a flexible backbone for IF, as its design adheres to the classic decomposition-reconstruction paradigm. Experimental results demonstrate that SigFusion achieves state-of-the-art performance across multiple IF tasks, including medical image fusion, infrared-visible image fusion, multi-focus image fusion, and multi-exposure image fusion.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning

Jiayu Wang
Yifei Ming
Zixuan Ke
Caiming Xiong
Shafiq Joty
Aws Albarghouthi
Frederic Sala

Reinforcement learning (RL) has become the dominant paradigm for improving the performance of language models on complex reasoning tasks. Despite the substantial empirical gains demonstrated by RL-based training methods like GRPO, a granular understanding of why and how RL enhances performance is still lacking. To bridge this gap, we introduce SPARKLE, a fine-grained analytic framework to dissect the effects of RL across three key dimensions: (1) plan following and execution, (2) knowledge integration, and (3) chain of subproblems. Using this framework, we gain insights beyond mere accuracy. For instance, providing models with explicit human-crafted, step-by-step plans can surprisingly degrade performance on the most challenging benchmarks, yet RL-tuned models exhibit greater robustness, experiencing markedly smaller performance drops than base or SFT models. This suggests that RL may not primarily enhance the execution of external plans but rather empower models to formulate and follow internal strategies better suited to their reasoning processes. Conversely, we observe that RL enhances models' ability to integrate provided knowledge into their reasoning process, yielding consistent gains across diverse tasks. Finally, we study whether difficult problems---those yielding no RL signals and mixed-quality reasoning traces---can still be effectively used for training. We introduce SparkleRL-PSS, a multi-stage RL pipeline that reuses hard problems with partial step scaffolding, guiding exploration effectively without additional data generation. Together, our findings provide a principled foundation for understanding how RL shapes model behavior, offering practical insights for building more adaptive, data-efficient, and interpretable RL pipelines for reasoning tasks. Our code, data, and checkpoints are available at: https: //sparkle-reasoning. github. io/.

PDF Details

YNIMG Journal 2025 Journal Article

Dynamic grouping of ongoing activity in V1 hypercolumns

Rui Zhang
Jiayu Wang
Xingya Cai
Rendong Tang
Haidong D. Lu

Neurons' spontaneous activity provides rich information about the brain. A single neuron's activity has close relationships with the local network. In order to understand such relationships, we studied the spontaneous activity of thousands of neurons in macaque V1 and V2 with two-photon calcium imaging. In V1, the ongoing activity was dominated by global fluctuations in which the activity of majority of neurons were correlated. Neurons' activity also relied on their relative locations within the local functional architectures, including ocular dominance, orientation, and color maps. Neurons with similar preferences dynamically grouped into co-activating ensembles and exhibited spatial patterns resembling the local functional maps. Different ensembles had different strengths and frequencies. This observation was consistent across all hypercolumn-sized V1 locations we examined. In V2, different imaging sites had different orientation and color features. However, the spontaneous activity in the sampled regions also correlated with the underlying functional architectures. These results indicate that functional architectures play an essential role in influencing neurons' spontaneous activity.

Details DOI

NeurIPS Conference 2025 Conference Paper

EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation

Xiaofeng Wang
Kang Zhao
Feng Liu
Jiayu Wang
Guosheng Zhao
Xiaoyi Bao
Zheng Zhu
Yingya Zhang

Video generation has emerged as a promising tool for world simulation, leveraging visual data to replicate real-world environments. Within this context, egocentric video generation, which centers on the human perspective, holds significant potential for enhancing applications in virtual reality, augmented reality, and gaming. However, the generation of egocentric videos presents substantial challenges due to the dynamic nature of first-person viewpoints, the intricate diversity of actions, and the complex variety of scenes encountered. Existing datasets are inadequate for addressing these challenges effectively. To bridge this gap, we present EgoVid-5M, the first high-quality dataset specifically curated for egocentric video generation. EgoVid-5M encompasses over 5 million egocentric video clips and is enriched with detailed action annotations, including fine-grained kinematic control and high-level textual descriptions. To ensure the integrity and usability of the dataset, we implement a sophisticated data cleansing pipeline designed to maintain frame consistency, action coherence, and motion smoothness under egocentric conditions. Furthermore, we introduce EgoDreamer, which is capable of generating egocentric videos driven simultaneously by action descriptions and kinematic control signals. The EgoVid-5M dataset, associated action annotations, and all data cleansing metadata will be released for the advancement of research in egocentric video generation.

PDF Details

AAAI Conference 2025 Conference Paper

FaceA-Net: Facial Attribute-Driven ID Preserving Image Generation Network

Jiayu Wang
Yue Yu
Jingjing Chen
Qi Dai
Yu-Gang Jiang

Recent advances in diffusion-based generative models have demonstrated superior performance in subject-driven image generation. Identity (ID) preserving image generation, as a subtask of subject-driven image generation, aims to generate customized images for specific human identity and has broad application potential. However, this task remains challenging due to the requirement for high ID fidelity and precise detail preservation. Additionally, generating high-quality context presents another challenge, as existing methods struggle to achieve both high ID fidelity and satisfactory context simultaneously. To address the issues of insufficient ID fidelity, we introduce a simple yet effective test-time fine-tuning approach. Specifically, we propose an attribute-driven training method that establishes global-level and local-level tasks to learn the global face feature and fine-grained attribute features, respectively. Furthermore, we introduce a novel ID-context decoupling framework that decouples image context generation from human ID generation, ensuring the quality of contextual content as well as facilitating the learning of ID information. Through extensive experiments, we demonstrate the effectiveness of the proposed method and showcase its capabilities across various applications.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Grammar-Aligned Decoding

Kanghee Park
Jiayu Wang
Taylor Berg-Kirkpatrick
Nadia Polikarpova
Loris D'Antoni

Large Language Models (LLMs) struggle with reliably generating highly structured outputs, such as program code, mathematical formulas, or well-formed markup. Constrained decoding approaches mitigate this problem by greedily restricting what tokens an LLM can output at each step to guarantee that the output matches a given constraint. Specifically, in grammar-constrained decoding (GCD), the LLM's output must follow a given grammar. In this paper we demonstrate that GCD techniques (and in general constrained decoding techniques) can distort the LLM's distribution, leading to outputs that are grammatical but appear with likelihoods that are not proportional to the ones given by the LLM, and so ultimately are low-quality. We call the problem of aligning sampling with a grammar constraint, grammar-aligned decoding (GAD), and propose adaptive sampling with approximate expected futures (ASAp), a decoding algorithm that guarantees the output to be grammatical while provably producing outputs that match the conditional probability of the LLM's distribution conditioned on the given grammar constraint. Our algorithm uses prior sample outputs to soundly overapproximate the future grammaticality of different output prefixes. Our evaluation on code generation and structured NLP tasks shows how ASAp often produces outputs with higher likelihood (according to the LLM's distribution) than existing GCD techniques, while still enforcing the desired grammatical constraints.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models

Jiayu Wang
Yifei Ming
Zhenmei Shi
Vibhav Vineet
Xin Wang
Yixuan Li
Neel Joshi

Large language models (LLMs) and vision-language models (VLMs) have demonstrated remarkable performance across a wide range of tasks and domains. Despite this promise, spatial understanding and reasoning—a fundamental component of human cognition—remains under-explored. We propose SpatialEval, a novel benchmark that covers diverse aspects of spatial reasoning such as relationship understanding, navigation, and counting. We conduct a comprehensive evaluation of competitive language and vision-language models. Our findings reveal several counter-intuitive insights that have been overlooked in the literature: (1) Spatial reasoning poses significant challenges where competitive models can fall behind random guessing; (2) Despite additional visual input, VLMs often under-perform compared to their LLM counterparts; (3) When both textual and visual information is available, multi-modal language models become less reliant on visual information if sufficient textual clues are provided. Additionally, we demonstrate that leveraging redundancy between vision and text can significantly enhance model performance. We hope our study will inform the development of multimodal models to improve spatial intelligence and further close the gap with human intelligence. Our code is available at https: //github. com/jiayuww/SpatialEval.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Lean Workbook: A large-scale Lean problem set formalized from natural language math problems

Huaiyuan Ying
Zijian Wu
Yihan Geng
Jiayu Wang
Dahua Lin
Kai Chen

Large language models have demonstrated impressive capabilities across various natural language processing tasks, especially in solving mathematical problems. However, large language models are not good at math theorem proving using formal languages like Lean. A significant challenge in this area is the scarcity of training data available in these formal languages. To address this issue, we propose a novel pipeline that iteratively generates and filters synthetic data to translate natural language mathematical problems into Lean 4 statements, and vice versa. Our results indicate that the synthetic data pipeline can provide useful training data and improve the performance of LLMs in translating and understanding complex mathematical problems and proofs. Our final dataset contains about 57K formal-informal question pairs along with searched proof from the math contest forum and 21 new IMO questions. We open-source our code at \url{https: //github. com/InternLM/InternLM-Math} and our data at \url{https: //huggingface. co/datasets/InternLM/Lean-Workbook}.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

FaceComposer: A Unified Model for Versatile Facial Content Creation

Jiayu Wang
Kang Zhao
Yifeng Ma
Shiwei Zhang
Yingya Zhang
Yujun Shen
Deli Zhao
Jingren Zhou

This work presents FaceComposer, a unified generative model that accomplishes a variety of facial content creation tasks, including text-conditioned face synthesis, text-guided face editing, face animation etc. Based on the latent diffusion framework, FaceComposer follows the paradigm of compositional generation and employs diverse face-specific conditions, e. g. , Identity Feature and Projected Normalized Coordinate Code, to release the model creativity at all possible. To support text control and animation, we clean up some existing face image datasets and collect around 500 hours of talking-face videos, forming a high-quality large-scale multi-modal face database. A temporal self-attention module is incorporated into the U-Net structure, which allows learning the denoising process on the mixture of images and videos. Extensive experiments suggest that our approach not only achieves comparable or even better performance than state-of-the-arts on each single task, but also facilitates some combined tasks with one-time forward, demonstrating its potential in serving as a foundation generative model in face domain. We further develop an interface such that users can enjoy our one-step service to create, edit, and animate their own characters. Code, dataset, model, and interface will be made publicly available.

PDF Details

YNIMG Journal 2023 Journal Article

Mesoscale functional connectivity in macaque visual areas

Xingya Cai
Haoran Xu
Chao Han
Peichao Li
Jiayu Wang
Rui Zhang
Rendong Tang
Chen Fang

Studies of resting-state functional connectivity (rsFC) have provided rich insights into the structures and functions of the human brain. However, most rsFC studies have focused on large-scale brain connectivity. To explore rsFC at a finer scale, we used intrinsic signal optical imaging to image the ongoing activity of the anesthetized macaque visual cortex. Differential signals from functional domains were used to quantify network-specific fluctuations. In 30-60 min resting-state imaging, a series of coherent activation patterns were observed in all three visual areas we examined (V1, V2, and V4). These patterns matched the known functional maps (ocular dominance, orientation, color) obtained in visual stimulation conditions. These functional connectivity (FC) networks fluctuated independently over time and exhibited similar temporal characteristics. Coherent fluctuations, however, were observed from orientation FC networks in different areas and even across two hemispheres. Thus, FC in the macaque visual cortex was fully mapped both on a fine scale and over a long range. Hemodynamic signals can be used to explore mesoscale rsFC in a submillimeter resolution.

Details DOI

IS Journal 2021 Journal Article

Isolation Forest Based Anomaly Detection Framework on Non-IID Data

Haolong Xiang
Jiayu Wang
Kotagiri Ramamohanarao
Zoran Salcic
Wanchun Dou
Xuyun Zhang

Anomaly detection is a significant but challenging data mining task in a wide range of applications. Different domains usually use different ways to measure the characteristics of data and to define the anomaly types. As a result, it is a big challenge to develop a versatile anomaly detection framework that can be universally applied with satisfactory performance in most, if not all, applications. In this article, we propose a generic isolation forest based ensemble framework named EDBHiForest, which can be universally applied to data spaces with arbitrary distance measures. It is realized through embedding the isolation forest structure with extended distance-based hashing (EDBH), which can significantly enhance the versatility and applicability of isolation forest based anomaly detection. This framework overcomes the limitations of existing isolation forest based methods that can only be applied to datasets with a very limited range of distance measure types. Extensive experiments on various non-independent and identically distributed datasets demonstrate the effectiveness and efficiency of our approach.

Details DOI