Author name cluster

Pengyu Wang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

11 papers

1 author row

JBHI Journal 2026 Journal Article

Physically Informed 3D Food Reconstruction: Methods and Results

Jiangpeng He
Yuhao Chen
Gautham Vinod
Xiaoyan Zhang
Talha Ibn Mahmud
Ahmad AlMughrabi
Umair Haroon
Ricardo Marques

Accurate food portion size estimation is a critical challenge in nutrition analysis and dietary assessment. Recent 3D reconstruction methods primarily focus on surface geometry, often neglecting the volumetric accuracy necessary for precise portion size estimation. In this work, we present three methods, NS-DRS, HR-CMS, and GS-GP, for real-scale 3D food reconstruction and volume estimation from limited 2D inputs. All three methods follow a shared three-stage pipeline consisting of 3D reconstruction, scale estimation using physical references such as visible checkerboards, and mesh refinement. While this high-level structure is consistent, the methods differ in their reconstruction backbones and scale-recovery mechanisms, and are designed for different input conditions, including both single-view and multi-view images. We evaluate each method on a diverse set of food items exhibiting variations in texture, shape, and camera pose, and assess their performance in terms of volume estimation and geometric accuracy. The results reveal complementary strengths among the pipelines, with NS-DRS performing better in volume estimation and GS-GP being more effective in 3D reconstruction. Across all settings, the three methods achieve 18–23% lower volume estimation error compared to the current state-of-the-art model. These findings demonstrate the effectiveness of physically informed and explainable reconstruction pipelines for accurate portion estimation, and support their potential use as scalable tools for dietary monitoring and clinical nutrition analysis.

Details DOI

AAAI Conference 2026 Conference Paper

Proactive Constrained Policy Optimization with Preemptive Penalty

Ning Yang
Pengyu Wang
Guoqing Liu
Haifeng Zhang
Pin Lyu
Jun Wang

Safe Reinforcement Learning (RL) often faces significant issues such as constraint violations and instability, necessitating the use of constrained policy optimization, which seeks optimal policies while ensuring adherence to specific constraints like safety. Typically, constrained optimization problems are addressed by the Lagrangian method, a post-violation remedial approach that may result in oscillations and overshoots. Motivated by this, we propose a novel method named Proactive Constrained Policy Optimization (PCPO) that incorporates a preemptive penalty mechanism. This mechanism integrates barrier items into the objective function as the policy nears the boundary, imposing a cost. Meanwhile, we introduce a constraint-aware intrinsic reward to guide boundary-aware exploration, which is activated only when the policy approaches the constraint boundary. We establish theoretical upper and lower bounds for the duality gap and the performance of the PCPO update, shedding light on the method's convergence characteristics. Additionally, to enhance the optimization performance, we adopt a policy iteration approach. An interesting finding is that PCPO demonstrates significant stability in experiments. Experimental results indicate that the PCPO framework provides a robust solution for policy optimization under constraints, with important implications for future research and practical applications.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

MS-BART: Unified Modeling of Mass Spectra and Molecules for Structure Elucidation

Yang Han
Pengyu Wang
Kai Yu
Xin Chen
Lu Chen

Mass spectrometry (MS) plays a critical role in molecular identification, significantly advancing scientific discovery. However, structure elucidation from MS data remains challenging due to the scarcity of annotated spectra. While large-scale pretraining has proven effective in addressing data scarcity in other domains, applying this paradigm to mass spectrometry is hindered by the complexity and heterogeneity of raw spectral signals. To address this, we propose MS-BART, a unified modeling framework that maps mass spectra and molecular structures into a shared token vocabulary, enabling cross-modal learning through large-scale pretraining on reliably computed fingerprint–molecule datasets. Multi-task pretraining objectives further enhance MS-BART's generalization by jointly optimizing denoising and translation task. The pretrained model is subsequently transferred to experimental spectra through finetuning on fingerprint predictions generated with MIST, a pre-trained spectral inference model, thereby enhancing robustness to real-world spectral variability. While finetuning alleviates the distributional difference, MS-BART still suffers molecular hallucination and requires further alignment. We therefore introduce a chemical feedback mechanism that guides the model toward generating molecules closer to the reference structure. Extensive evaluations demonstrate that MS-BART achieves SOTA performance across 5/12 key metrics on MassSpecGym and NPLIB1 and is faster by one order of magnitude than competing diffusion-based methods, while comprehensive ablation studies systematically validate the model's effectiveness and robustness. We provide the data and code at https: //github. com/OpenDFM/MS-BART.

PDF Details

AAAI Conference 2024 Conference Paper

DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning

Xinghao Wang
Junliang He
Pengyu Wang
Yunhua Zhou
Tianxiang Sun
Xipeng Qiu

Contrastive-learning-based methods have dominated sentence representation learning. These methods regularize the representation space by pulling similar sentence representations closer and pushing away the dissimilar ones and have been proven effective in various NLP tasks, e.g., semantic textual similarity (STS) tasks. However, it is challenging for these methods to learn fine-grained semantics as they only learn from the inter-sentence perspective, i.e., their supervision signal comes from the relationship between data samples. In this work, we propose a novel denoising objective that inherits from another perspective, i.e., the intra-sentence perspective. By introducing both discrete and continuous noise, we generate noisy sentences and then train our model to restore them to their original form. Our empirical evaluations demonstrate that this approach delivers competitive results on both semantic textual similarity (STS) and a wide range of transfer tasks, standing up well in comparison to contrastive-learning-based methods. Notably, the proposed intra-sentence denoising objective complements existing inter-sentence contrastive methodologies and can be integrated with them to further enhance performance. Our code is available at https://github.com/xinghaow99/DenoSent.

PDF Details DOI

EAAI Journal 2024 Journal Article

Graph-based multi-source domain adaptation with contrastive and collaborative learning for image deraining

Pengyu Wang
Hongqing Zhu
Huaqi Zhang
Ning Chen
Suyi Yang

Image deraining networks training on large-scale synthetic datasets have achieved good performance. However, when using these networks on unknown real-world scenes, a performance degradation will occurs due to the domain gap between synthetic and real rains. To alleviate this issue, this paper proposes a Graph-based Multi-source Domain Adaptive Deraining Network (GMDA-DN) with a joint contrastive and collaborative learning scheme. The proposed pipeline consists of the domain-shared image translation module and domain-specific image deraining modules. Firstly, considering the morphology correlations between synthetic and real rain streaks, we develop a Cross-graph Transformation Block (CTB) in the image translation module to produce realistic rain layers via the feature affine transformation guided by real rain information. Secondly, since similar rain streaks exist in multi-scale representations over both synthetic and real-world data. We propose a plug-and-play Scale-aware Graph Reasoning Block (SGRB), which achieves effective cross-scale information complementation and enhancement. Furthermore, to bridge multi-source domains and the target domain, a contrastive learning scheme is designed to push the predicted domain-specific rain layers far away as well as pull corresponding backgrounds closer at the pixel level, and a collaborative learning scheme is presented to align derained results across domains at the feature level. Extensive experiments demonstrate the proposed pipeline’s superiority on various synthetic and real-world datasets. For example, on R a i n 100 L and R a i n 100 H, our pipeline outperforms state-of-the-art image deraining methods by 1. 5 and 0. 5 in PSNR. Codes will be available at https: //github. com/wangpengyu0829/.

Details DOI

JBHI Journal 2024 Journal Article

MHD-Net: Memory-Aware Hetero-Modal Distillation Network for Thymic Epithelial Tumor Typing With Missing Pathology Modality

Huaqi Zhang
Jie Liu
Weifan Liu
Huang Chen
Zekuan Yu
Yixuan Yuan
Pengyu Wang
Jing Qin

Fusing multi-modal radiology and pathology data with complementary information can improve the accuracy of tumor typing. However, collecting pathology data is difficult since it is high-cost and sometimes only obtainable after the surgery, which limits the application of multi-modal methods in diagnosis. To address this problem, we propose comprehensively learning multi-modal radiology-pathology data in training, and only using uni-modal radiology data in testing. Concretely, a Memory-aware Hetero-modal Distillation Network (MHD-Net) is proposed, which can distill well-learned multi-modal knowledge with the assistance of memory from the teacher to the student. In the teacher, to tackle the challenge in hetero-modal feature fusion, we propose a novel spatial-differentiated hetero-modal fusion module (SHFM) that models spatial-specific tumor information correlations across modalities. As only radiology data is accessible to the student, we store pathology features in the proposed contrast-boosted typing memory module (CTMM) that achieves type-wise memory updating and stage-wise contrastive memory boosting to ensure the effectiveness and generalization of memory items. In the student, to improve the cross-modal distillation, we propose a multi-stage memory-aware distillation (MMD) scheme that reads memory-aware pathology features from CTMM to remedy missing modal-specific information. Furthermore, we construct a Radiology-Pathology Thymic Epithelial Tumor (RPTET) dataset containing paired CT and WSI images with annotations. Experiments on the RPTET and CPTAC-LUAD datasets demonstrate that MHD-Net significantly improves tumor typing and outperforms existing multi-modal methods on missing modality situations.

Details DOI

NeurIPS Conference 2024 Conference Paper

RealMAN: A Real-Recorded and Annotated Microphone Array Dataset for Dynamic Speech Enhancement and Localization

Bing Yang
Changsheng Quan
Yabo Wang
Pengyu Wang
Yujie Yang
Ying Fang
Nian Shao
Hui Bu

The training of deep learning-based multichannel speech enhancement and source localization systems relies heavily on the simulation of room impulse response and multichannel diffuse noise, due to the lack of large-scale real-recorded datasets. However, the acoustic mismatch between simulated and real-world data could degrade the model performance when applying in real-world scenarios. To bridge this simulation-to-real gap, this paper presents a new relatively large-scale Real-recorded and annotated Microphone Array speech&Noise (RealMAN) dataset. The proposed dataset is valuable in two aspects: 1) benchmarking speech enhancement and localization algorithms in real scenarios; 2) offering a substantial amount of real-world training data for potentially improving the performance of real-world applications. Specifically, a 32-channel array with high-fidelity microphones is used for recording. A loudspeaker is used for playing source speech signals (about 35 hours of Mandarin speech). A total of 83. 7 hours of speech signals (about 48. 3 hours for static speaker and 35. 4 hours for moving speaker) are recorded in 32 different scenes, and 144. 5 hours of background noise are recorded in 31 different scenes. Both speech and noise recording scenes cover various common indoor, outdoor, semi-outdoor and transportation environments, which enables the training of general-purpose speech enhancement and source localization networks. To obtain the task-specific annotations, speaker location is annotated with an omni-directional fisheye camera by automatically detecting the loudspeaker. The direct-path signal is set as the target clean speech for speech enhancement, which is obtained by filtering the source speech signal with an estimated direct-path propagation filter. Baseline experiments demonstrate that i) compared to using simulated data, the proposed dataset is indeed able to train better speech enhancement and source localization networks; ii) using various sub-arrays of the proposed 32-channel microphone array can successfully train variable-array networks that can be directly used to unseen arrays.

PDF Details DOI

TIST Journal 2024 Journal Article

Score-based Graph Learning for Urban Flow Prediction

Pengyu Wang
Xucheng Luo
Wenxin Tai
Kunpeng Zhang
Goce Trajcevsky
Fan Zhou

Accurate urban flow prediction (UFP) is crucial for a range of smart city applications such as traffic management, urban planning, and risk assessment. To capture the intrinsic characteristics of urban flow, recent efforts have utilized spatial and temporal graph neural networks to deal with the complex dependence between the traffic in adjacent areas. However, existing graph neural network based approaches suffer from several critical drawbacks, including improper graph representation of urban traffic data, lack of semantic correlation modeling among graph nodes, and coarse-grained exploitation of external factors. To address these issues, we propose DiffUFP, a novel probabilistic graph-based framework for UFP. DiffUFP consists of two key designs: (1) a semantic region dynamic extraction method that effectively captures the underlying traffic network topology, and (2) a conditional denoising score-based adjacency matrix generator that takes spatial, temporal, and external factors into account when constructing the adjacency matrix rather than simply concatenation in existing studies. Extensive experiments conducted on real-world datasets demonstrate the superiority of DiffUFP over the state-of-the-art UFP models and the effect of the two specific modules.

Details DOI

NeurIPS Conference 2024 Conference Paper

SpeechAlign: Aligning Speech Generation to Human Preferences

Dong Zhang
Zhaowei Li
Shimin Li
Xin Zhang
Pengyu Wang
Yaqian zhou
Xipeng Qiu

Speech language models have significantly advanced in generating realistic speech, with neural codec language models standing out. However, the integration of preference optimization to align speech outputs to human preferences is often neglected. This paper addresses this gap by first analyzing the distribution gap in codec language models, highlighting how it leads to discrepancies between the training and inference phases, which negatively affects performance. Then we explore leveraging preference optimization to bridge the distribution gap. We introduce SpeechAlign, an iterative self-improvement strategy that aligns speech language models to human preferences. SpeechAlign involves constructing a preference codec dataset contrasting golden codec tokens against synthetic tokens, followed by preference optimization to improve the codec language model. This cycle of improvement is carried out iteratively to steadily convert weak models to strong ones. Through both subjective and objective evaluations, we show that SpeechAlign can bridge the distribution gap and facilitating continuous self-improvement of the speech language model. Moreover, SpeechAlign exhibits robust generalization capabilities and works for smaller models. Demos are available at https: //0nutation. github. io/SpeechAlign. github. io/.

PDF Details DOI

JBHI Journal 2022 Journal Article

Cross-Boosted Multi-Target Domain Adaptation for Multi-Modality Histopathology Image Translation and Segmentation

Huaqi Zhang
Jie Liu
Pengyu Wang
Zekuan Yu
Weifan Liu
Huang Chen

Recent digital pathology workflows mainly focus on mono-modality histopathology image analysis. However, they ignore the complementarity between Haematoxylin & Eosin (H&E) and Immunohistochemically (IHC) stained images, which can provide comprehensive gold standard for cancer diagnosis. To resolve this issue, we propose a cross-boosted multi-target domain adaptation pipeline for multi-modality histopathology images, which contains Cross-frequency Style-auxiliary Translation Network (CSTN) and Dual Cross-boosted Segmentation Network (DCSN). Firstly, CSTN achieves the one-to-many translation from fluorescence microscopy images to H&E and IHC images for providing source domain training data. To generate images with realistic color and texture, Cross-frequency Feature Transfer Module (CFTM) is developed to pertinently restructure and normalize high-frequency content and low-frequency style features from different domains. Then, DCSN fulfills multi-target domain adaptive segmentation, where a dual-branch encoder is introduced, and Bidirectional Cross-domain Boosting Module (BCBM) is designed to implement cross-modality information complementation through bidirectional inter-domain collaboration. Finally, we establish Multi-modality Thymus Histopathology (MThH) dataset, which is the largest publicly available H&E and IHC image benchmark. Experiments on MThH dataset and several public datasets show that the proposed pipeline outperforms state-of-the-art methods on both histopathology image translation and segmentation.

Details DOI

TIST Journal 2021 Journal Article

Contrastive Trajectory Learning for Tour Recommendation

Fan Zhou
Pengyu Wang
Xovee Xu
Wenxin Tai
Goce Trajcevski

The main objective of Personalized Tour Recommendation (PTR) is to generate a sequence of point-of-interest (POIs) for a particular tourist, according to the user-specific constraints such as duration time, start and end points, the number of attractions planned to visit, and so on. Previous PTR solutions are based on either heuristics for solving the orienteering problem to maximize a global reward with a specified budget or approaches attempting to learn user visiting preferences and transition patterns with the stochastic process or recurrent neural networks. However, existing learning methodologies rely on historical trips to train the model and use the next visited POI as the supervised signal, which may not fully capture the coherence of preferences and thus recommend similar trips to different users, primarily due to the data sparsity problem and long-tailed distribution of POI popularity. This work presents a novel tour recommendation model by distilling knowledge and supervision signals from the trips in a self-supervised manner. We propose Contrastive Trajectory Learning for Tour Recommendation (CTLTR), which utilizes the intrinsic POI dependencies and traveling intent to discover extra knowledge and augments the sparse data via pre-training auxiliary self-supervised objectives. CTLTR provides a principled way to characterize the inherent data correlations while tackling the implicit feedback and weak supervision problems by learning robust representations applicable for tour planning. We introduce a hierarchical recurrent encoder-decoder to identify tourists’ intentions and use the contrastive loss to discover subsequence semantics and their sequential patterns through maximizing the mutual information. Additionally, we observe that a data augmentation step as the preliminary of contrastive learning can solve the overfitting issue resulting from data sparsity. We conduct extensive experiments on a range of real-world datasets and demonstrate that our model can significantly improve the recommendation performance over the state-of-the-art baselines in terms of both recommendation accuracy and visiting orders.

Details DOI