Author name cluster

Haoran Wei

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

15 papers

2 author rows

AAAI Conference 2026 Conference Paper

Towards High-Fidelity 3D Portrait Generation with Rich Details by Cross-View Prior-Aware Diffusion

Haoran Wei
Wencheng Han
Xingping Dong
Jianbing Shen

Recent diffusion-based Single-image 3D portrait generation methods typically employ 2D diffusion models to provide multi-view knowledge, which is then distilled into 3D representations. However, these methods usually struggle to produce high-fidelity 3D models, frequently yielding excessively blurred textures. We attribute this issue to the insufficient consideration of cross-view consistency during the diffusion process, resulting in significant disparities between different views and ultimately leading to blurred 3D representations. In this paper, we address this issue by comprehensively exploiting multi-view priors in both the conditioning and diffusion procedures to produce consistent, detail-rich portraits. From the conditioning standpoint, we propose a Hybrid Priors Diffusion model, which explicitly and implicitly incorporates multi-view priors as conditions to enhance the status consistency of the generated multi-view portraits. From the diffusion perspective, considering the significant impact of the diffusion noise distribution on detailed texture generation, we propose a Multi-View Noise Resampling Strategy integrated within the optimization process leveraging cross-view priors to enhance representation consistency. Extensive experiments show that our method produces 3D portraits with accurate geometry and rich details from a single image.

PDF Details DOI

AIIM Journal 2025 Journal Article

MMSupcon: An image fusion-based multi-modal supervised contrastive method for brain tumor diagnosis

Haoyu Wang
Jing Zhang
Siying Wu
Haoran Wei
Xun Chen
Yunwei Ou
Xiaoyan Sun

The diagnosis of brain tumors is pivotal for effective treatment, with MRI serving as a commonly used non-invasive diagnostic modality in clinical practices. Fundamentally, brain tumor diagnosis is a type of pattern recognition task that requires the integration of information from multi-modal MRI images. However, existing fusion strategies are hindered by the scarcity of multi-modal imaging samples. In this paper, we propose a new training paradigm tailored for the scenario of multi-modal imaging in brain tumor diagnosis, called multi-modal supervised contrastive learning method (MMSupcon). This method significantly enhances diagnostic accuracy through two key components: multi-modal medical image fusion and multi-modal supervised contrastive loss. First, the fusion component integrates complementary imaging modalities to generate information-rich samples. Second, by introducing fused samples to guide original samples in learning feature consistency or inconsistency among classes, our loss component effectively preserves the integrity of cross-modal information while maintaining the distinctiveness of individual modalities. Finally, MMSupcon is validated on a real-world brain tumor dataset collected from Beijing Tiantan Hospital, achieving state-of-the-art performance. Furthermore, additional experiments on two public BraTS glioma classification datasets also demonstrate our substantial performance improvements. The source code is released at https: //github. com/hywang02/MMSupcon.

Details DOI

ICML Conference 2025 Conference Paper

Perception in Reflection

Yana Wei
Liang Zhao
Kangheng Lin
En Yu
Yuang Peng
Runpei Dong
Jianjian Sun
Haoran Wei

We present a perception in reflection paradigm designed to transcend the limitations of current large vision-language models (LVLMs), which are expected yet often fail to achieve perfect perception initially. Specifically, we propose Reflective Perception (RePer), a dual-model reflection mechanism that systematically alternates between policy and critic models, enables iterative refinement of visual perception. This framework is powered by Reflective Perceptual Learning (RPL), which reinforces intrinsic reflective capabilities through a methodically constructed visual reflection dataset and reflective unlikelihood training Comprehensive experimental evaluation demonstrates RePer’s quantifiable improvements in image understanding, captioning precision, and hallucination reduction. Notably, RePer achieves strong alignment between model attention patterns and human visual focus, while RPL optimizes fine-grained and free-form preference alignment. These advancements establish perception in reflection as a robust paradigm for future multimodal agents, particularly in tasks requiring complex reasoning and multi-step manipulation. Project Page: https: //weiyana. github. io/Perception-in-Reflection

Details

NeurIPS Conference 2025 Conference Paper

Perception-R1: Pioneering Perception Policy with Reinforcement Learning

En Yu
Kangheng Lin
Liang Zhao
jisheng yin
Yana Wei
Yuang Peng
Haoran Wei
Jianjian Sun

Inspired by the success of DeepSeek-R1, we explore the potential of rule-based reinforcement learning (RL) in MLLM post-training for perception policy learning. While promising, our initial experiments reveal that incorporating a thinking process through RL does not consistently lead to performance gains across all visual perception tasks. This leads us to delve into the essential role of RL in the context of visual perception. In this work, we return to the fundamentals and explore the effects of RL on different perception tasks. We observe that the perceptual perplexity is a major factor in determining the effectiveness of RL. We also observe that reward design plays a crucial role in further approaching the upper limit of model perception. To leverage these findings, we propose Perception-R1, a scalable RL framework using GRPO during MLLM post-training. With a standard Qwen2-VL-2B-Instruct, Perception-R1 achieves +4. 2% on RefCOCO+, +17. 9% on PixMo-Count, +4. 2% on PageOCR, and notably, 31. 9% AP on COCO2017 val for the first time, establishing a strong baseline for perception policy learning.

PDF Details

ICLR Conference 2025 Conference Paper

Unhackable Temporal Reward for Scalable Video MLLMs

En Yu
Kangheng Lin
Liang Zhao
Yana Wei
Zining Zhu 0004
Haoran Wei
Jianjian Sun
Zheng Ge

In the pursuit of superior video-processing MLLMs, we have encountered a perplexing paradox: the “anti-scaling law”, where more data and larger models lead to worse performance. This study unmasks the culprit: “temporal hacking”, a phenomenon where models shortcut by fixating on select frames, missing the full video narrative. In this work, we systematically establish a comprehensive theory of temporal hacking, defining it from a reinforcement learning perspective, introducing the Temporal Perplexity (TPL) score to assess this misalignment, and proposing the Unhackable Temporal Rewarding (UTR) framework to mitigate the temporal hacking. Both theoretically and empirically, TPL proves to be a reliable indicator of temporal modeling quality, correlating strongly with frame activation patterns. Extensive experiments reveal that UTR not only counters temporal hacking but significantly elevates video comprehension capabilities. This work not only advances video-AI systems but also illuminates the critical importance of aligning proxy rewards with true objectives in MLLM development.

Details

IJCAI Conference 2024 Conference Paper

ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning

Liang Zhao
En Yu
Zheng Ge
Jinrong Yang
Haoran Wei
Hongyu Zhou
Jianjian Sun
Yuang Peng

Human-AI interactivity is a critical aspect that reflects the usability of Multimodal Large Language Models (MLLMs). However, existing end-to-end MLLMs only allow users to interact with them through language instructions, leading to the limitation of the interactive accuracy and efficiency. In this study, we present precise referring instructions that utilize diverse reference representations such as points and boxes as referring prompts to refer to the special region. This enables MLLMs to focus on the region of interest and achieve finer-grained interaction. Based on precise referring instruction, we propose ChatSpot, a unified end-to-end MLLM that supports diverse forms of interactivity including mouse clicks, drag-and-drop, and drawing boxes, which provides a more flexible and seamless interactive experience. We also construct a multi-grained vision-language instruction-following dataset based on existing datasets and GPT-4 generating. Furthermore, we design a series of evaluation tasks to assess the effectiveness of region recognition and interaction. Experimental results showcase ChatSpot's promising performance. Project page: https: //github. com/Ahnsun/ChatSpot.

PDF Details DOI

ICLR Conference 2024 Conference Paper

DreamLLM: Synergistic Multimodal Comprehension and Creation

Runpei Dong
Chunrui Han
Yuang Peng
Zekun Qi
Zheng Ge
Jinrong Yang
Liang Zhao
Jianjian Sun

This paper presents DreamLLM, a learning framework that first achieves versatile Multimodal Large Language Models (MLLMs) empowered with frequently overlooked synergy between multimodal comprehension and creation. DreamLLM operates on two fundamental principles. The first focuses on the generative modeling of both language and image posteriors by direct sampling in the raw multimodal space. This approach circumvents the limitations and information loss inherent to external feature extractors like CLIP, and a more thorough multimodal understanding is obtained. Second, DreamLLM fosters the generation of raw, interleaved documents, modeling both text and image contents, along with unstructured layouts. This allows DreamLLM to learn all conditional, marginal, and joint multimodal distributions effectively. As a result, DreamLLM is the first MLLM capable of generating free-form interleaved content. Comprehensive experiments highlight DreamLLM's superior performance as a zero-shot multimodal generalist, reaping from the enhanced learning synergy. Project page: https://dreamllm.github.io.

Details

AAAI Conference 2024 Conference Paper

GigaHumanDet: Exploring Full-Body Detection on Gigapixel-Level Images

Chenglong Liu
Haoran Wei
Jinze Yang
Jintao Liu
Wenxi Li
Yuchen Guo
Lu Fang

Performing person detection in super-high-resolution images has been a challenging task. For such a task, modern detectors, which usually encode a box using center and width/height, struggle with accuracy due to two factors: 1) Human characteristic: people come in various postures and the center with high freedom is difficult to capture robust visual pattern; 2) Image characteristic: due to vast scale diversity of input (gigapixel-level), distance regression (for width and height) is hard to pinpoint, especially for a person, with substantial scale, who is near the camera. To address these challenges, we propose GigaHumanDet, an innovative solution aimed at further enhancing detection accuracy for gigapixel-level images. GigaHumanDet employs the corner modeling method to avoid the potential issues of a high degree of freedom in center pinpointing. To better distinguish similar-looking persons and enforce instance consistency of corner pairs, an instance-guided learning approach is designed to capture discriminative individual semantics. Further, we devise reliable shape-aware bodyness equipped with a multi-precision strategy as the human corner matching guidance to be appropriately adapted to the single-view large scene. Experimental results on PANDA and STCrowd datasets show the superiority and strong applicability of our design. Notably, our model achieves 82.4% in term of AP, outperforming current state-of-the-arts by more than 10%.

PDF Details DOI

IROS Conference 2023 Conference Paper

Disentangled Discriminator for Unsupervised Domain Adaptation on Object Detection

Yangguang Zhu
Ping Guo
Haoran Wei
Xin Zhao
Xiangbin Wu

Object detection plays an important role in computer vision tasks such as autonomous driving, robotics, etc. Typically, a detection model is firstly trained on collected data and then deployed in real world. However, the discrepancy exists between training (source) and testing (target) data, which degrades the detection model's performance in the real world. To mitigate the negative effects, Unsupervised Domain Adaptation (UDA) methods learn the features of a shared domain via a discriminator. However, existing discriminators consider only the in-distribution adversarial learning, which ignore the out-of-distribution data of individual domains. In this paper, we propose a disentangled discriminator to consider the in-distribution and outliers separately. It aligns the source and target data with split branches under a gated strategy. We combine the disentangled discriminator with a Teacher-Student (T-S) framework that trains the student using labeled source data and unlabeled target data under a self-training mechanism. Specifically, the teacher network, that is updated with the parameters of student network via the exponential moving average, predicts pseudo labels for unlabeled data. The quality of pseudo labels can be improved after alleviating the domain discrepancy thanks to the disentangled discriminator. Extensive experiments on benchmarks demonstrate the superiority of the proposed method. Specifically, we achieve 53. 9% mAP on Foggy Cityscapes, which is 7. 2% higher than the Oracle.

Details

IJCAI Conference 2022 Conference Paper

Corner Affinity: A Robust Grouping Algorithm to Make Corner-guided Detector Great Again

Haoran Wei
Chenglong Liu
Ping Guo
Yangguang Zhu
Jiamei Fu
Bing Wang
Peng Wang

Corner-guided detector enjoys potential ability to yield precise bounding boxes. However, unreliable corner pairs, generated by heuristic grouping guidance, hinder the development of this detector. In this paper, we propose a novel corner grouping algorithm, termed as Corner Affinity, to significantly boost the reliability and robustness of corner grouping. The proposed Corner Affinity is a couple of two interactional factors, namely, 1) the structure affinity (SA), applying to generate preliminary corner pairs through the corresponding object's shallow construction information. 2) the contexts affinity (CA), running as optimizing corner pairs via embedding deeper semantic features of affiliated instances. Equipped with the Corner Affinity, a detector can produce high-quality bounding boxes upon preferable paired corner keypoints. Experimental results show the superiority of our design on multiple benchmark datasets. Specifically, for CornerNet baseline, the proposed Corner Affinity brings AP boostings of 5. 8% on COCO, 35. 8% on Citypersons, and 17. 2% on UCAS-AOD without bells and whistles.

PDF Details DOI

NeurIPS Conference 2022 Conference Paper

HumanLiker: A Human-like Object Detector to Model the Manual Labeling Process

Haoran Wei
Ping Guo
Yangguang Zhu
Chenglong Liu
Peng Wang

Popular object detection models generate bounding boxes in a different way than we humans. As an example, modern detectors yield object box either upon the regression of its center and width/height (center-guided detector), or by grouping paired estimated corners (corner-guided detector). However, that is not the pattern we manually label an object due to high degrees of freedom in searching centers or low efficiency of grouping corners. Empirically, humans run two steps to locate an object bounding box manually: 1) click the mouse at the top-left corner of object, and then drag the mouse to the bottom-right corner; 2) refine the corner positions to make the bounding box more precisely, if necessary. Inspired by this manual labeling process, we propose a novel human-like detector, termed as HumanLiker, which is devised as a two-stage end-to-end detector to simulate the two aforementioned. Like we humans in manual labeling, HumanLiker can effectively avert both the thorny center searching and heuristic corner grouping. Different from the mainstream detector branches, i. e. , the center/corner-guided methods, the HumanLiker provides a new paradigm which integrates the advantages of both branches to balance the detection efficiency and bounding box quality. On MS-COCO test-dev set, HumanLiker can achieve 50. 2%/51. 6% and 53. 8%/55. 6% in term of AP with ResNeXt-101 and SwinTransformer backbones in single/multi-scale testing, outperforming current popular center/corner-guided baselines (e. g. , DETR/CornerNet) by a large margin, with much less training epochs and higher inference FPS. Code will be available soon.

PDF Details

AAAI Conference 2022 Conference Paper

Non-parametric Online Learning from Human Feedback for Neural Machine Translation

Dongqi Wang
Haoran Wei
Zhirui Zhang
Shujian Huang
Jun Xie
Jiajun Chen

We study the problem of online learning with human feedback in the human-in-the-loop machine translation, in which the human translators revise the machine-generated translations and then the corrected translations are used to improve the neural machine translation (NMT) system. However, previous methods require online model updating or additional translation memory networks to achieve high-quality performance, making them inflexible and inefficient in practice. In this paper, we propose a novel non-parametric online learning method without changing the model structure. This approach introduces two k-nearest-neighbor (KNN) modules: one module memorizes the human feedback, which is the correct sentences provided by human translators, while the other balances the usage of the history human feedback and original NMT models adaptively. Experiments conducted on EMEA and JRC-Acquis benchmarks demonstrate that our proposed method obtains substantial improvements on translation accuracy and achieves better adaptation performance with less repeating human correction operations.

PDF Details

IS Journal 2021 Journal Article

How to Optimize an Academic Team When the Outlier Member is Leaving?

Shuo Yu
Jiaying Liu
Haoran Wei
Feng Xia
Hanghang Tong

An academic team is a highly cohesive collaboration group of scholars, which has been recognized as an effective way to improve scientific output in terms of both quality and quantity. However, the high staff turnover brings about a series of problems that may have negative influences on team performance. To address this challenge, we first detect the tendency of the member who may potentially leave. Here, the outlierness is defined with respect to familiarity, which is quantified by using collaboration intensity. It is assumed that if a team member has a higher familiarity with scholars outside the team, then this member might probably leave the team. To minimize the influence caused by the leaving of such an outlier member, we propose an optimization solution to find a proper candidate who can replace the outlier member. Based on random walk with graph kernel, our solution involves familiarity matching, skill matching, as well as structure matching. The proposed approach proves to be effective and outperforms existing methods when applied to computer science academic teams.

Details DOI

AAAI Conference 2020 Conference Paper

GRET: Global Representation Enhanced Transformer

Rongxiang Weng
Haoran Wei
Shujian Huang
Heng Yu
Lidong Bing
Weihua Luo
Jiajun Chen

Transformer, based on the encoder-decoder framework, has achieved state-of-the-art performance on several natural language generation tasks. The encoder maps the words in the input sentence into a sequence of hidden states, which are then fed into the decoder to generate the output sentence. These hidden states usually correspond to the input words and focus on capturing local information. However, the global (sentence level) information is seldom explored, leaving room for the improvement of generation quality. In this paper, we propose a novel global representation enhanced Transformer (GRET) to explicitly model global representation in the Transformer network. Speciﬁcally, in the proposed model, an external state is generated for the global representation from the encoder. The global representation is then fused into the decoder during the decoding process to improve generation quality. We conduct experiments in two text generation tasks: machine translation and text summarization. Experimental results on four WMT machine translation tasks and LCSTS text summarization task demonstrate the effectiveness of the proposed approach on natural language generation1.

PDF Details

ICML Conference 2020 Conference Paper

Learning to Navigate The Synthetically Accessible Chemical Space Using Reinforcement Learning

Sai Krishna Gottipati
Boris Sattarov
Sufeng Niu
Yashaswi Pathak
Haoran Wei
Shengchao Liu
Simon Blackburn
Karam M. J. Thomas

Over the last decade, there has been significant progress in the field of machine learning for de novo drug design, particularly in generative modeling of novel chemical structures. However, current generative approaches exhibit a significant challenge: they do not ensure that the proposed molecular structures can be feasibly synthesized nor do they provide the synthesis routes of the proposed small molecules, thereby seriously limiting their practical applicability. In this work, we propose a novel reinforcement learning (RL) setup for de novo drug design: Policy Gradient for Forward Synthesis (PGFS), that addresses this challenge by embedding the concept of synthetic accessibility directly into the de novo drug design system. In this setup, the agent learns to navigate through the immense synthetically accessible chemical space by subjecting initial commercially available molecules to valid chemical reactions at every time step of the iterative virtual synthesis process. The proposed environment for drug discovery provides a highly challenging test-bed for RL algorithms owing to the large state space and high-dimensional continuous action space with hierarchical actions. PGFS achieves state-of-the-art performance in generating structures with high QED and clogP. Moreover, we validate PGFS in an in-silico proof-of-concept associated with three HIV targets. Finally, we describe how the end-to-end training conceptualized in this study represents an important paradigm in radically expanding the synthesizable chemical space and automating the drug discovery process.

Details