Author name cluster

Cheng Han

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

8 papers

2 author rows

TMLR Journal 2026 Journal Article

Prompt-based Adaptation in Large-scale Vision Models: A Survey

Xi Xiao
Yunbei Zhang
Lin Zhao
Yiyang Liu
Xiaoying Liao
Zheda Mai
Xingjian Li
Xiao Wang

In computer vision, Visual Prompting (VP) and Visual Prompt Tuning (VPT) have recently emerged as lightweight and effective alternatives to full fine-tuning for adapting large-scale vision models within the ``pretrain-then-finetune'' paradigm. However, despite rapid progress, their conceptual boundaries remain blurred, as VP and VPT are frequently used interchangeably in current research, reflecting a lack of systematic distinction between these techniques and their respective applications. In this survey, we revisit the designs of VP and VPT from first principles, and conceptualize them within a unified framework termed Prompt-based Adaptation (PA). Within this framework, we distinguish methods based on their injection granularity: VP operates at the pixel level, while VPT injects prompts at the token level. We further categorize these methods by their generation mechanism into fixed, learnable, and generated prompts. Beyond the core methodologies, we examine PA’s integrations across diverse domains, including medical imaging, 3D point clouds, and vision-language tasks, as well as its role in test-time adaptation and trustworthy AI. We also summarize current benchmarks and identify key challenges and future directions. To the best of our knowledge, we are the first comprehensive survey dedicated to PA's methodologies and applications in light of their distinct characteristics. Our survey aims to provide a clear roadmap for researchers and practitioners in all area to understand and explore the evolving landscape of PA-related research.

PDF Details

JBHI Journal 2026 Journal Article

Resolving the Ambiguity of Complete-to-Partial Point Cloud Registration for Image-Guided Liver Surgery With Patches-to-Partial Matching

Zixin Yang
Jon S. Heiselman
Cheng Han
Kelly Merrell
Richard Simon
Cristian A. Linte

In image-guided liver surgery, the initial rigid alignment between preoperative and intraoperative data, often represented as point clouds, is crucial for providing sub-surface information from preoperative CT/MRI images to the surgeon during the procedure. Currently, this alignment is typically performed using semi-automatic methods, which, while effective to some extent, are prone to errors that demand manual correction. Alternatively, correspondence-based point cloud registration methods further offer a promising fully automatic solution. However, they may struggle in scenarios with limited intraoperative surface visibility, a common challenge in liver surgery, particularly in laparoscopic procedures, which we refer to as complete-to-partial ambiguity. We first illustrate this ambiguity by evaluating the performance of state-of-the-art learning-based point cloud registration methods on our carefully constructed in silico and in vitro datasets. Then, we propose a patches-to-partial matching strategy as a plug-and-play module to resolve the ambiguity, which can be seamlessly integrated into learning-based registration methods without disrupting their end-to-end structure. This approach effectively improves registration performance, especially in low-visibility conditions, reducing registration errors to 6. 7 mm ( $-$ 29%) in silico and 12. 5 mm ( $-$ 40%) in vitro, compared to state-of-the-art performance achieved by Lepard of 9. 5 mm and 20. 7 mm, respectively. The constructed benchmark and the proposed module establish a solid foundation for advancing applications of point cloud correspondence-based registration methods in image-guided liver surgery. Our code and datasets will be released at https://github.com/zixinyang9109/P2P.

Details DOI

NeurIPS Conference 2025 Conference Paper

All You Need is One: Capsule Prompt Tuning with a Single Vector

Yiyang Liu
James Liang
Heng Fan
Wenhao Yang
Yiming Cui
Xiaotian Han
Lifu Huangg
Dongfang Liu

Prompt-based learning has emerged as a parameter-efficient finetuning (PEFT) approach to facilitate Large Language Model (LLM) adaptation to downstream tasks by conditioning generation with task-aware guidance. Despite its successes, current prompt-based learning methods heavily rely on laborious grid searching for optimal prompt length and typically require considerable number of prompts, introducing additional computational burden. Worse yet, our pioneer findings indicate that the task-aware prompt design is inherently limited by its absence of instance-aware information, leading to a subtle attention interplay with the input sequence. In contrast, simply incorporating instance-aware information as a part of the guidance can enhance the prompt-tuned model performance without additional fine-tuning. Moreover, we find an interesting phenomenon, namely "attention anchor", that incorporating instance-aware tokens at the earliest position of the sequence can successfully preserve strong attention to critical structural information and exhibit more active attention interaction with all input tokens. In light of our observation, we introduce Capsule Prompt-Tuning (CaPT), an efficient and effective solution that leverages off-the-shelf, informative instance semantics into prompt-based learning. Our approach innovatively integrates both instance-aware and task-aware information in a nearly parameter-free manner (i. e. , one single capsule prompt). Empirical results demonstrate that our method can exhibit superior performance across various language tasks (e. g. , 84. 03\% average accuracy on T5-Large), serving as an "attention anchor, " while enjoying high parameter efficiency (e. g. , 0. 003\% of model parameters on Llama3. 2-1B).

PDF Details

NeurIPS Conference 2025 Conference Paper

Probabilistic Token Alignment for Large Language Model Fusion

Runjia Zeng
James Liang
Cheng Han
Zhiwen Cao
Jiahao Liu
Xiaojun Quan
Yingjie Victor Chen
Lifu Huangg

Training large language models (LLMs) from scratch can yield models with unique functionalities and strengths, but it is costly and often leads to redundant capabilities. A more cost-effective alternative is to fuse existing pre-trained LLMs with different architectures into a more powerful model. However, a key challenge in existing model fusion is their dependence on manually predefined vocabulary alignment, which may not generalize well across diverse contexts, leading to performance degradation in several evaluation. To solve this, we draw inspiration from distribution learning and propose the probabilistic token alignment method as a general and soft mapping for alignment, named as PTA-LLM. Our approach innovatively reformulates token alignment into a classic mathematical problem: optimal transport, seamlessly leveraging distribution-aware learning to facilitate more coherent model fusion. Apart from its inherent generality, PTA-LLM exhibits interpretability from a distributional perspective, offering insights into the essence of the token alignment. Empirical results demonstrate that probabilistic token alignment enhances the target model's performance across multiple capabilities.

PDF Details

NeurIPS Conference 2024 Conference Paper

Visual Fourier Prompt Tuning

Runjia Zeng
Cheng Han
Qifan Wang
Chunshu Wu
Tong Geng
Lifu Huang
Ying N. Wu
Dongfang Liu

With the scale of vision Transformer-based models continuing to grow, finetuning these large-scale pretrained models for new tasks has become increasingly parameter-intensive. Visual prompt tuning is introduced as a parameter-efficient finetuning (PEFT) method to this trend. Despite its successes, a notable research challenge persists within almost all PEFT approaches: significant performance degradation is observed when there is a substantial disparity between the datasets applied in pretraining and finetuning phases. To address this challenge, we draw inspiration from human visual cognition, and propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models. Our approach innovatively incorporates the Fast Fourier Transform into prompt embeddings and harmoniously considers both spatial and frequency domain information. Apart from its inherent simplicity and intuitiveness, VFPT exhibits superior performance across all datasets, offering a general solution to dataset challenges, irrespective of data disparities. Empirical results demonstrate that our approach outperforms current state-of-the-art baselines on two benchmarks, with low parameter usage (e. g. , 0. 57% of model parameters on VTAB-1k) and notable performance enhancements (e. g. , 73. 20% of mean accuracy on VTAB-1k). Our code is avaliable at https: //github. com/runtsang/VFPT.

PDF Details DOI

IJCAI Conference 2023 Conference Paper

Prompt Learns Prompt: Exploring Knowledge-Aware Generative Prompt Collaboration For Video Captioning

Liqi Yan
Cheng Han
Zenglin Xu
Dongfang Liu
Qifan Wang

Fine-tuning large vision-language models is a challenging task. Prompt tuning approaches have been introduced to learn fixed textual or visual prompts while freezing the pre-trained model in downstream tasks. Despite the effectiveness of prompt tuning, what do those learnable prompts learn remains unexplained. In this work, we explore whether prompts in the fine-tuning can learn knowledge-aware prompts from the pre-training, by designing two different sets of prompts in pre-training and fine-tuning phases respectively. Specifically, we present a Video-Language Prompt tuning (VL-Prompt) approach for video captioning, which first efficiently pre-train a video-language model to extract key information (e. g. , actions and objects) with flexibly generated Knowledge-Aware Prompt (KAP). Then, we design a Video-Language Prompt (VLP) to transfer the knowledge from the knowledge-aware prompts and fine-tune the model to generate full captions. Experimental results show the superior performance of our approach over several state-of-the-art baselines. We further demonstrate that the video-language prompts are well learned from the knowledge-aware prompts.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

Unified 3D Segmenter As Prototypical Classifiers

Zheyun Qin
Cheng Han
Qifan Wang
Xiushan Nie
Yilong Yin
Lu Xiankai

The task of point cloud segmentation, comprising semantic, instance, and panoptic segmentation, has been mainly tackled by designing task-specific network architectures, which often lack the flexibility to generalize across tasks, thus resulting in a fragmented research landscape. In this paper, we introduce ProtoSEG, a prototype-based model that unifies semantic, instance, and panoptic segmentation tasks. Our approach treats these three homogeneous tasks as a classification problem with different levels of granularity. By leveraging a Transformer architecture, we extract point embeddings to optimize prototype-class distances and dynamically learn class prototypes to accommodate the end tasks. Our prototypical design enjoys simplicity and transparency, powerful representational learning, and ad-hoc explainability. Empirical results demonstrate that ProtoSEG outperforms concurrent well-known specialized architectures on 3D point cloud benchmarks, achieving 72. 3%, 76. 4% and 74. 2% mIoU for semantic segmentation on S3DIS, ScanNet V2 and SemanticKITTI, 66. 8% mCov and 51. 2% mAP for instance segmentation on S3DIS and ScanNet V2, 62. 4% PQ for panoptic segmentation on SemanticKITTI, validating the strength of our concept and the effectiveness of our algorithm. The code and models are available at https: //github. com/zyqin19/PROTOSEG.

PDF Details

IROS Conference 2022 Conference Paper

Grasping State Analysis of Soft Manipulator Based on Flexible Tactile Sensor Array

Haoyuan Wang
Hongge Ru
Hongliang Lei
Chi Zhang
Cheng Han
Hao Wu 0028
Jian Huang 0001

Although the grasping state analysis is vital in the study of manipulators, the grasping state analysis of soft manipulators as an independent research topic is not much so far. This paper proposes a novel pneumatic soft manipulator with a flexible tactile sensor array (SM-FTSA). The flexible tactile sensor array comprises piezoresistive materials with a porous structure. An equal potential approach is adopted to realize the collection of tactile signals of the SM-FTSA. Inspired by the grasping analysis of rigid manipulators, we propose 4 grasping states for the SM-FTSA, including inflating, shaking, stable, and slipping. Based on the experimental data, we conduct grasping experiments on 12 objects with SM-FTSA, and we propose 10 features that reflect the grasping state. Several machine learning methods are utilized to classify the grasping state. Among them, the Random Forest method presents the best performance, and the average classification accuracy reaches 99%.

Details