Author name cluster

Sifeng He

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

4 papers

2 author rows

NeurIPS Conference 2025 Conference Paper

Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment

Pengfei Zhao
Rongbo Luan
Wei Zhang
Peng Wu
Sifeng He

Despite Contrastive Language–Image Pre-training (CLIP)'s remarkable capability to retrieve content across modalities, a substantial modality gap persists in its feature space. Intriguingly, we discover that off-the-shelf MLLMs (Multimodal Large Language Models) demonstrate powerful inherent modality alignment properties. While recent MLLM-based retrievers with unified architectures partially mitigate this gap, their reliance on coarse modality alignment mechanisms fundamentally limits their potential. In this work, We introduce MAPLE (Modality-Aligned Preference Learning for Embeddings), a novel framework that leverages the fine-grained alignment priors inherent in MLLM to guide cross-modal representation learning. MAPLE formulates the learning process as reinforcement learning with two key components: (1) Automatic preference data construction using off-the-shelf MLLM, and (2) a new Relative Preference Alignment (RPA) loss, which adapts Direct Preference Optimization (DPO) to the embedding learning setting. Experimental results show that our preference-guided alignment achieves substantial gains in fine-grained cross-modal retrieval, underscoring its effectiveness in handling nuanced semantic distinctions.

PDF Details

AAAI Conference 2024 Conference Paper

Let All Be Whitened: Multi-Teacher Distillation for Efficient Visual Retrieval

Zhe Ma
Jianfeng Dong
Shouling Ji
Zhenguang Liu
Xuhong Zhang
Zonghui Wang
Sifeng He
Feng Qian

Visual retrieval aims to search for the most relevant visual items, e.g., images and videos, from a candidate gallery with a given query item. Accuracy and efficiency are two competing objectives in retrieval tasks. Instead of crafting a new method pursuing further improvement on accuracy, in this paper we propose a multi-teacher distillation framework Whiten-MTD, which is able to transfer knowledge from off-the-shelf pre-trained retrieval models to a lightweight student model for efficient visual retrieval. Furthermore, we discover that the similarities obtained by different retrieval models are diversified and incommensurable, which makes it challenging to jointly distill knowledge from multiple models. Therefore, we propose to whiten the output of teacher models before fusion, which enables effective multi-teacher distillation for retrieval models. Whiten-MTD is conceptually simple and practically effective. Extensive experiments on two landmark image retrieval datasets and one video retrieval dataset demonstrate the effectiveness of our proposed method, and its good balance of retrieval performance and efficiency. Our source code is released at https://github.com/Maryeon/whiten_mtd.

PDF Details DOI

AAAI Conference 2023 Conference Paper

TransVCL: Attention-Enhanced Video Copy Localization Network with Flexible Supervision

Sifeng He
Yue He
Minlong Lu
Chen Jiang
Xudong Yang
Feng Qian
Xiaobo Zhang
Lei Yang

Video copy localization aims to precisely localize all the copied segments within a pair of untrimmed videos in video retrieval applications. Previous methods typically start from frame-to-frame similarity matrix generated by cosine similarity between frame-level features of the input video pair, and then detect and refine the boundaries of copied segments on similarity matrix under temporal constraints. In this paper, we propose TransVCL: an attention-enhanced video copy localization network, which is optimized directly from initial frame-level features and trained end-to-end with three main components: a customized Transformer for feature enhancement, a correlation and softmax layer for similarity matrix generation, and a temporal alignment module for copied segments localization. In contrast to previous methods demanding the handcrafted similarity matrix, TransVCL incorporates long-range temporal information between feature sequence pair using self- and cross- attention layers. With the joint design and optimization of three components, the similarity matrix can be learned to present more discriminative copied patterns, leading to significant improvements over previous methods on segment-level labeled datasets (VCSL and VCDB). Besides the state-of-the-art performance in fully supervised setting, the attention architecture facilitates TransVCL to further exploit unlabeled or simply video-level labeled data. Additional experiments of supplementing video-level labeled datasets including SVD and FIVR reveal the high flexibility of TransVCL from full supervision to semi-supervision (with or without video-level annotation). Code is publicly available at https://github.com/transvcl/TransVCL.

PDF Details DOI

IROS Conference 2017 Conference Paper

A regularized on-line sequential extreme learning machine with forgetting property for fast dynamic hysteresis modeling

Zelong Wu
Hui Tang 0003
Sifeng He
Jian Gao 0002
Xin Chen 0005
Chengqiang Cui
Yunbo He
Kai Zhang

Piezoelectric ceramics(PZT)actuator has been widely used in flexure-guided nanopositioning stage because of their high resolution. However, it is quite hard to achieve high-rate precision positioning control because of the complex hysteresis nonlinearity effect of PZT actuator. Thus, an online RELM algorithm with forgetting property(FReOS-ELM) is proposed to handle this issue. Firstly, we adopt regularized extreme learning machine(RELM)to build an intelligent hysteresis model. The training of the algorithm is completed only in one step, which avoids the shortcomings of the traditional hysteresis model based on artificial neural network(ANN) that slow training speed and easy to fall into the local minimum. Then, based on the regularized on-line sequential extreme learning machine(ReOS-ELM), an on-line RELM algorithm with forgetting property(FReOS-ELM) is designed, which can avoid the computational load of ReOS-ELM in the process of adding new data for learning on-line. In the experiment, a real-time voltage signal with varying frequencies and amplitudes is adopted, and the output displacement data of the nanopositioning stage is also acquired and analyzed. The results powerfully verify that the performance of the established hysteresis model based on the proposed FReOS-ELM is satisfactory, which can be used to improve the practical positioning performance for flexure nanopositioning stage.

Details