Author name cluster

Fei Shen

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

25 papers

2 author rows

AAAI Conference 2026 Conference Paper

Cross-modal Proxy Evolving for OOD Detection with Vision-Language Models

Hao Tang
Yu Liu
Shuanglin Yan
Fei Shen
Shengfeng He
Jing Qin

Reliable zero-shot detection of out-of-distribution (OOD) inputs is critical for deploying vision-language models in open-world settings. However, the lack of labeled negatives in zero-shot OOD detection necessitates proxy signals that remain effective under distribution shift. Existing negative-label methods rely on a fixed set of textual proxies, which (i) sparsely sample the semantic space beyond in-distribution (ID) classes and (ii) remain static while only visual features drift, leading to cross-modal misalignment and unstable predictions. In this paper, we propose CoEvo, a training- and annotation-free test-time framework that performs bidirectional, sample-conditioned adaptation of both textual and visual proxies. Specifically, CoEvo introduces a proxy-aligned co-evolution mechanism to maintain two evolving proxy caches, which dynamically mines contextual textual negatives guided by test images and iteratively refines visual proxies, progressively realigning cross-modal similarities and enlarging local OOD margins. Finally, we dynamically re-weight the contributions of dual-modal proxies to obtain a calibrated OOD score that is robust to distribution shift. Extensive experiments on standard benchmarks demonstrate that CoEvo achieves state-of-the-art performance, improving AUROC by 1.33% and reducing FPR95 by 45.98% on ImageNet-1K compared to strong negative-label baselines.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Fine-Grained Image Retrieval via Dual-Vision Adaptation

Xin Jiang
Meiqi Cao
Hao Tang
Fei Shen
Zechao Li

Fine-Grained Image Retrieval~(FGIR) faces challenges in learning discriminative visual representations to retrieve images with similar fine-grained features. Current leading FGIR solutions typically follow two regimes: enforce pairwise similarity constraints in the semantic embedding space, or incorporate a localization sub-network to fine-tune the entire model. However, such two regimes tend to overfit the training data while forgetting the knowledge gained from large-scale pre-training, thus reducing their generalization ability. In this paper, we propose a Dual-Vision Adaptation (DVA) approach for FGIR, which guides the frozen pre-trained model to perform FGIR through collaborative sample and feature adaptation. Specifically, we design Object-Perceptual Adaptation, which modifies input samples to help the pre-trained model perceive critical objects and elements within objects that are helpful for category prediction. Meanwhile, we propose In-Context Adaptation, which introduces a small set of parameters for feature adaptation without modifying the pre-trained parameters. This makes the FGIR task using these adapted features closer to the task solved during the pre-training. Additionally, to balance retrieval efficiency and performance, we propose Discrimination Perception Transfer to transfer the discriminative knowledge in the object-perceptual adaptation to the image encoder using the knowledge distillation mechanism. Extensive experiments show that DVA performs well on three fine-grained datasets.

PDF Details DOI

AAAI Conference 2026 Conference Paper

HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignment

Ruijia Wu
Ping Chen
Fei Shen
Shaoan Zhao
Qiang Hui
Huanlin Gao
Ting Lu
Zhaoxiang Liu

Contrastive vision-language models like CLIP have achieved impressive results in image-text retrieval by aligning image and text representations in a shared embedding space. However, these models often treat text as flat sequences, limiting their ability to handle complex, compositional, and long-form descriptions. In particular, they fail to capture two essential properties of language: semantic hierarchy, which reflects the multi-level compositional structure of text, and semantic monotonicity, where richer descriptions should result in stronger alignment with visual content. To address these limitations, we propose HiMo-CLIP, a representation-level framework that enhances CLIP-style models without modifying the encoder architecture. HiMo-CLIP introduces two key components: a hierarchical decomposition (HiDe) module that extracts latent semantic components from long-form text via in-batch PCA, enabling flexible, batch-aware alignment across different semantic granularities, and a monotonicity-aware contrastive loss (MoLo) that jointly aligns global and component-level representations, encouraging the model to internalize semantic ordering and alignment strength as a function of textual completeness. These components work together to produce structured, cognitively aligned cross-modal representations. Experiments on multiple image-text retrieval benchmarks show that HiMo-CLIP consistently outperforms strong baselines, particularly under long or compositional descriptions.

PDF Details DOI

AAAI Conference 2026 Conference Paper

IMAGGarment+: Efficient Attribute-Wise Diffusion for Garment Generation

Jian Yu
Fei Shen
Cong Wang
Yanpeng Sun
Hao Tang
Qin Guo
Xiaoyu Du

Diffusion models have advanced fine-grained garment generation, yet balancing controllability, efficiency, and texture fidelity remains challenging. Adapter-based methods often yield incoherent details, while full fine-tuning is computationally expensive and prone to overwriting pretrained priors. To address these limitations, we propose IMAGGarment+, an efficient diffusion framework for controllable and high-quality garment synthesis. It comprises two key modules designed for efficient and attribute-aware conditioning. First, we introduce an attribute-wise feature extractor (AFE) that disentangles key garment attributes, silhouette, logo, position, and color, into parallel latent streams. Each stream is optimized independently via LoRA, ensuring minimal parameter overhead while retaining expressive capacity. Second, we develop an attribute-adaptive attention (AA) module to inject attribute-specific cues into the generative process through a selective, layer-wise injection strategy. Specifically, silhouette and color features are injected into early decoder layers to guide structural and appearance formation, while logo features are propagated across all layers to ensure cross-scale consistency. Extensive experiments on fine-grained garment benchmarks demonstrate that IMAGGarment+ outperforms state-of-the-art baselines with less than 20% additional parameters, validating its effectiveness and efficiency.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Seeing through the Conﬂict: Transparent Knowledge Conﬂict Handling in Retrieval-Augmented Generation

Hua Ye
Siyuan Chen
Ziqi Zhong
Canran Xiao
Haoliang Zhang
Yuhan Wu
Fei Shen

Large language models (LLMs) equipped with retrieval—the Retrieval-Augmented Generation (RAG) paradigm—should combine their parametric knowledge with external evidence, yet in practice they often hallucinate, over-trust noisy snippets, or ignore vital context. We introduce TCR (Transparent Conflict Resolution), a plug-and-play framework that makes this decision process observable and controllable. TCR (i) disentangles semantic match and factual consistency via dual contrastive encoders, (ii) estimates self-answerability to gauge confidence in internal memory, and (iii) feeds the three scalar signals to the generator through a lightweight soft-prompt with SNR-based weighting. Across seven benchmarks TCR improves conflict detection (+5–18 F₁), raises knowledge-gap recovery by +21.4 percentage points and cuts misleading-context overrides by –29.3 percentage points, while adding only 0.3% parameters. The signals align with human judgements and expose temporal decision patterns.

PDF Details DOI

AAAI Conference 2026 Conference Paper

SGMHand: Structure-Guided Modulation for Structure-Aware Hand Inpainting

Chuancheng Shi
Shiming Guo
Ke Shui
Yixiang Chen
Fei Shen

Diffusion-based generative models have demonstrated remarkable capabilities in image synthesis, yet realistic hand generation remains a persistent challenge due to complex articulations, self-occlusion, and the lack of explicit structural guidance. To address these issues, we present SGMHand, a novel structure-guided hand inpainting framework that explicitly injects topological priors to enhance structural fidelity and spatial precision. Specifically, we present a structure-guided modulation (SGM) module that synergistically combines structure spatial attention with global feature calibration, enabling fine-grained geometric control over the generative process. Then, we devise a keypoint-aware (KA) loss that enforces topological coherence by aligning attention activations with structures, thereby bridging the gap between high-level semantics and low-level geometry. By jointly optimizing over structural constraints in both representation and learning objectives, SGMHand achieves semantically consistent and geometrically plausible hand synthesis, even under severe occlusion. Extensive experiments demonstrate the effectiveness and strong generalization ability of SGMHand across various foundation models, significantly enhancing the quality and realism of human image synthesis in diverse scenarios.

PDF Details DOI

AAAI Conference 2026 Conference Paper

StyleTailor: Towards Personalized Fashion Styling via Hierarchical Negative Feedback

Hongbo Ma
Fei Shen
Hongbin Xu
Xiaoce Wang
Gang Xu
Jinkai Zheng
Liangqiong Qu
Ming Li

The advancement of intelligent agents has revolutionized problem-solving across diverse domains, yet solutions for personalized fashion styling remain underexplored, which holds immense promise for promoting shopping experiences. In this work, we present StyleTailor, the first collaborative agent framework that seamlessly unifies personalized apparel design, shopping recommendation, virtual try-on, and systematic evaluation into a cohesive workflow. To this end, StyleTailor pioneers an iterative visual refinement paradigm driven by multi-level negative feedback, enabling adaptive and precise user alignment. Specifically, our framework features two core agents, i.e., Designer for personalized garment selection and Consultant for virtual try-on, whose outputs are progressively refined via hierarchical vision-language model feedback spanning individual items, complete outfits, and try-on efficacy. Counterexamples are aggregated into negative prompts, forming a closed-loop mechanism that enhances recommendation quality. To assess the performance, we introduce a comprehensive evaluation suite encompassing style consistency, visual quality, face similarity, and artistic appraisal. Extensive experiments demonstrate StyleTailor's superior performance in delivering personalized designs and recommendations, outperforming strong baselines without negative feedback and establishing a new benchmark for intelligent fashion systems.

PDF Details DOI

EAAI Journal 2025 Journal Article

Adaptive few-shot image augmentation for fine-grained industrial defects based on region-level modeling

Jing Wei
Qingfeng Shi
Zhengtao Zhang
Fei Shen
Huiyuan Luo

Image augmentation can improve detection performance in industrial inspection. However, traditional image augmentation fails to create novel defects. Current defect image generation often overlooks normal structures and struggles to capture fine-grained defect features with limited training samples. To address these challenges, we propose AHDEN (Adaptive Hierarchical Defect Editing Networks), which consists of AdaSDN (Adaptive Shape Design Networks) and H-DEN (Hierarchical Defect Editing Networks), to synthesize high-quality fine-grained defects with accurate binary masks by modeling the relationship between defects and normal regions. Firstly, AdaSDN generates adaptive and diversified defect masks by fusing the normal structures and random styles employing self-attention, incorporating the proposed proportion regularization and double discriminators to ensure that the mask aligns with the defect characteristics. Secondly, H-DEN designs defects based on normal textures and binary masks. The modeling of region-level relationships between defect and normal textures enables the generation of defects solely based on a single sample, and the defect strength can be controlled by fine-tuning fusion factors. Thirdly, we demonstrate the inadequacy of conventional metrics in evaluating industrial defects and propose a new metric, Texture Distance (TD), based on detailed texture features. Experimental results in various real-world industrial scenarios demonstrate that our TD metric provides more realistic evaluation results, and our AHDEN successfully generates diversified and high-quality defect samples even with only one training sample. Finally, AHDEN significantly improves the performance of various segmentation models and surpasses state-of-the-art methods with the IoU (Intersection over Union) improvement of 6. 33 % and the F1 score improvement of 5. 95 %.

Details DOI

NeurIPS Conference 2025 Conference Paper

Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization

Cong Wang
Zexuan Deng
Zhiwei Jiang
Yafeng Yin
Fei Shen
Zifeng Cheng
Shiping Ge
Shiwei Gan

Sign Language Video Generation (SLVG) seeks to generate identity-preserving sign language videos from spoken language texts. Existing methods primarily rely on the single coarse condition (e. g. , skeleton sequences) as the intermediary to bridge the translation model and the video generation model, which limits both the naturalness and expressiveness of the generated videos. To overcome these limitations, we propose SignViP, a novel SLVG framework that incorporate multiple fine-grained conditions for improved generation fidelity. Rather than directly translating error-prone high-dimensional conditions, SignViP adopts a discrete tokenization paradigm to integrate and represent fine-grained conditions (i. e. , fine-grained poses and 3D hands). SignViP contains three core components. (1) Sign Video Diffusion Model is jointly trained with a multi-condition encoder to learn continuous embeddings that encapsulate fine-grained motion and appearance. (2) Finite Scalar Quantization (FSQ) Autoencoder is further trained to compress and quantize these embeddings into discrete tokens for compact representation of the conditions. (3) Multi-Condition Token Translator is trained to translate spoken language text to discrete multi-condition tokens. During inference, Multi-Condition Token Translator first translates the spoken language text into discrete multi-condition tokens. These tokens are then decoded to continuous embeddings by FSQ Autoencoder, which are subsequently injected into Sign Video Diffusion Model to guide video generation. Experimental results show that SignViP achieves state-of-the-art performance across metrics, including video quality, temporal coherence, and semantic fidelity. The code is available at https: //github. com/umnooob/signvip/.

PDF Details

AAAI Conference 2025 Conference Paper

Boosting Consistency in Story Visualization with Rich-Contextual Conditional Diffusion Models

Fei Shen
Hu Ye
Sibo Liu
Jun Zhang
Cong Wang
Xiao Han
Yang Wei

Recent research showcases the considerable potential of conditional diffusion models for generating consistent stories. However, current methods, which primarily generate stories in a caption-dependent manner, often overlook the importance of contextual consistency and the relevance of frames during sequential generation. To address this, we propose a novel Rich-contextual Conditional Diffusion Models (RCDMs), a two-stage approach designed to enhance story generation's semantic consistency and temporal consistency. Specifically, in the first stage, the frame-prior transformer diffusion model is presented to predict the frame semantic embedding of the unknown clip by aligning the semantic correlations between the captions and frames of the known clip. The second stage establishes a robust model with rich contextual conditions, including reference images of the known clip, the predicted frame semantic embedding of the unknown clip, and text embeddings of all captions. By jointly injecting these rich contextual conditions at the image and feature levels, RCDMs can generate semantic and temporal consistency stories. Moreover, RCDMs can generate consistent stories with a single forward inference compared to autoregressive models. Our qualitative and quantitative results demonstrate that our proposed RCDMs outperform in challenging scenarios.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model

Dapeng Zhang
Fei Shen
Rui Zhao
Yinda Chen
Peng Zhi
Chenyang Li
Rui Zhou
Qingguo Zhou

Autonomous driving represents a prominent application of artificial intelligence. Recent approaches have shifted from focusing solely on common scenarios to addressing complex, long-tail situations such as subtle human behaviors, traffic accidents, and non-compliant driving patterns. Given the demonstrated capabilities of large language models (LLMs) in understanding visual and natural language inputs and following instructions, recent methods have integrated LLMs into autonomous driving systems to enhance reasoning, interpretability, and performance across diverse scenarios. However, existing methods typically rely either on real-world data, which is suitable for industrial deployment, or on simulation data tailored to rare or hard case scenarios. Few approaches effectively integrate the complementary advantages of both data sources. To address this limitation, we propose a novel VLM-guided, end-to-end adversarial transfer framework for autonomous driving that transfers long-tail handling capabilities from simulation to real-world deployment, named CoC-VLA. The framework comprises a teacher VLM model, a student VLM model, and a discriminator. Both the teacher and student VLM models utilize a shared base architecture, termed the Chain-of-Causality Visual–Language Model (CoC VLM), which integrates temporal information via an end-to-end text adapter. This architecture supports chain-of-thought reasoning to infer complex driving logic. The teacher and student VLM models are pre-trained separately on simulated and real-world datasets. The discriminator is trained adversarially to facilitate the transfer of long-tail handling capabilities from simulated to real-world environments by the student VLM model, using a novel backpropagation strategy. Experimental results show that our method effectively bridges the gap between simulation and real-world autonomous driving, indicating a promising direction for future research.

PDF Details

AAAI Conference 2025 Conference Paper

DVP-MVS: Synergize Depth-Edge and Visibility Prior for Multi-View Stereo

Zhenlong Yuan
Jinguo Luo
Fei Shen
Zhaoxin Li
Cong Liu
Tianlu Mao
Zhaoqi Wang

Patch deformation-based methods have recently exhibited substantial effectiveness in multi-view stereo, due to the incorporation of deformable and expandable perception to reconstruct textureless areas. However, such approaches typically focus on exploring correlative reliable pixels to alleviate match ambiguity during patch deformation, but ignore the deformation instability caused by mistaken edge-skipping and visibility occlusion, leading to potential estimation deviation. To remedy the above issues, we propose DVP-MVS, which innovatively synergizes depth-edge aligned and cross-view prior for robust and visibility-aware patch deformation. Specifically, to avoid unexpected edge-skipping, we first utilize Depth Anything V2 followed by the Roberts operator to initialize coarse depth and edge maps respectively, both of which are further aligned through an erosion-dilation strategy to generate fine-grained homogeneous boundaries for guiding patch deformation. In addition, we reform view selection weights as visibility maps and restore visible areas by cross-view depth reprojection, then regard them as cross-view prior to facilitate visibility-aware patch deformation. Finally, we improve propagation and refinement with multi-view geometry consistency by introducing aggregated visible hemispherical normals based on view selection and local projection depth differences based on epipolar lines, respectively. Extensive evaluations on ETH3D and Tanks & Temples benchmarks demonstrate that our method can achieve state-of-the-art performance with excellent robustness and generalization.

PDF Details DOI

ICLR Conference 2025 Conference Paper

Ensembling Diffusion Models via Adaptive Feature Aggregation

Cong Wang 0034
Kuan Tian
Yonghang Guan
Fei Shen
Zhiwei Jiang 0001
Qing Gu 0001
Jun Zhang 0018

The success of the text-guided diffusion model has inspired the development and release of numerous powerful diffusion models within the open-source community. These models are typically fine-tuned on various expert datasets, showcasing diverse denoising capabilities. Leveraging multiple high-quality models to produce stronger generation ability is valuable, but has not been extensively studied. Existing methods primarily adopt parameter merging strategies to produce a new static model. However, they overlook the fact that the divergent denoising capabilities of the models may dynamically change across different states, such as when experiencing different prompts, initial noises, denoising steps, and spatial locations. In this paper, we propose a novel ensembling method, Adaptive Feature Aggregation (AFA), which dynamically adjusts the contributions of multiple models at the feature level according to various states (i.e., prompts, initial noises, denoising steps, and spatial locations), thereby keeping the advantages of multiple diffusion models, while suppressing their disadvantages. Specifically, we design a lightweight Spatial-Aware Block-Wise (SABW) feature aggregator that adaptive aggregates the block-wise intermediate features from multiple U-Net denoisers into a unified one. The core idea lies in dynamically producing an individual attention map for each model's features by comprehensively considering various states. It is worth noting that only SABW is trainable with about 50 million parameters, while other models are frozen. Both the quantitative and qualitative experiments demonstrate the effectiveness of our proposed method.

Details

AAAI Conference 2025 Conference Paper

Exploring the Potential of Large Vision-Language Models for Unsupervised Text-Based Person Retrieval

Zongyi Li
Li Jianbo
Yuxuan Shi
Jiazhong Chen
Shijuan Huang
Linnan Tu
Fei Shen
Hefei Ling

The aim of text-based person retrieval is to identify pedestrians using natural language descriptions within a large-scale image gallery. Traditional methods rely heavily on manually annotated image-text pairs, which are resource-intensive to obtain. With the emergence of Large Vision-Language Models (LVLMs), the advanced capabilities of contemporary models in image understanding have led to the generation of highly accurate captions. Therefore, this paper explores the potential of employing Large Vision-Language Models for unsupervised text-based pedestrian image retrieval and proposes a Multi-grained Uncertainty Modeling and Alignment framework (MUMA). Initially, multiple Large Vision-Language Models are employed to generate diverse and hierarchically structured pedestrian descriptions across different styles and granularities. However, the generated captions inevitably introduce noise. To address this issue, an uncertainty-guided sample filtration module is proposed to estimate and filter out unreliable image-text pairs. Additionally, to simulate the diversity of styles and granularities in captions, a multi-grained uncertainty modeling approach is applied to model the distributions of captions, with each caption represented as a multivariate Gaussian distribution. Finally, a multi-level consistency distillation loss is employed to integrate and align the multi-grained captions, aiming to transfer knowledge across different granularities. Experimental evaluations conducted on three widely-used datasets demonstrate the significant advancements achieved by our approach.

PDF Details DOI

ICLR Conference 2025 Conference Paper

FaceShot: Bring Any Character into Life

Junyao Gao 0002
Yanan Sun 0005
Fei Shen
Xin Jiang 0010
Zhening Xing
Kai Chen 0026
Cairong Zhao

In this paper, we present ***FaceShot***, a novel training-free portrait animation framework designed to bring any character into life from any driven video without fine-tuning or retraining. We achieve this by offering precise and robust reposed landmark sequences from an appearance-guided landmark matching module and a coordinate-based landmark retargeting module. Together, these components harness the robust semantic correspondences of latent diffusion models to produce facial motion sequence across a wide range of character types. After that, we input the landmark sequences into a pre-trained landmark-driven animation model to generate animated video. With this powerful generalization capability, FaceShot can significantly extend the application of portrait animation by breaking the limitation of realistic portrait landmark detection for any stylized character and driven video. Also, FaceShot is compatible with any landmark-driven animation model, significantly improving overall performance. Extensive experiments on our newly constructed character benchmark CharacBench confirm that FaceShot consistently surpasses state-of-the-art (SOTA) approaches across any character domain. More results are available at our project website https://faceshot2024.github.io/faceshot/.

Details

AAAI Conference 2025 Conference Paper

IMAGDressing-v1: Customizable Virtual Dressing

Fei Shen
Xin Jiang
Xin He
Hu Ye
Cong Wang
Xiaoyu Du
Zechao Li
Jinhui Tang

Existing virtual try-on (VTON) methods provide only limited user control over garment attributes and generally overlook essential factors such as face, pose, and scene context. To address these limitations, we introduce the virtual dressing (VD) task, which aims to synthesize freely editable human images conditioned on fixed garments and optional user-defined inputs. We further propose a comprehensive affinity metric index (CAMI) to quantify the consistency between generated outputs and reference garments. We present IMAGDressing-v1, which leverages a garment-specific U-Net to integrate semantic features from CLIP and texture features from a VAE. To incorporate these garment features into a frozen denoising U-Net for flexible text-driven scene control, we employ a hybrid attention mechanism composed of frozen self-attention and trainable cross-attention layers. IMAGDressing-v1 seamlessly integrates with extension modules, such as ControlNet and IP-Adapter, enabling enhanced diversity and controllability. To alleviate data constraints, we introduce the Interactive Garment Pairing (IGPair) dataset, comprising over 300,000 garment–image pairs and a standardized data assembly pipeline. Extensive experiments demonstrate that IMAGDressing-v1 achieves state-of-the-art performance in controlled human image synthesis. The code and model will be available at https://github.com/muzishen/IMAGDressing.

PDF Details DOI

ICML Conference 2025 Conference Paper

Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model

Fei Shen
Cong Wang 0018
Junyao Gao 0002
Qin Guo
Jisheng Dang
Jinhui Tang 0001
Tat-Seng Chua

Recent advances in conditional diffusion models have shown promise for generating realistic TalkingFace videos, yet challenges persist in achieving consistent head movement, synchronized facial expressions, and accurate lip synchronization over extended generations. To address these, we introduce the M otion-priors C onditional D iffusion M odel ( MCDM ), which utilizes both archived and current clip motion priors to enhance motion prediction and ensure temporal consistency. The model consists of three key elements: (1) an archived-clip motion-prior that incorporates historical frames and a reference frame to preserve identity and context; (2) a present-clip motion-prior diffusion model that captures multimodal causality for accurate predictions of head movements, lip sync, and expressions; and (3) a memory-efficient temporal attention mechanism that mitigates error accumulation by dynamically storing and updating motion features. We also introduce the TalkingFace-Wild dataset, a multilingual collection of over 200 hours of footage across 10 languages. Experimental results demonstrate the effectiveness of MCDM in maintaining identity and motion continuity for long-term TalkingFace generation.

Details

AAAI Conference 2025 Conference Paper

MSP-MVS: Multi-Granularity Segmentation Prior Guided Multi-View Stereo

Zhenlong Yuan
Cong Liu
Fei Shen
Zhaoxin Li
Jinguo Luo
Tianlu Mao
Zhaoqi Wang

Recently, patch deformation-based methods have demonstrated significant strength in multi-view stereo by adaptively expanding the reception field of patches to help reconstruct textureless areas. However, such methods mainly concentrate on searching for pixels without matching ambiguity (i.e., reliable pixels) when constructing deformed patches, while neglecting the deformation instability caused by unexpected edge-skipping, resulting in potential matching distortions. Addressing this, we propose MSP-MVS, a method introducing multi-granularity segmentation prior for edge-confined patch deformation. Specifically, to avoid unexpected edge-skipping, we first aggregate and further refine multi-granularity depth edges gained from Semantic-SAM as prior to guide patch deformation within depth-continuous (i.e., homogeneous) areas. Moreover, to address attention imbalance caused by edge-confined patch deformation, we implement adaptive equidistribution and disassemble-clustering of correlative reliable pixels (i.e., anchors), thereby promoting attention-consistent patch deformation. Finally, to prevent deformed patches from falling into local-minimum matching costs caused by the fixed sampling pattern, we introduce disparity-sampling synergistic 3D optimization to help identify global-minimum matching costs. Evaluations on ETH3D and Tanks & Temples benchmarks prove our method obtains state-of-the-art performance with remarkable generalization.

PDF Details DOI

EAAI Journal 2025 Journal Article

Multi-component controllable diversified augmentation of industrial images based on feature disentanglement

Jing Wei
Qingfeng Shi
Fei Shen
Zhengtao Zhang

Industrial defect segmentation based on deep learning heavily relies on diverse training data to achieve superior performance. However, it is challenging to collect diversified defects on production lines. Therefore, to improve the industrial image diversity from multiple feature dimensions, we propose a spatially-varying augmentation method based on feature disentanglement. Firstly, the Disentanglement AutoEncoder (DisAE) is proposed to extract multi-dimensional operable features of industrial images and generate new samples with accurate masks. The encoder disentangles images into spatially-variable defect features and normal background features, modeling the spatially local mapping between images and features. The decoder employs category labels and strength factors to modulate defect features, thus disentangling defect textures and structures. Additionally, adversarial loss and random sampling are introduced to enhance the generated quality and diversity. Secondly, based on the disentangled features from DisAE, we design four types of spatially-varying feature transformations, including affine transformations, random erasing, feature combination, and style transformation, to independently modify five main components of industrial images, achieving controllable diversification of normal background, defect location, shape, content, and strength. Experimental results show that our method can independently control and diversify five components of defect samples, generating diversified new samples with realistic textures and accurate mask annotations. Finally, our method obtains optimal segmentation performance on various industrial products and demonstrates high application value.

Details DOI

NeurIPS Conference 2025 Conference Paper

SceneDecorator: Towards Scene-Oriented Story Generation with Scene Planning and Scene Consistency

Quanjian Song
Donghao Zhou
Jingyu Lin
Fei Shen
Jiaze Wang
Xiaowei Hu
Cunjian Chen
Pheng-Ann Heng

Recent text-to-image models have revolutionized image generation, but they still struggle with maintaining concept consistency across generated images. While existing works focus on character consistency, they often overlook the crucial role of scenes in storytelling, which restricts their creativity in practice. This paper introduces scene-oriented story generation, addressing two key challenges: (i) scene planning, where current methods fail to ensure scene-level narrative coherence by relying solely on text descriptions, and (ii) scene consistency, which remains largely unexplored in terms of maintaining scene consistency across multiple stories. We propose SceneDecorator, a training-free framework that employs VLM-Guided Scene Planning to ensure narrative coherence across different scenes in a ``global-to-local'' manner, and Long-Term Scene-Sharing Attention to maintain long-term scene consistency and subject diversity across generated stories. Extensive experiments demonstrate the superior performance of SceneDecorator, highlighting its potential to unleash creativity in the fields of arts, films, and games.

PDF Details

ICML Conference 2025 Conference Paper

UniMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation

Qin Guo
Ailing Zeng
Dongxu Yue
Ceyuan Yang
Yang Cao 0017
Hanzhong Guo
Fei Shen
Wei Liu 0005

Although significant advancements have been achieved in the progress of keypoint-guided Text-to-Image diffusion models, existing mainstream keypoint-guided models encounter challenges in controlling the generation of more general non-rigid objects beyond humans (e. g. , animals). Moreover, it is difficult to generate multiple overlapping humans and animals based on keypoint controls solely. These challenges arise from two main aspects: the inherent limitations of existing controllable methods and the lack of suitable datasets. First, we design a DiT-based framework, named UniMC, to explore unifying controllable multi-class image generation. UniMC integrates instance- and keypoint-level conditions into compact tokens, incorporating attributes such as class, bounding box, and keypoint coordinates. This approach overcomes the limitations of previous methods that struggled to distinguish instances and classes due to their reliance on skeleton images as conditions. Second, we propose HAIG-2. 9M, a large-scale, high-quality, and diverse dataset designed for keypoint-guided human and animal image generation. HAIG-2. 9M includes 786K images with 2. 9M instances. This dataset features extensive annotations such as keypoints, bounding boxes, and fine-grained captions for both humans and animals, along with rigorous manual inspection to ensure annotation accuracy. Extensive experiments demonstrate the high quality of HAIG-2. 9M and the effectiveness of UniMC, particularly in heavy occlusions and multi-class scenarios.

Details

ICLR Conference 2024 Conference Paper

Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models

Fei Shen
Hu Ye
Jun Zhang 0018
Cong Wang 0034
Xiao Han 0011
Yang Wei

Recent work has showcased the significant potential of diffusion models in pose-guided person image synthesis. However, owing to the inconsistency in pose between the source and target images, synthesizing an image with a distinct pose, relying exclusively on the source image and target pose information, remains a formidable challenge. This paper presents Progressive Conditional Diffusion Models (PCDMs) that incrementally bridge the gap between person images under the target and source poses through three stages. Specifically, in the first stage, we design a simple prior conditional diffusion model that predicts the global features of the target image by mining the global alignment relationship between pose coordinates and image appearance. Then, the second stage establishes a dense correspondence between the source and target images using the global features from the previous stage, and an inpainting conditional diffusion model is proposed to further align and enhance the contextual features, generating a coarse-grained person image. In the third stage, we propose a refining conditional diffusion model to utilize the coarsely generated image from the previous stage as a condition, achieving texture restoration and enhancing fine-detail consistency. The three-stage PCDMs work progressively to generate the final high-quality and high-fidelity synthesized image. Both qualitative and quantitative results demonstrate the consistency and photorealism of our proposed PCDMs under challenging scenarios. The code and model will be available at https://github.com/tencent-ailab/PCDMs.

Details

NeurIPS Conference 2024 Conference Paper

IMAGPose: A Unified Conditional Framework for Pose-Guided Person Generation

Fei Shen
Jinhui Tang

Diffusion models represent a promising avenue for image generation, having demonstrated competitive performance in pose-guided person image generation. However, existing methods are limited to generating target images from a source image and a target pose, overlooking two critical user scenarios: generating multiple target images with different poses simultaneously and generating target images from multi-view source images. To overcome these limitations, we propose IMAGPose, a unified conditional framework for pose-guided image generation, which incorporates three pivotal modules: a feature-level conditioning (FLC) module, an image-level conditioning (ILC) module, and a cross-view attention (CVA) module. Firstly, the FLC module combines the low-level texture feature from the VAE encoder with the high-level semantic feature from the image encoder, addressing the issue of missing detail information due to the absence of a dedicated person image feature extractor. Then, the ILC module achieves an alignment of images and poses to adapt to flexible and diverse user scenarios by injecting a variable number of source image conditions and introducing a masking strategy. Finally, the CVA module introduces decomposing global and local cross-attention, ensuring local fidelity and global consistency of the person image when multiple source image prompts. The three modules of IMAGPose work together to unify the task of person image generation under various user scenarios. Extensive experiment results demonstrate the consistency and photorealism of our proposed IMAGPose under challenging user scenarios. The code and model will be available at https: //github. com/muzishen/IMAGPose.

PDF Details DOI

IROS Conference 2006 Conference Paper

The Design of Digital Handwriting Forces Vector Ink and its Application in Online Signature Verification

ZhongCheng Wu
Fei Shen
Yong Yu 0003

The use of pen as an effective communication interface to computer becomes an active research area in recent years. However, the lack of complete solutions for interchange between platforms has hampered its application as no efficient device can record the entire procedure of human handwriting behavior. In this paper, we design a force Tablet (F-Tablet) for human-computer-interaction (HCI), which can acquire the kinematics and kinetics information of human handwriting, including strokes of pen-up and pen-down, pen nib trajectory and three-axis forces of pen tip directly and simultaneously. Any stylus- and pen-like device can be used to write on it. The core part of the system, named as F-Tablet, is introduced and its ink was defined according to InkML format. An improved DTW (dynamic time warping) algorithm is also put forward to verify the online signatures based on the digital handwriting forces vector ink. The iterative experiment is introduced to decide weights for writing forces in different direction and the classification threshold

Details

IROS Conference 2005 Conference Paper

The closed-loop human eye-brain-hand to computer (EBH-C) interface for hand sensory-motor coordination based on force tablet

ZhongCheng Wu
Fei Shen
Le Kang
Mingxu Wei
Yong Yu 0003
Bin Fang 0001

The mechanism of the sensory-to-motor transformation as well as motor-to-sensory transformation of human beings has attracted much attention in recent years. As no efficient device can record hand intrinsic behavior, it is difficult to get its neuro-physiological models of sensory-motor coordination. In this paper, we offer a system to acquire the kinematics and kinetics information of human hand movement through handwriting. Joined with human being, an eye-brain-hand to computer (EBH-C) interaction system is presented. In this human-in-the-loop-testing system, human beings acquire image, voice or text from computer by eyes or ears then write them down. The core part of the system, named as F-Tablet/spl trade/ is able to acquire the trajectory and three-axis forces of pen-tip directly and simultaneously. With the help of this system, we designed an experiment to evaluate the handwriting movements and forces controlling ability of different ages. Some experiment results were present. Aided with conventional analysis of electroencephalography (EEG) and magnetoencephalography (MEG), the whole procedures of information transmitting, acquired by eyes and ears, processed by brain, outputted and actuated by hand, can be recorded.

Details