Author name cluster

Shiwei Zhang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

9 papers

1 author row

AAAI Conference 2025 Conference Paper

FreeMask: Rethinking the Importance of Attention Masks for Zero-Shot Video Editing

Lingling Cai
Kang Zhao
Hangjie Yuan
Yingya Zhang
Shiwei Zhang
Kejie Huang

Text-to-video diffusion models have made remarkable advancements. Driven by their ability to generate temporally coherent videos, research on zero-shot video editing using these fundamental models has expanded rapidly. To enhance editing quality, structural controls are frequently employed in video editing. Among these techniques, cross-attention mask control stands out for its effectiveness and efficiency. However, when cross-attention masks are naively applied to video editing, they can introduce artifacts such as blurring and flickering. Our experiments uncover a critical factor overlooked in previous video editing research: cross-attention masks are not consistently clear but vary with model structure and denoising timestep. To address this issue, we propose the metric Mask Matching Cost (MMC) that quantifies this variability and propose FreeMask, a method for selecting optimal masks tailored to specific video editing tasks. Using MMC-selected masks, we further improve the masked fusion mechanism within comprehensive attention features, e.g., temp, cross, and self-attention modules. Our approach can be seamlessly integrated into existing zero-shot video editing frameworks with better performance, requiring no control assistance or parameter fine-tuning but enabling adaptive decoupling of unedited semantic layouts with mask precision control. Extensive experiments demonstrate that FreeMask achieves superior semantic fidelity, temporal consistency, and editing quality compared to state-of-the-art methods.

PDF Details DOI

JBHI Journal 2025 Journal Article

HPA-UNet: A Hybrid Post-Processing Attention U-Net for Tongue Segmentation

Leiyue Yao
Yuchen Xu
Shiwei Zhang
Jianying Xiong
Achyut Shankar
Mustufa Haider Abidi
Michele Nappi

Tongue diagnosis is the kernel method of Traditional Chinese Medicine (TCM), and it has been proved that the condition of the tongue can serve as an indicator of a person's health status. To automatically recognize a person's latent diseases by computer vision technology, getting the tongue segmentation from a picture with high precision has significant importance. However, the precision of tongue segmentation images in most prior methods is not satisfactory, which will inevitably result in misjudging. In this paper, an effective method is proposed for highly precise tongue segmentation, which is combined with an improved U-shaped neural network and an edge refinement post-processing method. The contributions are three-fold. First, a carefully designed data augmentation strategy is imported to prevent the network from over-fitting. Second, an updated U-shaped neural network is designed to segment tongue images with high precision. Third, a post-processing method is imported to refine the edge of the tongue segmentation further. The proposed method achieves competitive performance in almost all experiments on two datasets. Furthermore, the proposed post-processing method can effectively improve all classic neural networks in tongue segmentation, which strongly proves the flexibility and generalization of the proposed method.

Details DOI

NeurIPS Conference 2025 Conference Paper

TTS-VAR: A Test-Time Scaling Framework for Visual Auto-Regressive Generation

Zhekai Chen
Ruihang Chu
Yukang Chen
Shiwei Zhang
Yujie Wei
Yingya Zhang
Xihui Liu

Scaling visual generation models is essential for real-world content creation, yet requires substantial training and computational expenses. Alternatively, test-time scaling has garnered growing attention due to resource efficiency and promising performance. In this work, we present the first general test-time scaling framework for visual auto-regressive (VAR) models, TTS-VAR, modeling the generation process as a path searching problem. Inspired by VAR's hierarchical coarse-to-fine multi-scale generation, our framework integrates two key components: (i) At coarse scales, we observe that generated tokens are hard for evaluation, possibly leading to erroneous acceptance of inferior samples or rejection of superior samples. Noticing that the coarse scales contain sufficient structural information, we propose clustering-based diversity search. It preserves structural variety through semantic feature clustering, enabling later selection on samples with higher potential. (ii) In fine scales, resampling-based potential selection prioritizes promising candidates using potential scores, which are defined as reward functions incorporating multi-scale generation history. To dynamically balance computational efficiency with exploration capacity, we further introduce an adaptive descending batch size schedule throughout the causal generation process. Experiments on the powerful VAR model Infinity2B show a notable 8. 7% GenEval score improvement (0. 69→0. 75). Key insights reveal that early-stage structural features effectively influence final quality, and resampling efficacy varies across generation scales.

PDF Details

NeurIPS Conference 2025 Conference Paper

Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance

Ruihang Chu
Yefei He
Zhekai Chen
Shiwei Zhang
Xiaogang Xu
Bin Xia
Dingdong WANG
Hongwei Yi

We present Wan-Move, a simple and scalable framework that brings motion control to video generative models. Existing motion-controllable methods typically suffer from coarse control granularity and limited scalability, leaving their outputs insufficient for practical use. We narrow this gap by achieving precise and high-quality motion control. Our core idea is to directly make the original condition features motion-aware for guiding video synthesis. To this end, we first represent object motions with dense point trajectories, allowing fine-grained control over the scene. We then project these trajectories into latent space and propagate the first frame's features along each trajectory, producing an aligned spatiotemporal feature map that tells how each scene element should move. This feature map serves as the updated latent condition, which is naturally integrated into the off-the-shelf image-to-video model, e. g. , Wan-I2V-14B, as motion guidance without any architecture change. It removes the need for auxiliary motion encoders and makes fine-tuning base models easily scalable. Through scaled training, Wan-Move generates 5-second, 480p videos whose motion controllability rivals Kling 1. 5 Pro's commercial Motion Brush, as indicated by user studies. To support comprehensive evaluation, we further design MoveBench, a rigorously curated benchmark featuring diverse content categories and hybrid-verified annotations. It is distinguished by larger data volume, longer video durations, and high-quality motion annotations. Extensive experiments on MoveBench and the public dataset consistently show Wan-Move's superior motion quality. Code, models, and benchmark data are made available.

PDF Details

NeurIPS Conference 2024 Conference Paper

EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Models

Rui Zhao
Hangjie Yuan
Yujie Wei
Shiwei Zhang
Yuchao Gu
Lingmin Ran
Xiang Wang
Zhangjie Wu

Recent advancements in generation models have showcased remarkable capabilities in generating fantastic content. However, most of them are trained on proprietary high-quality data, and some models withhold their parameters and only provide accessible application programming interfaces (APIs), limiting their benefits for downstream tasks. To explore the feasibility of training a text-to-image generation model comparable to advanced models using publicly available resources, we introduce EvolveDirector. This framework interacts with advanced models through their public APIs to obtain text-image data pairs to train a base model. Our experiments with extensive data indicate that the model trained on generated data of the advanced model can approximate its generation capability. However, it requires large-scale samples of 10 million or more. This incurs significant expenses in time, computational resources, and especially the costs associated with calling fee-based APIs. To address this problem, we leverage pre-trained large vision-language models (VLMs) to guide the evolution of the base model. VLM continuously evaluates the base model during training and dynamically updates and refines the training dataset by the discrimination, expansion, deletion, and mutation operations. Experimental results show that this paradigm significantly reduces the required data volume. Furthermore, when approaching multiple advanced models, EvolveDirector can select the best samples generated by them to learn powerful and balanced abilities. The final trained model Edgen is demonstrated to outperform these advanced models. The code and model weights are available at https: //github. com/showlab/EvolveDirector.

PDF Details DOI

YNICL Journal 2024 Journal Article

The association of motor reserve and clinical progression in Parkinson’s disease

Xueqin Bai
Shiwei Zhang
Qiuyue Li
Tao Guo
Xiaojun Guan
Andan Qian
Shuangli Chen
Ronghui Zhou

OBJECTIVE: To explore the association of motor reserve (MR) and clinical progression in Parkinson's disease. METHODS: This longitudinal study using data from the Parkinson's progression markers initiative. Patients with de novo PD who underwent dopamine transporter scans at baseline and finished at least five years clinical follow-up assessments (including motor, cognitive, and non-motor symptoms) were included. The individual MR of PD patients were estimated based on initial motor deficits and striatal dopamine depletion using a residual model. Linear mixed-effects models (LME) were performed to examine the associations of baseline MR and clinical progression. RESULTS: A total of 303 de novo PD patients were included and the mean follow-up time was 8.95 years. Results of LME models revealed that the baseline MR was associated with motor, cognitive, and non-motor symptoms in PD patients. There was a significant interaction between MR and disease duration for longitudinal changes in motor (p < 0.001), cognitive (p = 0.028) and depression symptoms (p = 0.014). PD patients with lower MR had a more rapid progression to postural instability and cognitive impairment compared with those with higher MR (p = 0.002 and p = 0.001, respectively). CONCLUSIONS: The baseline MR of PD patients were associated with motor and non-motor symptoms and can predicted disease prognosis, suggesting that the initial MR in PD would be associated with the individual's capacity to cope with neurodegenerative process as well as comprehensive prognosis.

Details DOI

NeurIPS Conference 2023 Conference Paper

FaceComposer: A Unified Model for Versatile Facial Content Creation

Jiayu Wang
Kang Zhao
Yifeng Ma
Shiwei Zhang
Yingya Zhang
Yujun Shen
Deli Zhao
Jingren Zhou

This work presents FaceComposer, a unified generative model that accomplishes a variety of facial content creation tasks, including text-conditioned face synthesis, text-guided face editing, face animation etc. Based on the latent diffusion framework, FaceComposer follows the paradigm of compositional generation and employs diverse face-specific conditions, e. g. , Identity Feature and Projected Normalized Coordinate Code, to release the model creativity at all possible. To support text control and animation, we clean up some existing face image datasets and collect around 500 hours of talking-face videos, forming a high-quality large-scale multi-modal face database. A temporal self-attention module is incorporated into the U-Net structure, which allows learning the denoising process on the mixture of images and videos. Extensive experiments suggest that our approach not only achieves comparable or even better performance than state-of-the-arts on each single task, but also facilitates some combined tasks with one-time forward, demonstrating its potential in serving as a foundation generative model in face domain. We further develop an interface such that users can enjoy our one-step service to create, edit, and animate their own characters. Code, dataset, model, and interface will be made publicly available.

PDF Details

NeurIPS Conference 2023 Conference Paper

VideoComposer: Compositional Video Synthesis with Motion Controllability

Xiang Wang
Hangjie Yuan
Shiwei Zhang
Dayou Chen
Jiuniu Wang
Yingya Zhang
Yujun Shen
Deli Zhao

The pursuit of controllability as a higher standard of visual content creation has yielded remarkable progress in customizable image synthesis. However, achieving controllable video synthesis remains challenging due to the large variation of temporal dynamics and the requirement of cross-frame temporal consistency. Based on the paradigm of compositional generation, this work presents VideoComposer that allows users to flexibly compose a video with textual conditions, spatial conditions, and more importantly temporal conditions. Specifically, considering the characteristic of video data, we introduce the motion vector from compressed videos as an explicit control signal to provide guidance regarding temporal dynamics. In addition, we develop a Spatio-Temporal Condition encoder (STC-encoder) that serves as a unified interface to effectively incorporate the spatial and temporal relations of sequential inputs, with which the model could make better use of temporal conditions and hence achieve higher inter-frame consistency. Extensive experimental results suggest that VideoComposer is able to control the spatial and temporal patterns simultaneously within a synthesized video in various forms, such as text description, sketch sequence, reference video, or even simply hand-crafted motions. The code and models are publicly available athttps: //videocomposer. github. io.

PDF Details

NeurIPS Conference 2022 Conference Paper

Learning a Condensed Frame for Memory-Efficient Video Class-Incremental Learning

Yixuan Pei
Zhiwu Qing
Jun CEN
Xiang Wang
Shiwei Zhang
Yaxiong Wang
Mingqian Tang
Nong Sang

Recent incremental learning for action recognition usually stores representative videos to mitigate catastrophic forgetting. However, only a few bulky videos can be stored due to the limited memory. To address this problem, we propose FrameMaker, a memory-efficient video class-incremental learning approach that learns to produce a condensed frame for each selected video. Specifically, FrameMaker is mainly composed of two crucial components: Frame Condensing and Instance-Specific Prompt. The former is to reduce the memory cost by preserving only one condensed frame instead of the whole video, while the latter aims to compensate the lost spatio-temporal details in the Frame Condensing stage. By this means, FrameMaker enables a remarkable reduction in memory but keep enough information that can be applied to following incremental tasks. Experimental results on multiple challenging benchmarks, i. e. , HMDB51, UCF101 and Something-Something V2, demonstrate that FrameMaker can achieve better performance to recent advanced methods while consuming only 20% memory. Additionally, under the same memory consumption conditions, FrameMaker significantly outperforms existing state-of-the-arts by a convincing margin.

PDF Details