Author name cluster

Tao Gu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

4 papers

1 author row

AAAI Conference 2026 Conference Paper

InstructDubber: Instruction-based Alignment for Zero-shot Movie Dubbing

Zhedong Zhang
Liang Li
Gaoxiang Cong
Chunshan Liu
Yuhan Gao
Xiaowan Wang
Tao Gu
Yuankai Qi

Movie dubbing seeks to synthesize speech from a given script using a specific voice, while ensuring accurate lip synchronization and emotion-prosody alignment with the character’s visual performance. However, existing alignment approaches based on visual features face two key limitations: (1) they rely on complex, handcrafted visual preprocessing pipelines, including facial landmark detection and feature extraction; and (2) they generalize poorly to unseen visual domains, often resulting in degraded alignment and dubbing quality. To address these issues, we propose InstructDubber, a novel instruction-based alignment dubbing method for both robust in-domain and zero-shot movie dubbing. Specifically, we first feed the video, script, and corresponding prompts into a multimodal large language model to generate natural language dubbing instructions regarding the speaking rate and emotion state depicted in the video, which is robust to visual domain variations. Second, we design an instructed duration distilling module to mine discriminative duration cues from speaking rate instructions to predict lip-aligned phoneme-level pronunciation duration. Third, for emotion-prosody alignment, we devise an instructed emotion calibrating module, which fine-tunes an LLM-based instruction analyzer using ground truth dubbing emotion as supervision and predicts prosody based on the calibrated emotion analysis. Finally, the predicted duration and prosody, together with the script, are fed into the audio decoder to generate video-aligned dubbing. Extensive experiments on three major benchmarks demonstrate that InstructDubber outperforms state‑of‑the‑art approaches across both in‑domain and zero‑shot scenarios.

PDF Details DOI

AAAI Conference 2025 Conference Paper

OOTDiffusion: Outfitting Fusion Based Latent Diffusion for Controllable Virtual Try-On

Yuhao Xu
Tao Gu
Weifeng Chen
Arlene Chen

We present OOTDiffusion, a novel network architecture for realistic and controllable image-based virtual try-on (VTON). We leverage the power of pretrained latent diffusion models, designing an outfitting UNet to learn the detailed garment features. Without a redundant warping process, the garment features are precisely aligned with the target human body via the proposed outfitting fusion in the self-attention layers of the denoising UNet. In order to further enhance the controllability, we introduce outfitting dropout to the training process, which enables us to adjust the strength of the garment features through classifier-free guidance. Our comprehensive experiments on the VITON-HD and Dress Code datasets demonstrate that OOTDiffusion efficiently generates high-quality try-on results for arbitrary human and garment images, which outperforms other VTON methods in both realism and controllability, indicating a breakthrough in virtual try-on.

PDF Details DOI

TIST Journal 2020 Journal Article

DeepKey

Xiang Zhang
Lina Yao
Chaoran Huang
Tao Gu
Zheng Yang
Yunhao Liu

Biometric authentication involves various technologies to identify individuals by exploiting their unique, measurable physiological and behavioral characteristics. However, traditional biometric authentication systems (e.g., face recognition, iris, retina, voice, and fingerprint) are at increasing risks of being tricked by biometric tools such as anti-surveillance masks, contact lenses, vocoder, or fingerprint films. In this article, we design a multimodal biometric authentication system named DeepKey, which uses both Electroencephalography (EEG) and gait signals to better protect against such risk. DeepKey consists of two key components: an Invalid ID Filter Model to block unauthorized subjects, and an identification model based on attention-based Recurrent Neural Network (RNN) to identify a subject’s EEG IDs and gait IDs in parallel. The subject can only be granted access while all the components produce consistent affirmations to match the user’s proclaimed identity. We implement DeepKey with a live deployment in our university and conduct extensive empirical experiments to study its technical feasibility in practice. DeepKey achieves the False Acceptance Rate (FAR) and the False Rejection Rate (FRR) of 0 and 1.0%, respectively. The preliminary results demonstrate that DeepKey is feasible, shows consistent superior performance compared to a set of methods, and has the potential to be applied to the authentication deployment in real-world settings.

Details DOI

TIST Journal 2016 Journal Article

Recognizing Parkinsonian Gait Pattern by Exploiting Fine-Grained Movement Function Features

Tianben Wang
Zhu Wang
Daqing Zhang
Tao Gu
Hongbo Ni
Jiangbo Jia
Xingshe Zhou
Jing Lv

Parkinson's disease (PD) is one of the typical movement disorder diseases among elderly people, which has a serious impact on their daily lives. In this article, we propose a novel computation framework to recognize gait patterns in patients with PD. The key idea of our approach is to distinguish gait patterns in PD patients from healthy individuals by accurately extracting gait features that capture all three aspects of movement functions, that is, stability, symmetry, and harmony. The proposed framework contains three steps: gait phase discrimination, feature extraction and selection, and pattern classification. In the first step, we put forward a sliding window--based method to discriminate four gait phases from plantar pressure data. Based on the gait phases, we extract and select gait features that characterize stability, symmetry, and harmony of movement functions. Finally, we recognize PD gait patterns by applying a hybrid classification model. We evaluate the framework using an open dataset that contains real plantar pressure data of 93 PD patients and 72 healthy individuals. Experimental results demonstrate that our framework significantly outperforms the four baseline approaches.

Details DOI