PTUS: Photo-Realistic Talking Upper-Body Synthesis via 3D-Aware Motion Decomposition Warping

Talking upper-body synthesis is a promising task due to its versatile potential for video creation and consists of animating the body and face from a source image with the motion from a given driving video. However, prior synthesis approaches fall short in addressing this task and have been either limited to animating heads of a target person only, or have animated the upper body but neglected the synthesis of precise facial details. To tackle this task, we propose a Photo-realistic Talking Upper-body Synthesis method via 3D-aware motion decomposition warping, named PTUS, to both precisely synthesize the upper body as well as recover the details of the face such as blinking and lip synchronization. In particular, the motion decomposition mechanism consists of a face-body motion decomposition, which decouples the 3D motion estimation of the face and body, and a local-global motion decomposition, which decomposes the 3D face motion into global and local motions resulting in the transfer of facial expression. The 3D-aware warping module transfers the large-scale and subtle 3D motions to the extracted 3D depth-aware features in a coarse-tofine manner. Moreover, we present a new dataset, Talking-UB, which includes upper-body images with high-resolution faces, addressing the limitations of prior datasets that either consist of only facial images or upper-body images with blurry faces. Experimental results demonstrate that our proposed method can synthesize high-quality videos that preserve facial details, and achieves superior results compared to state-of-the-art cross-person motion transfer approaches. Code and collected dataset are released in https://github.com/cooluoluo/PTUS.

Authors

• Luoyang Lin Shenzhen Campus of Sun Yat-sen University
• Zutao Jiang Mohamed bin Zayed University of Artificial Intelligence
• Xiaodan Liang Shenzhen Campus of Sun Yat-sen University
• Liqian Ma ZMO AI
• Michael C. Kampffmeyer Department of Physics and Technology, UiT The Arctic University of Norway Norwegian Computing Center, Oslo, Norway
• Xiaochun Cao SUN YAT-SEN UNIVERSITY

Keywords

• CV: Computational Photography, Image & Video Synthesis
• CV: Motion & Tracking
• CV: Video Understanding & Activity Analysis