Image Conductor: Precision Control for Interactive Video Synthesis

Filmmaking and animation production often require sophisticated techniques for coordinating camera transitions and object movements, typically involving labor-intensive real-world capturing. Despite advancements in generative AI for video creation, achieving precise control over motion for interactive video asset generation remains challenging. To this end, we propose Image Conductor, a method for precise control of camera transitions and object movements to generate video assets from a single image. An well-cultivated training strategy is proposed to separate distinct camera and object motion by camera LoRA weights and object LoRA weights. To further eliminate motion ambiguity from ill-posed trajectories, we introduce a camera-free guidance technique during inference process, enhancing object movements while eliminating camera transitions. Additionally, we develop a trajectory-oriented video motion data curation pipeline for training. Quantitative and qualitative experiments demonstrate our method's precision and fine-grained control in generating motion-controllable videos from images, advancing the practical application of interactive video synthesis.

Authors

• Yaowei Li Pengcheng Laboratory Peking University
• Xintao Wang Kuaishou Technology
• Zhaoyang Zhang ARC Lab, Tencent PCG
• Zhouxia Wang Nanyang Technological University
• Ziyang Yuan Tsinghua University
• Liangbin Xie University of Macau Shenzhen Institute of Advanced Technology
• Ying Shan Tencent AI Lab ARC Lab, Tencent PCG
• Yuexian Zou School of Electronic and Computer Engineering, Peking University Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Peking University Shenzhen Graduate School

Keywords

No keywords are indexed for this paper.