Event-based video interpolation via complementary motion information

Video frame interpolation, the task of synthesizing intermediate frames to increase temporal resolution, often struggles with complex scenarios when constrained by the assumption of linear motion The advent of event cameras has led to significant progress in addressing this issue. Event cameras, with microsecond-level temporal resolution, bridge the gap between frames by providing accurate motion cues. However, current event-based video frame interpolation methods often overlook that event data primarily offers high-confidence features at scene edges during multi-modal feature fusion, which may limit the contribution of event signals to optical flow estimation. To address this, we propose a novel end-to-end learning framework that explicitly leverages the complementary characteristics of event signals and frames. Our method synergistically fuses dense contextual information from frames with sparse but precise edge motion from events via a proposed Edge Guided Attention (EGA) module. The EGA employs a coarse-to-fine strategy, where event-based optical flow directly refines the frame-based motion estimation at each level of a pyramidal architecture. Additionally, we introduce an event-based visibility map, co-learned within our event-processing network, to adaptively mitigate occlusions during the warping process. Extensive experiments conducted on a diverse suite of six benchmarks, including four synthetic and two real-world datasets validate the effectiveness of this novel approach. A dedicated discussion of the method’s trade-offs and potential limitations is presented in the Limitations section.

Authors

• Yuhan Liu University of Toronto
• LingHui Fu
• Hao Chen The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, CASIA & School of Artificial Intelligence, UCAS
• Zhen Yang
• Youfu Li City University of Hong Kong
• Yongjian Deng College of Computer Science, Beijing University of Technology

Keywords

• Event camera
• Event-based video frame interpolation
• Neuromorphic vision