TimeReplayer: Unlocking the Potential of Event Cameras for Video Interpolation

He, Weihua; You, Kaichao; Qiao, Zhendong; Xu, Jing; Zhang, Ziyang; Wang, Wenhui; Lu, Huchuan; Wang, Yaoyuan; Liao, Jianxing

doi:10.1109/cvpr52688.2022.01728

Cited by 29 publications

(16 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Time Lens++ [37] further improves the efficiency and performance via computing motion splines and multi-scale fusion separately. TimeReplayer [9] utilizes a cycle-consistency loss as supervision signal, making a model trained on low frame-rate videos also able to predict high-speed videos. All the methods above assume that the key frame is sharp, but in high-speed or low-illumination scenarios, the key frame inevitably gets blurred because of the high-speed motion within the exposure time, where these methods failed (Tab.…”

Section: Related Workmentioning

confidence: 99%

“…Previous event-based methods [9,37,38] solve eventbased frame interpolation based on (2). However, in the real-world setting, because of the finite exposure times of the two frames, the timestamps t 0 and t 1 should be replaced by time ranges, and the images I 0 and I 1 may be either sharp (small motion in the exposure time) or blurry (large motion in the exposure time).…”

Section: Problem Formulationmentioning

confidence: 99%

“…We use cascaded residual blocks to model this mapping. For each latent frame, previous methods collect the events in both time ranges and convert them to an event representation [9,38], which may incur inconsistencies in the result [37]. To mitigate this, we use a recurrent network to naturally model temporal information.…”

Section: General Architecturementioning

confidence: 99%

“…The events are recorded at high temporal resolution (in the order of µs) and high dynamic range (over 140 dB) within and between frames, providing valid compressed motion information. Previous works [9,37,38] show the potential of event cameras in VFI, comparing favorably to frame-only methods, especially in high-speed non-linear motion scenarios, by using spatially aligned events and RGB frames. These event-based VFI methods make the crucial assumption that the input images are sharp.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Event-Based Frame Interpolation with Ad-hoc Deblurring

Sun¹,

Sakaridis²,

Liang³

et al. 2023

Preprint

View full text Add to dashboard Cite

The performance of video frame interpolation is inherently correlated with the ability to handle motion in the input scene. Even though previous works recognize the utility of asynchronous event information for this task, they ignore the fact that motion may or may not result in blur in the input video to be interpolated, depending on the length of the exposure time of the frames and the speed of the motion, and assume either that the input video is sharp, restricting themselves to frame interpolation, or that it is blurry, including an explicit, separate deblurring stage before interpolation in their pipeline. We instead propose a general method for event-based frame interpolation that performs deblurring ad-hoc and thus works both on sharp and blurry input videos. Our model consists in a bidirectional recurrent network that naturally incorporates the temporal dimension of interpolation and fuses information from the input frames and the events adaptively based on their temporal proximity. In addition, we introduce a novel real-world high-resolution dataset with events and color videos named HighREV, which provides a challenging evaluation setting for the examined task. Extensive experiments on the standard GoPro benchmark and on our dataset show that our network consistently outperforms previous state-of-the-art methods on frame interpolation, single image deblurring and the joint task of interpolation and deblurring. Our code and dataset will be made publicly available.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Problem Formulationmentioning

confidence: 99%

Section: General Architecturementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Event-Based Frame Interpolation with Ad-hoc Deblurring

Sun¹,

Sakaridis²,

Liang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Besides the flow-/kernel-based classes, other VFI paradigms exist, for example based on pixel hallucination [18,50], phase information [51,52], event cameras [53][54][55], unsupervised learning [56,57], and meta-learning [58]. More recently, the joint problem of deblurring and interpolation has also been addressed in [59,60].…”

Section: A Video Frame Interpolationmentioning

confidence: 99%