Depth-Aware Video Frame Interpolation

Bao, Wenbo; Lai, Wei-Sheng; Ma, Chao; Zhang, Xiaoyun; Gao, Zhiyong; Yang, Ming–Hsuan

doi:10.1109/cvpr.2019.00382

Cited by 507 publications

(609 citation statements)

References 39 publications

Supporting

Mentioning

573

Contrasting

Order By: Relevance

“…Liu et al [6] propose that the synthesized interpolated frames are more reliable if they are cyclically used to reconstruct the input frames. Wenbo et al [1] introduce a depth-aware interpolation approach to better handle occlusions in the video frames. index of the target frame l index of the keyframe before the target frame r index of the keyframe after the target frame In frame n of the main videô In frame n of the auxiliary video Fn flow from the target frame It to In in the main videô Fn flow from the target frameÎt toÎn in the auxiliary video V l visibility map of the warped left keyframe.…”

Section: Video Frame Interpolationmentioning

confidence: 99%

Deep Slow Motion Video Reconstruction With Hybrid Imaging System

Paliwal

Kalantari

2020

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Slow motion videos are becoming increasingly popular, but capturing high-resolution videos at extremely high frame rates requires professional high-speed cameras. To mitigate this problem, current techniques increase the frame rate of standard videos through frame interpolation by assuming linear motion between the existing frames. While this assumption holds true for simple cases with small motion, in challenging cases the motion is usually complex and this assumption is no longer valid. Therefore, they typically produce results with unnatural motion in these challenging cases. In this paper, we address this problem using two video streams as the input; an auxiliary video with high frame rate and low spatial resolution, providing temporal information, in addition to the standard main video with low frame rate and high spatial resolution. We propose a two-stage deep learning system consisting of alignment and appearance estimation that reconstructs high resolution slow motion video from the hybrid video input. For alignment, we propose to use a set of pre-trained and trainable convolutional neural networks (CNNs) to compute the flows between the missing frame and the two existing frames of the main video by utilizing the content of the auxiliary video frames. We then warp the existing frames using the flows to produce a set of aligned frames. For appearance estimation, we propose to combine the aligned and auxiliary frames using a context and occlusion aware CNN. We train our model on a set of synthetically generated hybrid videos and show high-quality results on a wide range of test scenes. We further demonstrate the practicality of our approach by showing the performance of our system on two real dual camera setups with small baseline.

show abstract

Section: Video Frame Interpolationmentioning

confidence: 99%

Deep Slow Motion Video Reconstruction With Hybrid Imaging System

Paliwal

Kalantari

2020

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Recently, deep neural networks have shown excellent performance in optical flow estimation [6], [7], [25]. Given two consecutive color images C t−1 and C t+1 , video interpolation [1], [8], [14] aims to generate the intermediate color image C t using a bidirectional optical flow:…”

Section: A Intermediate Depth Map Interpolationmentioning

confidence: 99%

“…Interpolation techniques have been widely used in lots of computer vision and robotics tasks, which can be classified into two categories, i.e., temporal interpolation [1], [8], [14] and spatial interpolation [10], [12], [26]. In video processing, video interpolation aims to temporally generate an intermediate frame using two consecutive frames.…”

Section: Introductionmentioning

confidence: 99%

“…This method achieves real-time temporal interpolation for high resolution videos. Bao et al [1] perceived both the depth and flow information to address the strong occlusion problem during new frame synthesis. To interpolate more in-between frames, Jiang et al [8] proposed a variable-length multi-frame interpolation arXiv:1909.07137v1 [cs.CV] 16 Sep 2019 method to generate a frame at any time step between two given frames.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

PLIN: A Network for Pseudo-LiDAR Point Cloud Interpolation

Liu

Liao

Lin

et al. 2020

Sensors

View full text Add to dashboard Cite

LiDAR sensors can provide dependable 3D spatial information at a low frequency (around 10Hz) and have been widely applied in the field of autonomous driving and UAV. However, the camera with a higher frequency (around 20Hz) has to be decreased so as to match with LiDAR in a multisensor system. In this paper, we propose a novel Pseudo-LiDAR interpolation network (PLIN) to increase the frequency of Li-DAR sensors. PLIN can generate temporally and spatially highquality point cloud sequences to match the high frequency of cameras. To achieve this goal, we design a coarse interpolation stage guided by consecutive sparse depth maps and motion relationship. We also propose a refined interpolation stage guided by the realistic scene. Using this coarse-to-fine cascade structure, our method can progressively perceive multi-modal information and generate accurate intermediate point clouds.To the best of our knowledge, this is the first deep framework for Pseudo-LiDAR point cloud interpolation, which shows appealing applications in navigation systems equipped with LiDAR and cameras. Experimental results demonstrate that PLIN achieves promising performance on the KITTI dataset, significantly outperforming the traditional interpolation method and the state-of-the-art video interpolation technique.

show abstract

“…Appendix A contains the authors' teams and affiliations. AIM 2019 webpage: http://www.vision.ee.ethz.ch/aim19/ in video frame interpolation [26,25,34,35,33,13,52,20,27,37,2]. However, there has been no standard dataset for video frame interpolation, and most of the existing methods are trained from different datasets.…”

Section: Introductionmentioning

confidence: 99%

AIM 2019 Challenge on Video Temporal Super-Resolution: Methods and Results

Nah

Kim

Han

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

Self Cite

View full text Add to dashboard Cite

Videos contain various types and strengths of motions that may look unnaturally discontinuous in time when the recorded frame rate is low. This paper reviews the first AIM challenge on video temporal super-resolution (frame interpolation) with a focus on the proposed solutions and results. From low-frame-rate (15 fps) video sequences, the challenge participants are asked to submit higher-framerate (60 fps) video sequences by estimating temporally intermediate frames. We employ the REDS VTSR dataset derived from diverse videos captured in a hand-held camera for training and evaluation purposes. The competition had 62 registered participants, and a total of 8 teams competed in the final testing phase. The challenge winning methods achieve the state-of-the-art in video temporal superresolution. Resolution Members: Seungjun Nah 1

show abstract

Depth-Aware Video Frame Interpolation

Cited by 507 publications

References 39 publications

Deep Slow Motion Video Reconstruction With Hybrid Imaging System

Deep Slow Motion Video Reconstruction With Hybrid Imaging System

PLIN: A Network for Pseudo-LiDAR Point Cloud Interpolation

AIM 2019 Challenge on Video Temporal Super-Resolution: Methods and Results

Contact Info

Product

Resources

About