Scene-Adaptive Video Frame Interpolation via Meta-Learning

Choi, Myungsub; Choi, Janghoon; Baik, Seung‐Chul; Kim, Tae Hyun; Lee, Kyoung Mu

doi:10.1109/cvpr42600.2020.00946

Cited by 46 publications

(17 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, [29], [42], [45], [52] addressed the motion blur and motion aliasing of complex scenes in temporal frame interpolation. Choi et al [5] proposed a novel scene adaptation framework to further improve the frame interpolation models via meta-learning. Instead of synthesizing the intermediate LR frames as current VFI methods do, our one-stage framework interpolates features from two neighboring LR frames to directly synthesize LR feature maps for missing frames without explicit supervision.…”

Section: Video Frame Interpolationmentioning

confidence: 99%

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

Xiang

Tian

Zhang

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

185

210

View full text Add to dashboard Cite

In this paper, we address the space-time video super-resolution, which aims at generating a high-resolution (HR) slow-motion video from a low-resolution (LR) and low frame rate (LFR) video sequence. A naive method is to decompose it into two sub-tasks: video frame interpolation (VFI) and video super-resolution (VSR). Nevertheless, temporal interpolation and spatial upscaling are intra-related in this problem. Two-stage approaches cannot fully make use of this natural property. Besides, state-of-the-art VFI or VSR deep networks usually have a large frame reconstruction module in order to obtain high-quality photo-realistic video frames, which makes the two-stage approaches have large models and thus be relatively time-consuming. To overcome the issues, we present a one-stage space-time video super-resolution framework, which can directly reconstruct an HR slow-motion video sequence from an input LR and LFR video. Instead of reconstructing missing LR intermediate frames as VFI models do, we temporally interpolate LR frame features of the missing LR frames capturing local temporal contexts by a feature temporal interpolation module. Extensive experiments on widely used benchmarks demonstrate that the proposed framework not only achieves better qualitative and quantitative performance on both clean and noisy LR frames but also is several times faster than recent state-of-the-art two-stage networks. The source code is released in https:// github.com/ Mukosame/ Zooming-Slow-Mo-CVPR-2020.

show abstract

Section: Video Frame Interpolationmentioning

confidence: 99%

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

Xiang

Tian

Zhang

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

185

210

View full text Add to dashboard Cite

show abstract

“…Some recent researches enhance the performance of interpolation by making use of auxiliary information (e.g., more reference frames [39,47] and high frame rate video with low spatial resolution [65]). Besides, a good initialization and fine-tuning of pre-trained sub-networks (such as PWC-Net [17], RBPN [62], Megadepth [63]) can greatly help to produce high quality interpolation results.…”

Section: Discussion and Limitationsmentioning

confidence: 99%

“…Xiang et al [46] proposed a one-stage space-time video super-resolution for jointly frame interpolation and superresolution. Choi et al [47] proposed to improve the performance of an interpolation algorithm by incorporating meta-learning.…”

Section: Single Frame Interpolationmentioning

confidence: 99%

Multiple Video Frame Interpolation via Enhanced Deformable Separable Convolution

Cheng

Chen

2020

Preprint

View full text Add to dashboard Cite

Generating non-existing frames from a consecutive video sequence has been an interesting and challenging problem in the video processing field. Recent kernel-based interpolation methods predict pixels with a single convolution process that convolves source frames with spatially adaptive local kernels. However, when scene motion is larger than the pre-defined kernel size, these methods are prone to yield less plausible results and they cannot directly generate a frame at an arbitrary temporal position because the learned kernels are tied to the midpoint in time between the input frames. In this paper, we try to solve these problems and propose a novel approach that we refer to as enhanced deformable separable convolution (EDSC) to estimate not only adaptive kernels, but also offsets, masks and biases to make the network obtain information from non-local neighborhood. During the learning process, different intermediate time step can be involved as a control variable by means of the coord-conv trick, allowing the estimated components to vary with different input temporal information. This makes our method capable to produce multiple in-between frames. Furthermore, we investigate the relationships between our method and other typical kernel-and flow-based methods. Experimental results show that our method performs favorably against the state-of-the-art methods across a broad range of datasets. Code will be publicly available on URL: https://github.com/Xianhang/EDSC-pytorch.

show abstract

“…Thus, we adopt a lightweight optical flow network [31] on LR frames and a flow refine network [26] to get the middle flow on HR frames, and we try a new supervised flow loss to achieve better perception. Recently, meta-learning is also introduced into frame interpolation [7]; CAIN [8] adapts channel attention into VFI; and EDSC [6] uses ConvLSTM to learn motion offset for implicit motion compensation.…”

Section: Video Frame Interpolation (Vfi)mentioning

confidence: 99%

How Video Super-Resolution and Frame Interpolation Mutually Benefit

Zhou

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Video super-resolution (VSR) and video frame interpolation (VFI) are inter-dependent for enhancing videos of low resolution and low frame rate. However, most studies treat VSR and temporal VFI as independent tasks. In this work, we design a spatial-temporal superresolution network based on exploring the interaction between VSR and VFI. The main idea is to improve the middle frame of VFI by the super-resolution (SR) frames and feature maps from VSR. In the meantime, VFI also provides extra information for VSR and thus, through interacting, the SR of consecutive frames of the original video can also be improved by the feedback from the generated middle frame. Drawing on this, our approach leverages a simple interaction of VSR and VFI and achieves state-of-the-art performance on various datasets. Due to such a simple strategy, our approach is universally applicable to any existing VSR or VFI networks for effectively improving their video enhancement performance.

show abstract

Scene-Adaptive Video Frame Interpolation via Meta-Learning

Cited by 46 publications

References 24 publications

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

Multiple Video Frame Interpolation via Enhanced Deformable Separable Convolution

How Video Super-Resolution and Frame Interpolation Mutually Benefit

Contact Info

Product

Resources

About