Direct Video Frame Interpolation With Multiple Latent Encoders

Abstract

We present a simple but effective video interpolation framework that can be applied to various types of videos including conventional videos and 360 • videos. Our main idea is to predict the latent feature of an intermediate frame, through the latent feature encoders between encoder and decoder networks, without explicitly computing optical flow or depth maps. The latent feature encoders take latent features of input images and then predict the latent feature of a target image, i.e. an intermediate frame. Afterward, the decoder network reconstructs the target image from the latent feature. The proposed framework consists of fully convolutional networks, and it is therefore end-to-end trainable from scratch without requiring additional information except for consecutive frames. We experimentally verify the superiority of proposed method by comparing it to state-of-the-art methods with various types of datasets. Moreover, an ablation study is carried out to analyze the key components of the proposed method. Our proposed method performs interpolation in latent domain, it is advantageous to apply various video interpolation (e.g. NIR and depth videos) without limiting the type of input data. INDEX TERMSVideo interpolation, 360 • video interpolation, latent space learning, convolutional neural network VOLUME XX, 2020This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2021.3053695, IEEE Access Direct Video Frame Interpolation with Multiple Latent EncodersFIGURE 4. Network architecture of the image encoder, latent encoder, and latent decoder. Here, k, n, and s denote the convolution kernel size, the number of filter maps, and the stride, respectively.

DRVI: Dual Refinement for Video Interpolation

“…Many recently published papers focus on addressing the motion analysis. For example, FI-NET [2] computes optical flow at feature level instead of image level to make the motion estimation more accurate; [3] learns the latent motion features instead of learning the optical flow as the motion feature; [4] and [5] learn from 4 input images instead of 2 images and add some techniques like long short term memory (LSTM) and Multi-Frame Pyramid Refinement to predict the motions.…”