Enhanced Bi-directional Motion Estimation for Video Frame Interpolation

Jin, Xin; Wu, Longhai; Shen, Guotao; Chen, Youxin; Chen, Jie; Koo, Ja-Yoon

doi:10.1109/wacv56688.2023.00502

Cited by 18 publications

(18 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Existing mainstream methods of video frame interpolation can be generally categorized into flow-based methods [7][8][9]11,12,[15][16][17][18][19][20][21][22][23][24][25][26], kernel-based methods [1,5,10,[27][28][29][30], phase-based methods [31], and hallucination-based methods [2,3,14,32].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Arbitrary Timestep Video Frame Interpolation with Time-Dependent Decoding

Zhang,

Ren,

Yan

et al. 2024

Mathematics

View full text Add to dashboard Cite

Given an observed low frame rate video, video frame interpolation (VFI) aims to generate a high frame rate video, which has smooth video frames with higher frames per second (FPS). Most existing VFI methods often focus on generating one frame at a specific timestep, e.g., 0.5, between every two frames, thus lacking the flexibility to increase the video’s FPS by an arbitrary scale, e.g., 3. To better address this issue, in this paper, we propose an arbitrary timestep video frame interpolation (ATVFI) network with time-dependent decoding. Generally, the proposed ATVFI is an encoder–decoder architecture, where the interpolation timestep is an extra input added to the decoder network; this enables ATVFI to interpolate frames at arbitrary timesteps between input frames and to increase the video’s FPS at any given scale. Moreover, we propose a data augmentation method, i.e., multi-width window sampling, where video frames can be split into training samples with multiple window widths, to better leverage training frames for arbitrary timestep interpolation. Extensive experiments were conducted to demonstrate the superiority of our model over existing baseline models on several testing datasets. Specifically, our model trained on the GoPro training set achieved 32.50 on the PSNR metric on the commonly used Vimeo90k testing set.

show abstract

Section: Related Workmentioning

confidence: 99%

“…VFIformer [23] refines the warped frames with a Transformer [34] structure. UPR-Net [24] uses a unified pyramid recurrent network to estimate optical flow and synthesize frames. AMT [25] leverages bidirectional correlation volumes for all pairs of pixels in flow estimation and feature update.…”

Section: Related Workmentioning

confidence: 99%

Arbitrary Timestep Video Frame Interpolation with Time-Dependent Decoding

Zhang,

Ren,

Yan

et al. 2024

Mathematics

View full text Add to dashboard Cite

show abstract

“…where k ≤ c o and m, ñ indicate subtracting half of the kernel size, i.e., m = m − ⌊ k h 2 ⌋. To reduce the model size and improve generalization, some models reuse the same convolution block multiple times (Jin et al 2023;Sim, Oh, and Kim 2021;Teed and Deng 2020). However, the standard convolution operation requires the input and output to always have the same predefined number of channels, which hinders the ability to handle diverse inputs.…”

Section: Partial Kernel Convolution (Pkconv)mentioning

confidence: 99%

“…However, recurrent encoders in the spatial dimension have yet to be wellexplored for optical flow. Some works adopted it for video interpolation to generate multi-scale features, but that required either producing features of a fixed size (Sim, Oh, and Kim 2021) or treating the entire optical flow network as a recurrent unit (Jin et al 2023;Zhang, Zhao, and Wang 2020), which decreases the flexibility of the model.…”

Section: Recurrent Models For Optical Flowmentioning

confidence: 99%

Recurrent Partial Kernel Network for Efficient Optical Flow Estimation

Morimitsu,

Zhu,

et al. 2024

AAAI

View full text Add to dashboard Cite

Optical flow estimation is a challenging task consisting of predicting per-pixel motion vectors between images. Recent methods have employed larger and more complex models to improve the estimation accuracy. However, this impacts the widespread adoption of optical flow methods and makes it harder to train more general models since the optical flow data is hard to obtain. This paper proposes a small and efficient model for optical flow estimation. We design a new spatial recurrent encoder that extracts discriminative features at a significantly reduced size. Unlike standard recurrent units, we utilize Partial Kernel Convolution (PKConv) layers to produce variable multi-scale features with a single shared block. We also design efficient Separable Large Kernels (SLK) to capture large context information with low computational cost. Experiments on public benchmarks show that we achieve state-of-the-art generalization performance while requiring significantly fewer parameters and memory than competing methods. Our model ranks first in the Spring benchmark without finetuning, improving the results by over 10% while requiring an order of magnitude fewer FLOPs and over four times less memory than the following published method without finetuning. The code is available at github.com/hmorimitsu/ptlflow/tree/main/ptlflow/models/rpknet.

show abstract

“…Previous crossbreed coding schemes depended on each pixel-space operation to reduce repetitive actions in space and time [22]. Still, these systems could need help with motion estimation or motion pay [23]. The work built an element-fieldcoding video network (FCV) by modelling every significant operation in the component space (motion estimation, motion compression, motion remuneration, and lingering compression).…”

Section: Related Workmentioning

confidence: 99%

A Loss-initiated GAN-based Convolutional LSTM Method for Compression and Motion Estimation-Based Objective Enhancement in Images and Videos

Mudhavath,

Kumar,

Yadav

et al. 2024

Preprint

View full text Add to dashboard Cite

The issues of finding a suitable loss function for perceived similarity and enhancing perceptual quality in substantially compressed videos still need to be resolved. The LIGAN-Conv-LSTM is a convolutional long-short-term memory system that integrates a loss-initialised generative adversarial network. This system was developed to address the challenge of defining unified training objectives that improve both rough and smooth content. The goal is to enhance compression quality by employing advanced feature selection and prediction techniques. The study begins by presenting a comprehensive examination of multimedia, emphasising its diverse attributes, encompassing retention, awareness, conversation, and knowledge. The research employs Shape Local Binary Texture (SLBT), Overlapped Segmentation-based Independent Component Analysis (OS-ICA), and Scale-Invariant Feature Transform (SIFT) as approaches to extract useful features. Furthermore, this research acknowledges the necessity for enhanced motion estimates. To address this, a hybrid technique is proposed to integrate the most advantageous elements of coyote and tuna swarm optimisation. The objective is to enhance the accessibility of feature selection, a crucial factor in improving classifier performance. The primary contribution of the research involves replacing standard motion estimating algorithms with a Conv-LSTM network, enabling the prediction of concise latent representations for future frames. The study's results emphasise the efficacy of the proposed hybrid search algorithm in achieving a high level of matching accuracy, as demonstrated by testing on a representative collection of video clips. The planned task's accuracy level is 97.73, demonstrating its cost-effective productivity.

show abstract

Enhanced Bi-directional Motion Estimation for Video Frame Interpolation

Cited by 18 publications

References 34 publications

Arbitrary Timestep Video Frame Interpolation with Time-Dependent Decoding

Arbitrary Timestep Video Frame Interpolation with Time-Dependent Decoding

Recurrent Partial Kernel Network for Efficient Optical Flow Estimation

A Loss-initiated GAN-based Convolutional LSTM Method for Compression and Motion Estimation-Based Objective Enhancement in Images and Videos

Contact Info

Product

Resources

About