FloLPIPS: A Bespoke Video Quality Metric for Frame Interpolation

Danier, Duolikun; Zhang, Fan; Bull, David

doi:10.1109/pcs56426.2022.10018062

Cited by 10 publications

(5 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite the paradigmatic shift from the mainstream VFI methods, we adhere to the commonly adopted VFI benchmarking protocol and evaluate the proposed method on various VFI test sets, covering both low and high resolution content (up to 4K). Our results demonstrate that LDMVFI performs favorably against the state of the art in terms of three perceptual metrics (Zhang et al 2018;Danier, Zhang, and Bull 2022b;Heusel et al 2017). We also conducted a user study to collect subjective judgments on the quality of full HD videos interpolated by our method benchmarked against several competitive counterparts; this further confirms the favorable perceptual performance of LDMVFI.…”

Section: Forward Diffusionmentioning

confidence: 63%

“…Evaluation Methods. As the main focus of this work is on improving the perceptual quality of interpolated content, we adopt a perceptual image quality metric LPIPS (Zhang et al 2018), and a bespoke VFI metric, FloLPIPS (Danier, Zhang, and Bull 2022b) for performance evaluation. These metrics have shown superior correlation with human judgments of VFI quality compared to commonly used quality measurements, PSNR and SSIM (Wang et al 2004).…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

LDMVFI: Video Frame Interpolation with Latent Diffusion Models

Danier,

Zhang,

Bull

2024

AAAI

View full text Add to dashboard Cite

Existing works on video frame interpolation (VFI) mostly employ deep neural networks that are trained by minimizing the L1, L2, or deep feature space distance (e.g. VGG loss) between their outputs and ground-truth frames. However, recent works have shown that these metrics are poor indicators of perceptual VFI quality. Towards developing perceptually-oriented VFI methods, in this work we propose latent diffusion model-based VFI, LDMVFI. This approaches the VFI problem from a generative perspective by formulating it as a conditional generation problem. As the first effort to address VFI using latent diffusion models, we rigorously benchmark our method on common test sets used in the existing VFI literature. Our quantitative experiments and user study indicate that LDMVFI is able to interpolate video content with favorable perceptual quality compared to the state of the art, even in the high-resolution regime. Our code is available at https://github.com/danier97/LDMVFI.

show abstract

Section: Forward Diffusionmentioning

confidence: 63%

Section: Methodsmentioning

confidence: 99%

LDMVFI: Video Frame Interpolation with Latent Diffusion Models

Danier,

Zhang,

Bull

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Their results show more than 20% coding gain with VMAF in the loss function when compared to a pure MSE loss. There has been sufficient work that show DFQMs [6,8,9] are more closely Fig. 2.…”

Section: Related Workmentioning

confidence: 94%

“…Historic DFQMs (Q) [6,8,9] are based on comparing the internal representations of a reference (x) and degraded (y) patch, Q(F (x), F (y)). A key point is that the Fz are extracted from frozen layers of a network trained for classification.…”

Section: Deep Feature Selectionmentioning

confidence: 99%

See 1 more Smart Citation

Learnt Deep Hyperparameter selection in Adversarial Training for compressed video enhancement with perceptual critic

Ramsook¹,

Kokaram²

2023

Preprint

View full text Add to dashboard Cite

Image based Deep Feature Quality Metrics (DFQMs) have been shown to better correlate with subjective perceptual scores over traditional metrics. The fundamental focus of these DFQMs is to exploit internal representations from a large scale classification network as the metric feature space. Previously, no attention has been given to the problem of identifying which layers are most perceptually relevant. In this paper we present a new method for selecting perceptually relevant layers from such a network, based on a neuroscience interpretation of layer behaviour. The selected layers are treated as a hyperparameter to the critic network in a W-GAN. The critic uses the output from these layers in the preliminary stages to extract perceptual information. A video enhancement network is trained adversarially with this critic. Our results show that the introduction of these selected features into the critic yields up to 10% (FID) and 15% (KID) performance increase against other critic networks that do not exploit the idea of optimised feature selection.

show abstract

Video Frame Interpolation for Large Motion with Generative Prior

Huang,

Jia,

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

FloLPIPS: A Bespoke Video Quality Metric for Frame Interpolation

Cited by 10 publications

References 30 publications

LDMVFI: Video Frame Interpolation with Latent Diffusion Models

LDMVFI: Video Frame Interpolation with Latent Diffusion Models

Learnt Deep Hyperparameter selection in Adversarial Training for compressed video enhancement with perceptual critic

Video Frame Interpolation for Large Motion with Generative Prior

Contact Info

Product

Resources

About