Hacking VMAF and VMAF NEG: Vulnerability to Different Preprocessing Methods

Siniukov, Maksim; Antsiferova, Anastasia; Kulikov, Dmitriy; Vatolin, Dmitriy

doi:10.1145/3508259.3508272

Cited by 9 publications

(3 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has been shown that VMAF's scores can approximate well the scores that humans would give [25]. However, it is vulnerable to pre-processing methods that distort the video to artificially increase its VMAF score (without improving its user-perceived quality) [28].…”

Section: Problemmentioning

confidence: 99%

“…The typical way to address MOS's limitations is by taking the human out of the loop and focusing on specific applications. E.g., VMAF [18] leverages machine learning to learn a mapping from low-level videoperformance metrics to MOS scores; this does remove the biases and inconsistencies due to human subjectivity, but is susceptible to manipulation [28].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning a QoE Metric from Social Media and Gaming Footage

Alvarez,

Argyraki

2023

Proceedings of the 22nd ACM Workshop on Hot Topics in Networks

View full text Add to dashboard Cite

Defining a universal metric for Quality of Experience (QoE) is notoriously hard due to the complex relationship between low-level performance metrics and user satisfaction. The most common metric, the Mean Opinion Score (MOS), has wellknown biases and inconsistency issues. We propose an alternative that leverages (a) social-media comments on network performance and (b) streaming footage that includes performance numbers. We argue that our proposal is feasible for online gaming, and it may apply to other applications in the near future. We discuss its potential to enable a direct mapping from low-level performance metrics to accurate QoE scores-the golden standard for assessing user satisfaction.

show abstract

Section: Problemmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Learning a QoE Metric from Social Media and Gaming Footage

Alvarez,

Argyraki

2023

Proceedings of the 22nd ACM Workshop on Hot Topics in Networks

View full text Add to dashboard Cite

show abstract

“…libaom (Deng, Han, and Xu 2020) and LCEVC (V-Nova 2023) have options that optimize bitstream for increasing a VMAF score. Such tuning was designed to improve the visual quality of the encoded video; however, as VMAF is a learning-based metric, it may decrease perceptual quality (Zvezdakova et al 2019;Siniukov et al 2021). Using unstable image quality metrics as a perceptual proxy in a loss function may lead to incorrect restoration results (Ding et al 2021).…”

Section: Introductionmentioning

confidence: 99%

Comparing the Robustness of Modern No-Reference Image- and Video-Quality Metrics to Adversarial Attacks

Antsiferova,

Abud,

Gushchin

et al. 2024

AAAI

View full text Add to dashboard Cite

Nowadays, neural-network-based image- and video-quality metrics perform better than traditional methods. However, they also became more vulnerable to adversarial attacks that increase metrics' scores without improving visual quality. The existing benchmarks of quality metrics compare their performance in terms of correlation with subjective quality and calculation time. Nonetheless, the adversarial robustness of image-quality metrics is also an area worth researching. This paper analyses modern metrics' robustness to different adversarial attacks. We adapted adversarial attacks from computer vision tasks and compared attacks' efficiency against 15 no-reference image- and video-quality metrics. Some metrics showed high resistance to adversarial attacks, which makes their usage in benchmarks safer than vulnerable metrics. The benchmark accepts submissions of new metrics for researchers who want to make their metrics more robust to attacks or to find such metrics for their needs. The latest results can be found online: https://videoprocessing.ai/benchmarks/metrics-robustness.html.

show abstract