FovVideoVDP

Abstract

FovVideoVDP is a video difference metric that models the spatial, temporal, and peripheral aspects of perception. While many other metrics are available, our work provides the first practical treatment of these three central aspects of vision simultaneously. The complex interplay between spatial and temporal sensitivity across retinal locations is especially important for displays that cover a large field-of-view, such as Virtual and Augmented Reality displays, and associated methods, such as foveated rendering. Our metric is derived from psychophysical studies of the early visual system, which model spatio-temporal contrast sensitivity, cortical magnification and contrast masking. It accounts for physical specification of the display (luminance, size, resolution) and viewing distance. To validate the metric, we collected a novel foveated rendering dataset which captures quality degradation due to sampling and reconstruction. To demonstrate our algorithm's generality, we test it on 3 independent foveated video datasets, and on a large image quality dataset, achieving the best performance across all datasets when compared to the state-of-the-art.

Towards Attention-aware Rendering for Virtual and Augmented Reality

“…2.1). In this section, we experimentally validate whether integration of our attention-aware perceptual model improves performance of the state-of-the-art visual difference predictor FovVideoVDP [Mantiuk et al 2021] in predicting visibility of foveation artifacts under varying attention conditions. To that end, we emulate a simple foveated renderer and we separately calibrate the foveation intensity for three different attention regimes in a user study.…”

“…This model also captured luminance dependence and was later refit to model stationary content at higher eccentricities [Watson 2018]. Recently, models capturing eccentricity dependence for the full spatio-temporal domain were also presented [Krajancich et al 2021;Mantiuk et al 2021;.…”

“…As seen in our results, distribution of contrast in specific images affects the acceptable foveation intensity. This is modeled by visual difference predictors such as FovVideoVDP [Mantiuk et al 2021], which decomposes an image into spatio-temporal frequency bands and models their visibility by utilizing the CSF and a contrast masking model. However, while explicitly modeling retinal eccentricity, the original CSF does not account for attention.…”

Video Quality Assessment Considering the Features of the Human Visual System

“…A more fundamental approach to creating video quality metrics includes low-level visual modelling based on psychophysical models, such as the contrast sensitivity function (CSF) [3]: the threshold at which a human observer can detect change in a given brightness pattern as a function of spatial and temporal frequency. The artefacts visible to the human visual system in early vision are regulated by the function of visual sensitivity.…”

“…Unfortunately, popular objective prediction models correlate poorly with subjective perceptions of quality by the human visual system (HVS) and depend on the systems or processes involved [2]. On the other hand, there are algorithmically complex video quality metrics (VQM) based on models of the human visual system [1], [3] , and an open question is whether complex video quality metrics based on models of the human visual system provide significantly better predictions than objective metrics. Another problem when used of visual models, is in developing video quality metrics we must represent the HVS in software, a task impeded by the limited new fundamental knowledge of the HVS perception of video content using modern equipment.…”