Recover Subjective Quality Scores from Noisy Measurements

Abstract

Simple quality metrics such as PSNR are known to not correlate well with subjective quality when tested across a wide spectrum of video content or quality regime. Recently, efforts have been made in designing objective quality metrics trained on subjective data (e.g. VMAF), demonstrating better correlation with video quality perceived by human. Clearly, the accuracy of such a metric heavily depends on the quality of the subjective data that it is trained on. In this paper, we propose a new approach to recover subjective quality scores from noisy raw measurements, using maximum likelihood estimation, by jointly estimating the subjective quality of impaired videos, the bias and consistency of test subjects, and the ambiguity of video contents all together. We also derive closed-from expression for the confidence interval of each estimate. Compared to previous methods which partially exploit the subjective information, our approach is able to exploit the information in full, yielding tighter confidence interval and better handling of outliers without the need for z-scoring or subject rejection. It also handles missing data more gracefully. Finally, as side information, it provides interesting insights on the test subjects and video contents.

Current State of Mixed Reality Technology for Digital Retail: A Literature Review

“…[32,40] and [50] talks about quality levels in free viewpoint videos and 3D videos; [36] talks about light field imaging which is another form of immersive 3D environment. Other mentionable works for 3D media are [39,45,47] and [51].…”

“…• Customer experience: [7,11,12,16,17,19,20,25,53,73] • Product experience: [7,21,23,24] • QoE: [29,30,45,47,49] • Influence factors in QoE: [28, 31, 33-41, 43, 44, 46, 48] • Pipeline/Analysis of 3D environments: [50,51,[67][68][69][70] • Acceptance of technologies: [9,15,24,26,27] • Methodologies to evaluate experience: [8,9,12,18] • Used cases of MR: [7-9, 13, 14, 66] Now the articles clustered as customer experience and product experience talks about different experience frameworks in Retail. They also talk about different components and constraints attached to the experience factors.…”

Modeling and estimating the subjects’ diversity of opinions in video quality assessment: a neural network based approach

“…The analysis is conducted on five subjectively annotated datasets, i.e., the ITS4S dataset [7,22], the Netflix public dataset [15] and three datasets released by the Video Quality Expert Group (VQEG): the VQEG-HD1 [27], VQEG-HD3 [27], and VQEG-HD5 [27]. Sample images taken from SRCs in those datasets are shown in Fig.…”

“…outlier detection, likelihood estimation, etc.) to deal with the problem of identifying unusual and strange behavior in the data [10,11,[11][12][13]15].…”

A common framework for the evaluation of psychophysiological visual quality assessment

“…In quality assessment outliers can occur on trial-level and on subject-level. For conventional quality assessment a heuristic outlier removal method has been prescribed in [31]; recently, more sophisticated statistically-motivated approaches to outlier detection have been proposed [37]. As mentioned earlier in this subsection simple trial-wise outlier rejection methods are widely used in neurophysiological quality assessment.…”