The Accuracy of Subjects in a Quality Experiment: A Theoretical Subject Model

Janowski, Lucjan; Pinson, Margaret H.

doi:10.1109/tmm.2015.2484963

Cited by 67 publications

(73 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The parameter a quantifies the variance of the user ratings more appropriately than the average over all stimuli. Moreover, it characterises application categories and correlates with task difficulty [14]. For VQA, the SOS parameter a was reported to fall in the range [0.11, 0.21] [13].…”

Section: B Crowdsourcing Results and Analysismentioning

confidence: 99%

The Konstanz natural video database (KoNViD-1k)

Hosu

Hahn

Jenadeleh

et al. 2017

2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX)

260

245

View full text Add to dashboard Cite

Subjective video quality assessment (VQA) strongly depends on semantics, context, and the types of visual distortions. Currently, all existing VQA databases include only a small number of video sequences with artificial distortions. The development and evaluation of objective quality assessment methods would benefit from having larger datasets of real-world video sequences with corresponding subjective mean opinion scores (MOS), in particular for deep learning purposes. In addition, the training and validation of any VQA method intended to be 'general purpose' requires a large dataset of video sequences that are representative of the whole spectrum of available video content and all types of distortions. We report our work on KoNViD-1k, a subjectively annotated VQA database consisting of 1,200 publicdomain video sequences, fairly sampled from a large public video dataset, YFCC100m. We present the challenges and choices we have made in creating such a database aimed at 'in the wild' authentic distortions, depicting a wide variety of content.

show abstract

Section: B Crowdsourcing Results and Analysismentioning

confidence: 99%

The Konstanz natural video database (KoNViD-1k)

Hosu

Hahn

Jenadeleh

et al. 2017

2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX)

260

245

View full text Add to dashboard Cite

show abstract

“…Z-score transformation of subjective scores is proposed in [7] as a pre-processing step prior to subject rejection. A theoretical model for subjects' influence on test scores is proposed in [13], which is similar in spirit to ours. The authors have focused on validating the model using real data, which also provides support for this work.…”

Section: Related Workmentioning

confidence: 96%

Recover Subjective Quality Scores from Noisy Measurements

Bampis

2017

2017 Data Compression Conference (DCC)

View full text Add to dashboard Cite

Simple quality metrics such as PSNR are known to not correlate well with subjective quality when tested across a wide spectrum of video content or quality regime. Recently, efforts have been made in designing objective quality metrics trained on subjective data (e.g. VMAF), demonstrating better correlation with video quality perceived by human. Clearly, the accuracy of such a metric heavily depends on the quality of the subjective data that it is trained on. In this paper, we propose a new approach to recover subjective quality scores from noisy raw measurements, using maximum likelihood estimation, by jointly estimating the subjective quality of impaired videos, the bias and consistency of test subjects, and the ambiguity of video contents all together. We also derive closed-from expression for the confidence interval of each estimate. Compared to previous methods which partially exploit the subjective information, our approach is able to exploit the information in full, yielding tighter confidence interval and better handling of outliers without the need for z-scoring or subject rejection. It also handles missing data more gracefully. Finally, as side information, it provides interesting insights on the test subjects and video contents.

show abstract

“…In rating experiments the random variable associated with the quality can be expressed using the following model of observer rating behavior [22]:…”

Section: A Observer Modelmentioning

confidence: 99%

From Pairwise Comparisons and Rating to a Unified Quality Scale

Pérez-Ortiz

Mikhailiuk

Zerman

et al. 2020

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

The goal of psychometric scaling is the quantification of perceptual experiences, understanding the relationship between an external stimulus, the internal representation and the response. In this paper, we propose a probabilistic framework to fuse the outcome of different psychophysical experimental protocols, namely rating and pairwise comparisons experiments. Such a method can be used for merging existing datasets of subjective nature and for experiments in which both measurements are collected. We analyze and compare the outcomes of both types of experimental protocols in terms of time and accuracy in a set of simulations and experiments with benchmark and realworld image quality assessment datasets, showing the necessity of scaling and the advantages of each protocol and mixing. Although most of our examples focus on image quality assessment, our findings generalize to any other subjective quality-of-experience task.

show abstract

The Accuracy of Subjects in a Quality Experiment: A Theoretical Subject Model

Cited by 67 publications

References 15 publications

The Konstanz natural video database (KoNViD-1k)

The Konstanz natural video database (KoNViD-1k)

Recover Subjective Quality Scores from Noisy Measurements

From Pairwise Comparisons and Rating to a Unified Quality Scale

Contact Info

Product

Resources

About