Crowdsourcing subjective image quality evaluation

Abstract

Subjective tests are generally regarded as the most reliable and definitive methods for assessing image quality. Nevertheless, laboratory studies are time consuming and expensive. Thus, researchers often choose to run informal studies or use objective quality measures, producing results which may not correlate well with human perception. In this paper we propose a cost-effective and convenient subjective quality measure called crowdMOS, obtained by having internet workers participate in MOS (mean opinion score) subjective quality studies. Since these workers cannot be supervised, we propose methods for detecting and discarding inaccurate or malicious scores. To facilitate this process, we offer an open source set of tools for Amazon Mechanical Turk, which is an internet marketplace for crowdsourcing. These tools completely automate the test design, score retrieval and statistical analysis, abstracting away the technical details of Mechanical Turk and ensuring a user-friendly, affordable and consistent test methodology. We demonstrate crowdMOS using data from the LIVE subjective quality image dataset, showing that it delivers accurate and repeatable results.Index Terms-crowdsourcing, subjective quality, quality assessment, mean opinion score, MOS, mechanical turk.

Crowdsourced study of subjective image quality

“…To the best of our knowledge, we are aware of only one other project [8] reporting efforts made in the same spirit as our work that we report here. However, the authors of [8] tested their crowdsourcing system only on 116 JPEG compressed images from the legacy LIVE Image Quality Database [6] and gathered opinion scores from only forty subjects.…”

“…However, the authors of [8] tested their crowdsourcing system only on 116 JPEG compressed images from the legacy LIVE Image Quality Database [6] and gathered opinion scores from only forty subjects. By contrast, we have so far collected over 320,000 human opinion scores on 1,163 naturally distorted images from over 7,000 distinct subjects and we plan to collect more than 350,000 subjective judgments overall, making it the world's largest, most comprehensive study of perceptual image quality ever conducted.…”

Impact of interactivity on the assessment of quality of experience for light field content

“…However, several studies have proven the efficacy of crowdsourcing-based tests [16], [17]. Moreover, while crowdsourcing is usually linked to larger standard errors, due to variability of conditions, the opposite has been observed in our experiment.…”

The impact of video-quality-level switching on user quality of experience in dynamic adaptive streaming over HTTP

“…These tests are generally performed under laboratory conditions. Nevertheless, in recent years, some studies [3,4] show the possibility to perform image or video quality assessment through remote assessors using the Internet.…”

“…2 Overview of DASH, quality adaptation, and visual quality assessment methods DASH is a new standard developed by 3GPP and MPEG [4,5,10] aiming to encode video files using different encoder parameters. Different versions of the same video are obtained and stored in a video server; in which each video version represents a different VQL.…”