On Spammer Detection In Crowdsourcing Pairwise Comparison Tasks: Case Study On Two Multimedia Qoe Assessment Scenarios

Abstract

The last decade has brought a surge in crowdsourcing platforms' popularity for the subjective quality evaluation of multimedia content. The lower need for intervention during the experiment and more expansive participant pools of crowdsourcing platforms encourage researchers to join this trend. However, the unreliability of the participant behaviors puts a barrier in the wide adoption of these platforms. Although many works exist to detect unreliable observers in rating experiments, there is still a lack of methodology for detecting unreliable observers in quality evaluation of multimedia content using pairwise comparison. In this work, we propose methods to identify irregular annotator behaviors in pairwise comparison paradigm. We compare the proposed methods' efficiency for two scenarios: quality evaluation of traditional 2D images and 3D interactive multimedia. We conducted two crowdsourcing experiments for two different Quality of Experience assessment tasks and inserted carefully designed synthetic spammer profiles to evaluate the proposed tools. Our results suggest that the detection of unreliable observers is highly task-dependent. The influence of the spammer behavior intensity and the proportion of spammers among the observers can be more severe on tasks with higher subjectivity. Based on these findings, we provide guidelines and recommendations towards developing spammer detection algorithms for subjective pairwise quality evaluation of multimedia content.

When is the Cleaning of Subjective Data Relevant to Train UGC Video Quality Metrics?

“…for ACR or Double Stimulus Impairment Scale (DSIS) scores. In case of binary answers, such as the expression of preference during a Pair Comparison (PC) test, recent works [16,17] favored the use of dissimilarity metrics (Rogers-Tanimoto (RT) dissimilarity [18], Cohen's Kappa Coefficient [19]) to estimate inter-observer agreement.…”

RV-TMO: Large-Scale Dataset for Subjective Quality Assessment of Tone Mapped Images

“…An early example of subjective IQA on crowdsourcing shows promise by comparing crowdsourcing and laboratory experiment results [12]. Recent works raise concerns on the effects of QoE tasks on crowdsourcing subjective experiments [13]. LIVE In the Wild [14] IQA dataset consists of over 350000 opinion scores on 1162 images.…”

“…Although several methodologies have been proposed for interobserver agreement and outlier detection in rating experiments [29], [30], there are not many well-established methodologies for ranking experiments. Ak et al [13] showed that inter-observer agreement in pairwise comparison experiments can be measured based on Rogers-Tanimoto (RT) dissimilarity measure. A similar variation to such metric, known as Jaccard index [31], has been developed by Paul Jaccard.…”

“…Therefore, each playlist is represented in a separate column on the figure. Based on the synthetic spammer profiles described in [13], we created an expected spammer RT dissimilarity range for each playlist. For a given playlist, each observer's RT dissimilarity distributions are then compared.…”

A machine-learning framework to predict TMO preference based on image and visual attention features

“…The main characteristics of these datasets are presented in Table I. 1) Exp-TMO Dataset: The Exp-TMO dataset has been recently published in [26]. This dataset contains originally 20 HDR sources processed with 4 different TMOs (see as Hypothetical Reference Circuits HRC) leading to a total of 120 pairs of tone mapped stimuli used in our experiment.…”