A Comprehensive Analysis of Crowdsourcing for Subjective Evaluation of Tone Mapping Operators

Ak, Ali; Goswami, Abhishek; Hauser, Wolf; Callet, Patrick Le; Dufaux, Frédéric

doi:10.2352/issn.2470-1173.2021.9.iqsp-262

Cited by 2 publications

(3 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As also pointed out in the aforementioned study, subjective data collected via crowdsourcing may be noisy due to the lack of control in the experiment conditions. Several methods have been developed to filter unreliable observers and noisy data [14], [15]. Ak et al [14] provided comprehensive analysis on the methods to detect unreliable observers in a crowdsourcing experiment for aesthetic quality evaluation of TMOs.…”

Section: Related Workmentioning

confidence: 99%

“…Several methods have been developed to filter unreliable observers and noisy data [14], [15]. Ak et al [14] provided comprehensive analysis on the methods to detect unreliable observers in a crowdsourcing experiment for aesthetic quality evaluation of TMOs. In literature, there are also traditional methods, such as reliability checks [16] and gold standards [17] where previously obtained results from reliable participants have been compared to crowdsourcers' responses to detect unreliable behaviours.…”

Section: Related Workmentioning

confidence: 99%

“…50 observers, 28 female and 22 male with 22.6 years average age, were recruited in total. Due to lower attention span of 1 Dataset and implementation codes for analysis can be found at ftp://ftp.polytech.univ-nantes.fr/TMOEval PilotStudies participants in crowdsourcing experiments, we split the initial dataset into 4 playlists of 5 SRCs with 30 comparisons in each [24] [14]. Each participant was asked to complete all 4 playlists without any constraint on the break between playlists.…”

Section: Subjective Experiments Platformsmentioning

confidence: 99%

See 2 more Smart Citations

Reliability of Crowdsourcing for Subjective Quality Evaluation of Tone Mapping Operators

Goswami

Hauser

et al. 2021

2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP)

Self Cite

View full text Add to dashboard Cite

Tone mapping operators (TMO) are functions which map high dynamic range (HDR) images to limited dynamic media while aiming to preserve the perceptual cues of the scene that govern its aesthetic quality. Evaluating aesthetic quality of TMOs is non-trivial due to the high subjectivity of preference involved. Traditionally, TMO aesthetic quality has been evaluated via subjective experiments in a controlled laboratory environment. However, the last decade has brought a surge in popularity of crowdsourcing as an alternative methodology to conduct subjective experiments. However, uncontrolled experiment conditions and unreliability of participant behaviour puts doubts on the trustworthiness of the collected data. In this study, we explore the possibility of using crowdsourcing platforms for subjective quality evaluation of TMOs. We have conducted three experiments with systematic changes to investigate the effect of experiment conditions and participant recruitment methods on the collected subjective data. Our results show that subjective evaluation of TMO aesthetic quality can be conducted on Prolific crowdsourcing platform with negligible differences in comparison to laboratory experiments. Furthermore, we provide objective conclusions about the effect of number of observers on the certainty of the pairwise comparison results.

show abstract

Section: Related Workmentioning

confidence: 99%