How reliable are annotations via crowdsourcing

Abstract

The creation of golden standard datasets is a costly business. Optimally more than one judgment per document is obtained to ensure a high quality on annotations. In this context, we explore how much annotations from experts differ from each other, how different sets of annotations influence the ranking of systems and if these annotations can be obtained with a crowdsourcing approach. This study is applied to annotations of images with multiple concepts. A subset of the images employed in the latest ImageCLEF Photo Annotation competition was manually annotated by expert annotators and non-experts with Mechanical Turk. The inter-annotator agreement is computed at an image-based and concept-based level using majority vote, accuracy and kappa statistics. Further, the Kendall τ and Kolmogorov-Smirnov correlation test is used to compare the ranking of systems regarding different ground-truths and different evaluation measures in a benchmark scenario. Results show that while the agreement between experts and non-experts varies depending on the measure used, its influence on the ranked lists of the systems is rather small. To sum up, the majority vote applied to generate one annotation set out of several opinions, is able to filter noisy judgments of non-experts to some extent. The resulting annotation set is of comparable quality to the annotations of experts.

Ranking Images Based on Aesthetic Qualities

“…These include the use of pre-labelled gold standard dataset, collecting multiple data labels for majority voting and modelling the labeller quality [9,120,124,81]. To validate annotations performed in this work, measures of agreement and correlation will be utilized [86,95]. These can be computed from the annotated lists efficiently to check consistency.…”

“…Measures for this process will be based on existing measures of agreement and correlation in literature but will be incorporated to test the problems being considered [86,95].…”

“…One is annotated by an expert and the other one is repeated in all lists for annotators. In the study conducted in [95] it was found that expert annotators showed a high consistency in annotations measured on agreement and correlation.…”

“…Due to these factors the areas of tackling annotation noise, matching tasks to workers, bonus payments for high quality results has gained considerable attention [95,51,56,113].…”

“…The level of difference between expert annotations was investigated in [95]. They verified if the annotations differ much and whether they need to be repeated.…”