Measuring the Effectiveness of Gamesourcing Expert Oil Painting Annotations

Abstract

Abstract. Tasks that require users to have expert knowledge are difficult to crowdsource. They are mostly too complex to be carried out by non-experts and the available experts in the crowd are difficult to target. Adapting an expert task into a non-expert user task, thereby enabling the ordinary "crowd" to accomplish it, can be a useful approach. We studied whether a simplified version of an expert annotation task can be carried out by non-expert users. Users conducted a game-style annotation task of oil paintings. The obtained annotations were compared with those from experts. Our results show a significant agreement between the annotations done by experts and non-experts, that users improve over time and that the aggregation of users' annotations per painting increases their precision.

On the impact of knowledge extraction and aggregation on crowdsourced annotation of visual artworks

“…Authors found crowd annotators, drawn from museum attendees, to use a different vocabulary than professional ones, but that such annotations were effective to improve the retrieval of the artworks. In [15], crowds…”

“…Previous works investigated how crowds can support the artwork annotation process [13,14,15]. For instance, "The Steve project" [13] studied crowd tagging of collections from more than 12 USA-based museums and compared crowd and professional taggers.…”

Measuring and improving data quality of media collections for professional tasks

“…We investigated whether crowd workers can perform a simplified version of an expert task if they are given assistance and how well they perform compared to experts [8]. We showed that the crowd workers' contributions were largely in line with the experts' judgements and that some cases of strong disagreement indicated need for re-evaluation on the experts' side.…”

“…For tasks with higher complexity or required expert knowledge, however, users must be trained in order to fully understand the task and provide high quality contributions. Our initial study [8] suggests that by combining these three components and creating feedback loops between them, we can create a system that successively leads to substantial improvements in data sets.…”

“…For tasks with higher complexity or required expert knowledge, however, users must be trained in order to fully understand the task and provide high quality contributions. Our initial study [8] suggests that by combining these three components and creating feedback loops between them, we can create a system that successively leads to substantial improvements in data sets.In order to measure the improvement of the data quality, we need a suitable quality metric. Using the data custodians' judgements as a quality measure for data may not always reflect the usefulness of the data for professional users.…”