CrowdTruth: Machine-Human Computation Framework for Harnessing Disagreement in Gathering Annotated Data

Abstract

Abstract. In this paper, we introduce the CrowdTruth open-source software framework for machine-human computation, that implements a novel approach to gathering human annotation data in a wide range of annotation tasks and on a variety of media (e.g. text, images, videos). The CrowdTruth approach captures human semantics through a pipeline of three processes: a) combining various machine processing of text, image and video in order to understand better the input content and optimise its suitability for micro-tasks, thus optimise the time and cost of the crowdsourcing process; b) providing reusable human-computing task templates to collect the maximum diversity in the human interpretation, thus collect richer human semantics; and c) implementing 'disagreement metrics', i.e. CrowdTruth metrics, to support deep analysis of the quality and semantics of the crowdsourcing data. Instead of the traditional inter-annotator agreement, we use their disagreement as a useful signal to evaluate the data quality, ambiguity, and vagueness. In this paper we demonstrate the innovative CrowdTruth approaches embodied in the software to: 1) support processing of different text, image and video data; 2) support a variety of annotation tasks; 3) harness worker disagreement with CrowdTruth metrics; and 4) provide an interface to support data analysis and visualisation. In previous work we introduced the CrowdTruth methodology with examples for semantic interpretation of medical text for relation and factor extraction, and with newspaper text for event extraction. In this paper, we demonstrate the applicability and robustness of the approach to a wide variety of problems across a number of domains. We also show the advantages of using open standards and the extensibility of the framework with new data modalities and annotation tasks, as well as its openness to external services.

Crowdsourcing Disagreement for Collecting Semantic Annotation

“…We have published work on CrowdTruth [14], our framework for crowdsourcing ground truth data. CrowdTruth connects with both Amazon Mechanical Turk and CrowdFlower for launching and monitoring tasks, and implements the disagreement metrics for a live analysis of the results from the crowd.…”

On the impact of knowledge extraction and aggregation on crowdsourced annotation of visual artworks

“…This is an interesting phenomena, also observed in other knowledge-intensive content annotation use cases (e.g. medical [33]). The result suggests the need for more articulated annotations campaigns, possibly organised in workflows [34] that interleave automatic and human operations.…”

Crowdsourcing and the Semantic Web: A Research Manifesto

“…Novel frameworks for quality modeling and assessment are needed to reflect the richness of domain knowledge and insight that collective intelligence could bring into traditional computational processes, going beyond the rigid, Boolean view of ground truth used in many computer science areas. The work of Inel et al (2014) around the CrowdTruth framework proposes a useful starting point. Methods that match ontologies to microtask design could also offer a helpful way to prune the space of possible solutions.…”

“…Sarasua and Thimm (2014) propose Crowd Work CV, an ontology to capture crowd workers' and requesters' information across different crowdsourcing platforms. The CrowdTruth framework proposes a useful schema for annotating the provenance of crowdsourced data (Inel et al, 2014). A second challenge will be to identify the most appropriate level of granularity to record activities and their results to meet the requirements of various use cases (data aggregation, quality control, tasks assignment, personalization etc.…”

“…For this powerful idea to materialize, one needs tools that identify links between the individual data sets (e. g. annotating results from the crowds with external Linked Data URIs) or discover relevant crowd computing functionality. Connecting crowdsourced Linked Data to the LOD cloud 5 poses specific challenges: for instance, one could use this form of disambiguation of crowd results to identify incorrect answers or to resolve disagreement cases which are just a signal for poor accuracy rather than diversity of viewpoints (Inel et al, 2014). Offering a full-fledged semantic workflow solution remains one of the greatest challenges of Semantic Web research; narrowing the domain down to crowdsourcing scenarios might reduce its complexity.…”