Computing Crowd Consensus with Partial Agreement

Abstract

Abstract-Crowdsourcing has been widely established as a means to enable human computation at large-scale, in particular for tasks that require manual labelling of large sets of data items. Answers obtained from heterogeneous crowd workers are aggregated to obtain a robust result. However, existing methods for answer aggregation are designed for discrete tasks, where answers are given as a single label per item. In this paper, we consider partial-agreement tasks that are common in many applications such as image tagging and document annotation, where items are assigned sets of labels. Common approaches for the aggregation of partial-agreement answers either (i) reduce the problem to several instances of an aggregation problem for discrete tasks or (ii) consider each label independently. Going beyond the state-of-the-art, we propose a novel Bayesian nonparametric model to aggregate the partial-agreement answers in a generic way. This model enables us to compute the consensus of partially-sound and partiallycomplete worker answers, while taking into account mutual relationships in labels and different answer sets. We also show how this model is instantiated for incremental learning, incorporating new answers from crowd workers as they arrive. An evaluation of our method using real-world datasets reveals that it consistently outperforms the state-of-the-art in terms of precision, recall, and robustness against faulty workers and data sparsity.

Maximal fusion of facts on the web with credibility guarantee

“…This assumption matches with real-life settings, e.g. training data, prior knowledge, crowdsourcing, human expert [11,20,21,22,23]. Note that the output size is also known as recall since the size of ground truth is fixed.…”

What-If Analysis with Conflicting Goals: Recommending Data Ranges for Exploration

“…Also, the baseline performs worse for small compression ratios (e.g. 10%), highlighting the practicality of our approach: Acknowledging cognitive load limits of users (b ≤ 20 according to [20,21,42,43,[47][48][49]), our approach helps to identify important data regularities, outperforming a (naive) uniform partitioning. Even with a compression ratio of 90%, the distortion of uniform partitioning is two times higher than our approach.…”

Active Multilabel Crowd Consensus

“…The sample-label information collected via crowdsourcing is generally erroneous, due to the fact that online workers may lack expertise and proper incentives [30], [32]. This heterogeneous nature leads to the diverse submission quality of the completed tasks, pressing an urgent need for quality control [26], [27], [29], [33]- [39].…”

“…As a result, they may perform poorly when dealing with the more general multi-label data setting, where each object may have a set of non-exclusive labels, and labels may exhibit semantic correlations. Several multi-label crowd consensus algorithms have been recently proposed [26], [27], [29], [38], [39]. Nowak et al [38] studied the inter-annotator agreement for multilabel image annotation and focused on the annotation quality differences between expert and non-expert workers.…”