Minimizing Efforts in Validating Crowd Answers

Abstract

In recent years, crowdsourcing has become essential in a wide range of Web applications. One of the biggest challenges of crowdsourcing is the quality of crowd answers as workers have wide-ranging levels of expertise and the worker community may contain faulty workers. Although various techniques for quality control have been proposed, a post-processing phase in which crowd answers are validated is still required. Validation is typically conducted by experts, whose availability is limited and who incur high costs. Therefore, we develop a probabilistic model that helps to identify the most beneficial validation questions in terms of both, improvement of result correctness and detection of faulty workers. Our approach allows us to guide the expert's work by collecting input on the most problematic cases, thereby achieving a set of high quality answers even if the expert does not validate the complete answer set. Our comprehensive evaluation using both real-world and synthetic datasets demonstrates that our techniques save up to 50% of expert efforts compared to baseline methods when striving for perfect result correctness. In absolute terms, for most cases, we achieve close to perfect correctness after expert input has been sought for only 20% of the questions.

Computing Crowd Consensus with Partial Agreement

“…Interactive crowdsourcing applications [42] might require very fast response time. However, simple divideand-conquer methods such as matrix partitioning [43] are not applicable, since a split-up of the answer matrix causes information loss on worker communities and item clusters.…”

“…Most answer aggregation algorithms operate in batch mode; hence, the aggregated answers would be recomputed from scratch every time a new worker answer arrives. There are only a few approaches on incremental answer aggregation, such as online EM [56]-which targets incremental updates when a new answer arrives and i-EM [43], [57]-which targets incremental updates whenever the ground truth is extended. However, tailoring such incremental methods for our setting is non-trivial due to the dependency between labels and the community modelling of workers and items; e.g.…”

“…a single new answer can be propagated to change the whole model, resulting in inefficient execution. Matrix partitioning [43] may also be used to reduce the computational effort, yet this incurs the loss of information about the dependencies between workers, which is essential for modelling the worker community. CPA provides a first method for answer aggregation for multi-label tasks, which scales not only in the number of possible labels, but also in the number of possible answers (subsets of possible labels).…”

Efficient Feedback Collection for Pay-as-you-go Source Selection

“…These sorted sources are then combined following a greedy approach (lines [15][16][17][18][19][20][21][22][23][24][25][26], that gives preference to those sources with the highest estimated precision, and stops when the threshold is met (lines [17][18][19][20][21][22]. If the threshold is not yet reached by adding the new sources to the subset of candidate sources S , we compute a new cut-off, to divide those sources that will be considered in a potential solution from those that will not (lines 16 and 23-25).…”

“…There has been significant recent interest in targeting the most effective feedback, in particular for crowdsourcing (e.g. [11,21,23,28,35]). …”