Community-based bayesian aggregation models for crowdsourcing

Venanzi, Matteo; Guiver, John; Kazai, Gabriella; Kohli, Pushmeet; Shokouhi, Milad

doi:10.1145/2566486.2567989

Cited by 200 publications

(193 citation statements)

References 9 publications

Supporting

Mentioning

193

Contrasting

Order By: Relevance

“…However, existing worker models [22], [23], [24], [25] cannot be applied directly in partial-agreement answer aggregation, since worker answers may be partially overlapping. Moreover, interpreting a missing label as a negative answer is not always correct and thus shall be cross-checked by answers from other workers.…”

Section: Motivating Examplementioning

confidence: 99%

Computing Crowd Consensus with Partial Agreement

Hùng

Viet

Tâm

et al. 2018

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

Abstract-Crowdsourcing has been widely established as a means to enable human computation at large-scale, in particular for tasks that require manual labelling of large sets of data items. Answers obtained from heterogeneous crowd workers are aggregated to obtain a robust result. However, existing methods for answer aggregation are designed for discrete tasks, where answers are given as a single label per item. In this paper, we consider partial-agreement tasks that are common in many applications such as image tagging and document annotation, where items are assigned sets of labels. Common approaches for the aggregation of partial-agreement answers either (i) reduce the problem to several instances of an aggregation problem for discrete tasks or (ii) consider each label independently. Going beyond the state-of-the-art, we propose a novel Bayesian nonparametric model to aggregate the partial-agreement answers in a generic way. This model enables us to compute the consensus of partially-sound and partiallycomplete worker answers, while taking into account mutual relationships in labels and different answer sets. We also show how this model is instantiated for incremental learning, incorporating new answers from crowd workers as they arrive. An evaluation of our method using real-world datasets reveals that it consistently outperforms the state-of-the-art in terms of precision, recall, and robustness against faulty workers and data sparsity.

show abstract

Section: Motivating Examplementioning

confidence: 99%

Computing Crowd Consensus with Partial Agreement

Hùng

Viet

Tâm

et al. 2018

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

show abstract

“…This is a common problem with approaches to inference that use maximum likelihood or maximum a-posteriori solutions [4]. In order to overcome this limitation, algorithms for aggregating crowdsourced data including SFilter [8] and Bayesian Classifier Combination (BCC) [14,30] capture the uncertainty in the workers' skill levels or bias, as well as the uncertainty in the aggregated labels. Unfortunately, these methods do not exploit the text features of documents, and consequently require each document to be labelled by the crowd, often multiple times, to obtain confident classifications.…”

Section: Aggregating Judgementsmentioning

confidence: 99%

“…It learns both the confusion matrices of each community and each worker but, like IBCC, it does not account for text features in the documents [30]. We run CBCC with three communities as suggested in the original paper, for both CF and SP.…”

Section: Independent Bayesian Classifier Combination (Ibcc)mentioning

confidence: 99%

See 1 more Smart Citation

Language Understanding in the Wild

Simpson

Venanzi

Reece

et al. 2015

Proceedings of the 24th International Conference on World Wide Web

Self Cite

View full text Add to dashboard Cite

Social media has led to the democratisation of opinion sharing. A wealth of information about public opinions, current events, and authors' insights into specific topics can be gained by understanding the text written by users. However, there is a wide variation in the language used by different authors in different contexts on the web. This diversity in language makes interpretation an extremely challenging task. Crowdsourcing presents an opportunity to interpret the sentiment, or topic, of free-text. However, the subjectivity and bias of human interpreters raise challenges in inferring the semantics expressed by the text. To overcome this problem, we present a novel Bayesian approach to language understanding that relies on aggregated crowdsourced judgements. Our model encodes the relationships between labels and text features in documents, such as tweets, web articles, and blog posts, accounting for the varying reliability of human labellers. It allows inference of annotations that scales to arbitrarily large pools of documents. Our evaluation using two challenging crowdsourcing datasets shows that by efficiently exploiting language models learnt from aggregated crowdsourced labels, we can provide up to 25% improved classifications when only a small portion, less than 4% of documents has been labelled. Compared to the six state-of-the-art methods, we reduce by up to 67% the number of crowd responses required to achieve comparable accuracy. Our method was a joint winner of the CrowdFlowerCrowdScale 2013 Shared Task challenge at the conference on Human Computation and Crowdsourcing (HCOMP 2013).

show abstract

“…Several research works have focused on improving the quality of crowdsourcing results by using a variety of techniques ranging from worker pre-screening methods and effective crowdsourcing task design [42], to using gamification and incentive mechanisms [19,59], and answer aggregation methods [67]. Due to the low entry barrier, crowdsourcing has become truly ubiquitous [68].…”

Section: Introduction "Sometimes the Internet Fee Is Greater Than Thementioning

confidence: 99%

Modus Operandi of Crowd Workers

Gadiraju

Checco

Gupta

et al. 2017

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that of work environments; defined as the hardware and software affordances at the disposal of crowd workers which are used to complete microtasks on crowdsourcing platforms. In this paper, we reveal the significant role of work environments in the shaping of crowd work. First, through a pilot study surveying the good and bad experiences workers had with UI elements in crowd work, we revealed the typical issues workers face. Based on these findings, we then deployed over 100 distinct microtasks on CrowdFlower, addressing workers in India and USA in two identical batches. These tasks emulate the good and bad UI element designs that characterize crowdsourcing microtasks. We recorded hardware specifics such as CPU speed and device type, apart from software specifics including the browsers used to complete tasks, operating systems on the device, and other properties that define the work environments of crowd workers. Our findings indicate that crowd workers are embedded in a variety of work environments which influence the quality of work produced. To confirm and validate our data-driven findings we then carried out semi-structured interviews with a sample of Indian and American crowd workers from this platform. Depending on the design of UI elements in microtasks, we found that some work environments support crowd workers more than others. Based on our overall findings resulting from all the three studies, we introduce ModOp, a tool that helps to design crowdsourcing microtasks that are suitable for diverse crowd work environments. We empirically show that the use of ModOp results in reducing the cognitive load of workers, thereby improving their user experience without effecting the accuracy or task completion time.

show abstract

Community-based bayesian aggregation models for crowdsourcing

Cited by 200 publications

References 9 publications

Computing Crowd Consensus with Partial Agreement

Computing Crowd Consensus with Partial Agreement

Language Understanding in the Wild

Modus Operandi of Crowd Workers

Contact Info

Product

Resources

About