Time-Sensitive Bayesian Information Aggregation for Crowdsourcing Systems

Abstract

Many aspects of the design of efficient crowdsourcing processes, such as defining worker’s bonuses, fair prices and time limits of the tasks, involve knowledge of the likely duration of the task at hand. In this work we introduce a new time–sensitive Bayesian aggregation method that simultaneously estimates a task’s duration and obtains reliable aggregations of crowdsourced judgments. Our method, called BCCTime, uses latent variables to represent the uncertainty about the workers’ completion time, the tasks’ duration and the workers’ accuracy. To relate the quality of a judgment to the time a worker spends on a task, our model assumes that each task is completed within a latent time window within which all workers with a propensity to genuinely attempt the labelling task (i.e., no spammers) are expected to submit their judgments. In contrast, workers with a lower propensity to valid labelling, such as spammers, bots or lazy labellers, are assumed to perform tasks considerably faster or slower than the time required by normal workers. Specifically, we use efficient message-passing Bayesian inference to learn approximate posterior probabilities of (i) the confusion matrix of each worker, (ii) the propensity to valid labelling of each worker, (iii) the unbiased duration of each task and (iv) the true label of each task. Using two realworld public datasets for entity linking tasks, we show that BCCTime produces up to 11% more accurate classifications and up to 100% more informative estimates of a task’s duration compared to state–of–the–art methods

A Bayesian Approach for Sequence Tagging with Crowds

“…We expand this previous work by detailing the relationships between several annotator models and extending them to sequential classification. Here we focus on the core annotator representation, rather than extensions for clustering annotators (Venanzi et al, 2014;Moreno et al, 2015), modeling their dynamics (Simpson et al, 2013), adapting to task difficulty (Whitehill et al, 2009;Bachrach et al, 2012), or time spent by annotators (Venanzi et al, 2016).…”

Resource-Based View of Information Systems: Sustainable and Transient Competitive Advantage Perspectives

“…Cloud resource, though developed to possess unique competencies, is neither rare nor inimitable (Mitra et al, 2017). The Internet of Things and crowdsourcing, on the other hand, are highly reliant on external sources of information (Santaro et al, 2017;Venanzi et al, 2016). Meanwhile, big data will also concede on this dependence, acknowledging that it's roles will continue to morph as a long-term investment (Sedera et al, 2016).…”

Actively Estimating Crowd Annotation Consensus

“…A relatively smaller portion of the existing work in the active crowd-labeling literature concentrates on categorical annotations (Welinder & Perona, 2010;Yan, Rosales, Fung, & Dy, 2011;Mozafari, Sarkar, Franklin, Jordan, & Madden, 2014;Zhu, Xu, & Yan, 2015;Kamar, Hacker, & Horvitz, 2012;Kamar, Kapoor, & Horvitz, 2013Venanzi, Guiver, Kohli, & Jennings, 2016). These methods may also be adapted for binary annotations by considering only two categories.…”

“…Kamar et al (2015) focus on the problem of rectifying task-related bias of annotators and show that active learning with expert annotators can be used for alleviating bias. Venanzi et al (2016) use a time-sensitive Bayesian aggregation method to estimate the labeling duration and annotator profile in crowdsourcing systems. They detect bots, spammers or lazy annotators from the duration of their labeling process (either too short or too long).…”