Learning from Crowds and Experts

Abstract

SummaryCrowdsourcing services are often used to collect a large amount of labeled data for machine learning. Although they provide us an easy way to get labels at very low cost in a short period, they have serious limitations. One of them is the variable quality of the crowd-generated data. There have been many attempts to increase the reliability of crowdgenerated data and the quality of classifiers obtained from such data. However, in these problem settings, relatively few researchers have tried using expert-generated data to achieve further improvements. In this paper, we apply three models that deal with the problem of learning from crowds to this problem: a latent class model, a personal classifier model, and a data-dependent error model. We evaluate these methods against two baseline methods on a real data set to demonstrate the effectiveness of combining crowd-generated data and expert-generated data.

Minimizing Efforts in Validating Crowd Answers

“…Our approach aims at guiding an expert when validating input from crowd workers, which is different to other approaches for crowdsourcing that include experts, such as [17,23,24]. In particular, Karger et al [24] rely on experts that know the reliability of crowd workers, a premise that is not realistic in the general setting for crowdsourcing explored in this work, to prove the optimality of their approach.…”

“…Other work focuses on a related, but fundamentally different problem. The techniques presented in [17,23] target the identification of correct labels for new objects based on the labels for known objects, whereas we aim at validation, i.e., finding the correct labels for known objects. Truth Finding.…”

Learning from Crowds under Experts’ Supervision

“…Kajino et al [9] addressed this problem by extending some existing models straightforwardly. Wauthier and Jordan [18] also used some expert labels.…”

“…A large α will cause the prior distribution over w to peak steeply on the mean, thus the affect of absorbing one expert label will Set μ = μpost, Σ = Σpost; 8. end for 9. Output: Personal classifier ensemble F, mean μ and covariance matrix Σ; be relatively small, leading to a final classifier depending heavily on the mean of prior, which is the simple combination of personal classifiers.…”

Recovering Missing Labels of Crowdsourcing Workers

“…This trigged a new research subject on modeling the low-quality data in the crowdsourcing scenario. As Sheng et al [17] have shown that obtaining more labels is helpful even when the label quality is low, models have been proposed to infer the ground truth [11,12] or learn a classifier [16,10] from large but unreliable crowdsourcing datasets.…”