Max-Margin Majority Voting for Learning from Crowds

Abstract

Learning-from-crowds aims to design proper aggregation strategies to infer the unknown true labels from the noisy labels provided by ordinary web workers. This paper presents max-margin majority voting (M3V) to improve the discriminative ability of majority voting and further presents a Bayesian generalization to incorporate the flexibility of generative methods on modeling noisy observations with worker confusion matrices for different application settings. We first introduce the crowdsourcing margin of majority voting, then we formulate the joint learning as a regularized Bayesian inference (RegBayes) problem, where the posterior regularization is derived by maximizing the margin between the aggregated score of a potential true label and that of any alternative label. Our Bayesian model naturally covers the Dawid-Skene estimator and M3V as its two special cases. Due to the flexibility of our model, we extend it to handle crowdsourced labels with an ordinal structure with the main ideas about the crowdsourcing margin unchanged. Moreover, we consider an online learning-from-crowds setting where labels coming in a stream. Empirical results demonstrate that our methods are competitive, often achieving better results than state-of-the-art estimators.

Improving Learning-from-Crowds through Expert Validation

“…The input of our method are crowdsourced labels and the corresponding inferred correct labels. The inferred labels are computed by the M 3 V model [Tian and Zhu, 2015]. To incorporate expert labels, we first use an instance selection method with a complete uncertainty assessment to find the most informative instances to be validated by an expert (Fig.…”

“…To solve this problem, we have developed a loss-driven algorithm based on the M 3 V model [Tian and Zhu, 2015]. The main feature of our algorithm is that it jointly considers the influence of expert labels and other important factors such as the likelihood of crowdsourced labels.…”

“…However, due to individual differences among workers in terms of background, knowledge, and expertise, crowdsourced labels may be noisy and poor in quality. Researchers have developed a variety of effective learning-from-crowd algorithms to estimate correct labels from noisy data [Tian and Zhu, 2015;Zhou et al, 2012]. Although these algorithms achieve some success in increasing accuracy, the unsupervised nature of these algorithms limits their performance.…”

“…Our label propagation algorithm is based on the max-margin majority voting (M 3 V) model [Tian and Zhu, 2015], which is a state-of-the-art learning-from-crowd model to estimate the true labels from noisy data. This model introduces the concept of margin to improve its discriminative ability.…”

“…It is factorized as the product of a Dirichlet prior over Φ, a spherical Gaussian prior over η, and a uniform prior over y [Tian and Zhu, 2015]. p(X c |R) is the likelihood and is computed by the method proposed by Dawid and Skene [Dawid and Skene, 1979].…”