Deep Learning from Crowds

Abstract

Over the last few years, deep learning has revolutionized the field of machine learning by dramatically improving the state-of-the-art in various domains. However, as the size of supervised artificial neural networks grows, typically so does the need for larger labeled datasets. Recently, crowdsourcing has established itself as an efficient and cost-effective solution for labeling large sets of data in a scalable manner, but it often requires aggregating labels from multiple noisy contributors with different levels of expertise. In this paper, we address the problem of learning deep neural networks from crowds. We begin by describing an EM algorithm for jointly learning the parameters of the network and the reliabilities of the annotators. Then, a novel general-purpose crowd layer is proposed, which allows us to train deep neural networks end-to-end, directly from the noisy labels of multiple annotators, using only backpropagation. We empirically show that the proposed approach is able to internally capture the reliability and biases of different annotators and achieve new state-of-the-art results for various crowdsourced datasets across different settings, namely classification, regression and sequence labeling.

Deep Generative Crowdsourcing Learning with Worker Correlation Utilization

“…This idea was subsequently extended to other types of classifier models such as the Gaussian process classifier [13] . As deep learning booms, deep crowdsourcing learning using DNNs as classifier models has become a research trend in the crowdsourcing field [15][16][17] . In Ref.…”

“…To avoid the computation overhead of the EM algorithm, Refs. [16,17] added a layer of coefficients behind the classifier output as the worker capability parameter in light of the structure of neural networks. Thus, the classifier parameters and the worker capability parameters could be considered as parameters at different layers of the network and further updated in an end-to-end fashion by stochastic gradient descent.…”

“…Although Refs. [16,17] avoided the high computation complexity of EM optimization, their DNNs not only lacked the interpretable structure of the probabilistic graphical model but also failed to guarantee that the annotation likelihood or its lower bound was maximized.…”

“…This idea was then extended to other types of classifier priors, such as the Gaussian process classifier [13] . As deep learning achieves significant progress in various fields such as visual speech [14] , deep crowdsourcing learning that incorporates DNNs (to assimilate their representational learning advantages) has become a research trend in crowdsourcing [15][16][17] . Since EM-style optimization requires optimizing the classifier in each iteration, its computation complexity is undoubtedly high when the neural network model is complex [15] , which renders efficient optimization a concern for deep crowdsourcing learning.…”

“…Considering the multilayer structure of neural networks, an approach to solve this problem is to add a layer of coefficients behind the output layer of the neural network classifier to portray worker capability. Thus, the final output layer of the classifier corresponds to the crowdsourcing annotation prediction [16,17] . The optimization can be performed in an end-to-end manner, i.e., stochastic gradient descent of all parameters in the network is conducted on the loss of crowdsourcing annotation predictions and the classifier and worker capabilities are estimated simultaneously.…”