Deep Generative Crowdsourcing Learning with Worker Correlation Utilization

Abstract

Traditional supervised learning requires the groundtruth labels for the training data, which can be difficult to collect in many cases. In contrast, crowdsourcing learning collects noisy annotations from multiple non-expert workers and infers the latent true labels through some aggregation approach. In this paper, we notice that existing deep crowdsourcing work does not sufficiently model worker correlations, which is, however, shown to be helpful for learning by previous non-deep learning approaches. We propose a deep generative crowdsourcing learning approach to incorporate the strengths of Deep Neural Networks (DNNs) and exploit worker correlations. The model comprises a DNN classifier as a prior and an annotation generation process. A mixture model of workers' capabilities within each class is introduced into the annotation generation process for worker correlation modeling. For adaptive trade-off between model complexity and data fitting, we implement fully Bayesian inference. Based on the naturalgradient stochastic variational inference techniques developed for the Structured Variational AutoEncoder (SVAE), we combine variational message passing for conjugate parameters and stochastic gradient descent for DNN parameters into a unified framework for efficient endto-end optimization. Experimental results on 22 real crowdsourcing datasets demonstrate the effectiveness of the proposed approach.

Collective Assignment of Complex Crowdsourcing Tasks Based on the KM Algorithm

“…Crowdsourcing is a new type of task assignment model, which uses the wisdom of the masses to distribute and collaborate to complete tasks [1]. In traditional crowdsourcing platforms such as Amazon Mechanical Turk, workers can independently complete assigned simple tasks in a short period of time, such as image annotation [2], text labeling [3], etc. With the increasing complexity of crowdsourcing application scenarios, tasks are also becoming more and more complex.…”