Achieving Approximate Global Optimization of Truth Inference for Crowdsourcing Microtasks

Abstract

Microtask crowdsourcing is a form of crowdsourcing in which work is decomposed into a set of small, self-contained tasks, which each can typically be completed in a matter of minutes. Due to the various capabilities and knowledge background of the voluntary participants on the Internet, the answers collected from the crowd are ambiguous and the final answer aggregation is challenging. In this process, the choice of quality control strategies is important for ensuring the quality of the crowdsourcing results. Previous work on answer estimation mainly used expectation–maximization (EM) approach. Unfortunately, EM provides local optimal solutions and the estimated results will be affected by the initial value. In this paper, we extend the local optimal result of EM and propose an approximate global optimal algorithm for answer aggregation of crowdsourcing microtasks with binary answers. Our algorithm is expected to improve the accuracy of real answer estimation through further likelihood maximization. First, three worker quality evaluation models are presented based on static and dynamic methods, respectively, and the local optimal results are obtained based on the maximum likelihood estimation method. Then, a dominance ordering model (DOM) is proposed according to the known worker responses and worker categories for the specified crowdsourcing task to reduce the space of potential task-response sequence while retaining the dominant sequence. Subsequently, a Cut-point neighbor detection algorithm is designed to iteratively search for the approximate global optimal estimation in a reduced space, which works on the proposed dominance ordering model (DOM). We conduct extensive experiments on both simulated and real-world datasets, and the experimental results illustrate that the proposed approach can obtain better estimation results and has higher performance than regular EM-based algorithms.

Classifying Workers for Mitigating Adversarial Attacks in Crowdsourcing

“…It predicts the chances of a worker giving a correct answer using a PGM and a maximum likelihood function such as EM for maximizing the probability of a true answer [23,24]. Optimization-based methods use worker skills and the features such as worker reliability and worker confidence for optimizing the prediction of correct answers during the truth inference process [25,26]. In NN methods, the set of answers is fed to the neural network layer and infers the true answers [27,28].…”

A deep active learning-based and crowdsourcing-assisted solution for named entity recognition in Chinese historical corpora

“…Wallace et al propose a new ALC method to reduce the overall user bias of the aggregated labels by considering user voting behavior patterns across multiple items (Wallace et al, 2022). Some other studies ameliorate the aggregation quality using Bayesian-based optimization (Li et al, 2019;Wang et al, 2019;Cui et al, 2021). Cui et al, for instance, designs a novel pipeline approach that includes two key steps: one is the space reduction of potential task-response sequences based on a dominance ordering model and the other step is the global optimal estimation based on a newly-designed cut-point neighbor detection algorithm (Cui et al, 2021).…”

Clustering Based Approach for Ground Truth Inference in Crowdsourced Data

“…Due to the presence of noisy labels in crowdsourced data, generally, such data are not used for decision making in their raw form until further processing is done to infer ground truth from them (Bi, Wang, Kwok, & Tu, 2014;Adeogun & Odumuyiwa, 2019;Xu, Jiang & Li, 2021;Cui et. al., 2021).…”