Aggregating Unstructured Submissions for Reliable Answers in Crowdsourcing Systems

Kurup, Ayswarya R; Sajeev, G P

doi:10.1109/ised48680.2019.9096224

Cited by 4 publications

(4 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most of the conventional truth inference algorithms use Probabilistic Graphical models (PGMs). It predicts the chances of a worker giving a correct answer using a PGM and a maximum likelihood function such as EM for maximizing the probability of a true answer [23,24]. Optimization-based methods use worker skills and the features such as worker reliability and worker confidence for optimizing the prediction of correct answers during the truth inference process [25,26].…”

Section: Truth Inference and Quality Control In Crowdsourcingmentioning

confidence: 99%

Classifying Workers for Mitigating Adversarial Attacks in Crowdsourcing

First,

Second

2023

Preprint

View full text Add to dashboard Cite

Crowdsourcing is adopted as a fast and cost-effective system for human computation and acquiring data for training models in machine learning. Although crowdsourcing has broad applicability, it still has the following challenges. Normally, workers have access to crowdsourcing platforms with simple authentication mechanisms. As a result, malicious workers may get into the system and submit unreliable answers rendering the platform fraudulent. Moreover, when workers perform tasks, their lack of expertise and the difficulty of tasks affects the accuracy. Truth inference benefits the data quality and worker reliability. A truth inference algorithm estimates workers' trustworthiness and correctness of the answers from their responses. Besides, it helps in filtering out low-quality answers. However, due to adversarial attacks by malicious workers, ground truth inference algorithms perform poorly. This research considers how to defend against adversarial attacks on crowdsourcing platforms and improve the truth inference process. The proposed method estimates the trust and reliability scores of the workers and classifies them as normal and malicious workers. Based on this classification , tasks are assigned to the workers. Moreover, these predicted scores are used for inferring the correct answers, thereby improving the ground truth inference. Experiments consistently show that the proposed truth inference method is tolerant to adversarial attacks with competent accuracy.

show abstract

Section: Truth Inference and Quality Control In Crowdsourcingmentioning

confidence: 99%

Classifying Workers for Mitigating Adversarial Attacks in Crowdsourcing

First,

Second

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In the rapidly growing domain called crowdsourcing [19], labels of an item provided by many workers are aggregated to render an estimation to the true label of the item. Probability 3 http://www.course.gxu.edu.cn/portal models [20]- [22], neural networks [23]- [25], weighted sum [26], [27], or majority vote [28], [29] is used to achieve the aggregation. Among these aggregation methods, majority vote is the simplest aggregation way that chooses what the majority of workers agree on as the final label to the item, and thus is error-prone when there are many spammers.…”

Section: A Opinion Aggregationmentioning

confidence: 99%

Leveraging Cognitive Diagnosis to Improve Peer Assessment in MOOCs

Liu

et al. 2021

IEEE Access

View full text Add to dashboard Cite

A major challenge faced by popular massive open online courses (MOOCs) is the assessment of large-scale open-ended assignments submitted by students. Recently, peer assessment has become a mainstream paradigm that helps grade open-ended assignments on a large scale. In peer assessment, students also become graders in grading a small number of their peers' assignments, and the peer grades are then aggregated to predict a true score for each assignment. The collected peer grades are usually inaccurate because graders have different reliabilities and biases. To improve accuracy, several probabilistic graph models have been proposed to model the reliability and bias of each grader. However, none of these models consider graders' competency information in the assignments to be graded, which has been found to be very effective. We propose two new probabilistic graph models to improve the accuracy of cardinal peer assessments based on the well-accepted cognitive diagnosis technique. Specifically, the cognitive diagnosis model DINA is applied to determine grader competency based on historical tests or assignments. Then, this information is used to optimize the modeling of grader reliability in each of the proposed models. Moreover, an effective model inference algorithm is proposed to infer true scores of assignments. Experimental results based on real world datasets show that the two proposed models outperform state-of-the-art models and that consideration of grader competency contributes to improved score estimation.

show abstract

“…Besides, some works [23][24][25][26][27] use probabilistic models for truth-value inference of crowdsourcing tasks. Li et al [23] proposed a Bayesian model (BWA) with conjugate before solving the classification problem of crowdsourcing labels, extending from discrete binary classification tasks to multiclass classification tasks, and a direct inference is performed using expectation-maximization (EM).…”

Section: Related Workmentioning

confidence: 99%

“…Li et al [23] proposed a Bayesian model (BWA) with conjugate before solving the classification problem of crowdsourcing labels, extending from discrete binary classification tasks to multiclass classification tasks, and a direct inference is performed using expectation-maximization (EM). Kurup et al [24] proposed an iterative probabilistic model-based approach for crowdsourcing task aggregation. The quality of workers was estimated using a predictive model with the expertness, reliability, and task easiness of the workers as parameters, which used true answers as latent variables.…”

Section: Related Workmentioning

confidence: 99%

Achieving Approximate Global Optimization of Truth Inference for Crowdsourcing Microtasks

Cui

Chen

et al. 2021

Data Sci. Eng.

View full text Add to dashboard Cite

Microtask crowdsourcing is a form of crowdsourcing in which work is decomposed into a set of small, self-contained tasks, which each can typically be completed in a matter of minutes. Due to the various capabilities and knowledge background of the voluntary participants on the Internet, the answers collected from the crowd are ambiguous and the final answer aggregation is challenging. In this process, the choice of quality control strategies is important for ensuring the quality of the crowdsourcing results. Previous work on answer estimation mainly used expectation–maximization (EM) approach. Unfortunately, EM provides local optimal solutions and the estimated results will be affected by the initial value. In this paper, we extend the local optimal result of EM and propose an approximate global optimal algorithm for answer aggregation of crowdsourcing microtasks with binary answers. Our algorithm is expected to improve the accuracy of real answer estimation through further likelihood maximization. First, three worker quality evaluation models are presented based on static and dynamic methods, respectively, and the local optimal results are obtained based on the maximum likelihood estimation method. Then, a dominance ordering model (DOM) is proposed according to the known worker responses and worker categories for the specified crowdsourcing task to reduce the space of potential task-response sequence while retaining the dominant sequence. Subsequently, a Cut-point neighbor detection algorithm is designed to iteratively search for the approximate global optimal estimation in a reduced space, which works on the proposed dominance ordering model (DOM). We conduct extensive experiments on both simulated and real-world datasets, and the experimental results illustrate that the proposed approach can obtain better estimation results and has higher performance than regular EM-based algorithms.

show abstract

Aggregating Unstructured Submissions for Reliable Answers in Crowdsourcing Systems

Cited by 4 publications

References 20 publications

Classifying Workers for Mitigating Adversarial Attacks in Crowdsourcing

Classifying Workers for Mitigating Adversarial Attacks in Crowdsourcing

Leveraging Cognitive Diagnosis to Improve Peer Assessment in MOOCs

Achieving Approximate Global Optimization of Truth Inference for Crowdsourcing Microtasks

Contact Info

Product

Resources

About