Quality management on Amazon Mechanical Turk

Abstract

Crowdsourcing services, such as Amazon Mechanical Turk, allow for easy distribution of small tasks to a large number of workers. Unfortunately, since manually verifying the quality of the submitted results is hard, malicious workers often take advantage of the verification difficulty and submit answers of low quality. Currently, most requesters rely on redundancy to identify the correct answers. However, redundancy is not a panacea. Massive redundancy is expensive, increasing significantly the cost of crowdsourced solutions. Therefore, we need techniques that will accurately estimate the quality of the workers, allowing for the rejection and blocking of the low-performing workers and spammers.However, existing techniques cannot separate the true (unrecoverable) error rate from the (recoverable) biases that some workers exhibit. This lack of separation leads to incorrect assessments of a worker's quality. We present algorithms that improve the existing state-of-the-art techniques, enabling the separation of bias and error. Our algorithm generates a scalar score representing the inherent quality of each worker. We illustrate how to incorporate cost-sensitive classification errors in the overall framework and how to seamlessly integrate unsupervised and supervised techniques for inferring the quality of the workers. We present experimental results demonstrating the performance of the proposed algorithm under a variety of settings.

Cooperation and contagion in web-based, networked public goods experiments

“…In addition to its role as a labor market, however, AMT can also be thought of as a convenient pool of subjects willing to participate in laboratory-style behavioral experiments. Mechanical Turk and other web-based experimental platforms are becoming increasingly popular with behavioral science researchers, in part because they allow experiments to be run faster and more cheaply, and in part because they afford access to potentially a much broader cross-section of the population than is typical of university-based lab experiments [34][35][36][37][38]. A second contribution of this work, therefore, is to advance the scope of behavioral experiments conducted on AMT to include networked games and more generally, games where all players play simultaneously.…”

Ontology based search result optimisation using singular matrix

“…Our inspiring frame work is a disaster management situation, inspired by the experience in building business continuity information network [3] for disaster situations in India. During calamities, we have many users and concerns proclaiming and investigating information.…”

Constructing Context-Aware Sentiment Lexicons with an Asynchronous Game with a Purpose

“…During the task, output agreement setups place workers in teams and require them to agree on their answers [10]. Posterior measures are applied after the work is done, by filtering or aggregating data to remove answers that are irrelevant or malicious [22]. Preemptive measures are applied before collecting answers, by making workers aware of the desired level of performance.…”