Incentivizing high quality crowdwork

Abstract

We study the causal effects of financial incentives on the quality of crowdwork. We focus on performance-based payments (PBPs), bonus payments awarded to workers for producing high quality work. We design and run randomized behavioral experiments on the popular crowdsourcing platform Amazon Mechanical Turk with the goal of understanding when, where, and why PBPs help, identifying properties of the payment, payment structure, and the task itself that make them most effective. We provide examples of tasks for which PBPs do improve quality. For such tasks, the effectiveness of PBPs is not too sensitive to the threshold for quality required to receive the bonus, while the magnitude of the bonus must be large enough to make the reward salient. We also present examples of tasks for which PBPs do not improve quality. Our results suggest that for PBPs to improve quality, the task must be effort-responsive: the task must allow workers to produce higher quality work by exerting more effort. We also give a simple method to determine if a task is effort-responsive a priori. Furthermore, our experiments suggest that all payments on Mechanical Turk are, to some degree, implicitly performance-based in that workers believe their work may be rejected if their performance is sufficiently poor. Finally, we propose a new model of worker behavior that extends the standard principal-agent model from economics to include a worker's subjective beliefs about his likelihood of being paid, and show that the predictions of this model are in line with our experimental findings. This model may be useful as a foundation for theoretical studies of incentives in crowdsourcing markets.

Conducting perception research over the internet: a tutorial review

“…Those who have looked at whether the payment amount influences how many people are willing to take part (i.e., how many sign-up) and/or how seriously they take the experiment (discussed later in the section entitled ‘5. Random responding’) have, perhaps somewhat surprisingly, shown little relation between reward and effort; only the rate of recruitment seems to be influenced by payment size ( Paolacci & Chandler, 2014 ; Mason & Watts, 2010 ; though do see Ho et al, 2015 , who show that bonuses can sometimes improve task performance). A different picture emerges however, when you ask MTurkers themselves what motivates people to take part in low wage research.…”

Pay Rates and Subject Performance in Social Science Experiments Using Crowdsourced Online Samples

“…As Ho, Slivkins, Suri, and Vaughan describe: "even when standard, unconditional payments are used and no explicit acceptance criteria is specified, workers may behave as if the payments are implicitly performance-based since they believe their work may be rejected if its quality is sufficiently low" (Ho et al, 2015). In such scenarios, different pay rates have been demonstrated to motivate workers to do a greater quantity of work, but not at higher quality (Mason and Watts, 2009).…”

“…In such scenarios, different pay rates have been demonstrated to motivate workers to do a greater quantity of work, but not at higher quality (Mason and Watts, 2009). Similarly, several studies have shown that, when work is verifiable based upon accuracy or correctness, pay rates can influence worker behavior positively (Finnerty et al, 2013;Horton and Chilton, 2010;Ho et al, 2015;Ye et al, 2017).…”

ParkCrowd: Reliable Crowdsensing for Aggregation and Dissemination of Parking Space Information

“…Moreover, it does not justify the reward distribution on other aspects, such as data quality, expertise or efforts level of the crowd workers, because the negotiated price in the auction process is usually not directly related to the results of data collection process. Therefore, incentivisation schemes which distribute rewards based on individual contributions of the workers after the data has been collected have been proposed recently [26], [29], [35]. [26] employs an entropy based method to quantify the contributions of the workers while [29] constructs the reputation of the workers for reward distribution.…”