The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk

Litman, Leib; Robinson, Jonathan; Rosenzweig, Cheskie

doi:10.3758/s13428-014-0483-x

Cited by 239 publications

(209 citation statements)

References 22 publications

Supporting

Mentioning

200

Contrasting

Order By: Relevance

“…Mason and Watts [29] found that paying workers more increased the number of tasks workers chose to complete, but did not increase performance on each task. That increased pay did not increase performance was observed by other authors as well [4,10,25,30]. In this paper, we found that paying workers more can actually increase their performance for some types of tasks.…”

Section: Introductionsupporting

confidence: 74%

Incentivizing high quality crowdwork

Slivkins

Suri

et al. 2016

SIGecom Exch.

View full text Add to dashboard Cite

We study the causal effects of financial incentives on the quality of crowdwork. We focus on performance-based payments (PBPs), bonus payments awarded to workers for producing high quality work. We design and run randomized behavioral experiments on the popular crowdsourcing platform Amazon Mechanical Turk with the goal of understanding when, where, and why PBPs help, identifying properties of the payment, payment structure, and the task itself that make them most effective. We provide examples of tasks for which PBPs do improve quality. For such tasks, the effectiveness of PBPs is not too sensitive to the threshold for quality required to receive the bonus, while the magnitude of the bonus must be large enough to make the reward salient. We also present examples of tasks for which PBPs do not improve quality. Our results suggest that for PBPs to improve quality, the task must be effort-responsive: the task must allow workers to produce higher quality work by exerting more effort. We also give a simple method to determine if a task is effort-responsive a priori. Furthermore, our experiments suggest that all payments on Mechanical Turk are, to some degree, implicitly performance-based in that workers believe their work may be rejected if their performance is sufficiently poor. Finally, we propose a new model of worker behavior that extends the standard principal-agent model from economics to include a worker's subjective beliefs about his likelihood of being paid, and show that the predictions of this model are in line with our experimental findings. This model may be useful as a foundation for theoretical studies of incentives in crowdsourcing markets.

show abstract

Section: Introductionsupporting

confidence: 74%

Incentivizing high quality crowdwork

Slivkins

Suri

et al. 2016

SIGecom Exch.

View full text Add to dashboard Cite

show abstract

“…When they entered the code into a textbox on the MTurk website, they were paid 20 cents. Litman, Robinson, and Rosenzweig (2015) showed that compensation in this range is sufficient to ensure high quality data in both the USA and India.…”

Section: Study 1 Methodsmentioning

confidence: 99%

Is sadness blue? The problem of using figurative language for emotions on psychological tests

2016

View full text Add to dashboard Cite

Psychological tests sometimes include figurative language like I feel blue. However, figurative language may not mean the same thing cross-culturally. Previous research found cross-cultural evidence for 14 conceptual metaphors and metonymies for emotions (e.g., sadness is blue). Our two studies asked participants (total n = 795) in the USA and India whether happiness, sadness, anger, and fear are associated with certain descriptors (blue, down, bright, etc.). Most participants in both countries endorsed ten of the 14 hypothesized associations; however, the percentage of participants endorsing an association was often far from 100 %. For example, in the USA, only 71.7 % associated hot with anger and only 65.9 % associated blue with sadness. Moreover, descriptors were often associated with more than one emotion. Furthermore, only two associations (happiness is up and bright) were endorsed by more than 90 % of participants in both countries and had descriptors that were not associated with additional emotions. We conclude that figurative language is often ambiguous and should be used with caution on psychological tests unless there is evidence the language is understood cross-culturally. Advice to this effect is currently lacking from psychometrics textbooks and should be added.

show abstract

“…In India, 65% of turkers are male, the average age is 30.5 years, and 81% have a college education (19). Making money is the top motivation for using Mechanical Turk, ahead of other factors such as enjoyment and killing time (20). Evidence regarding the influence of compensation rates is conflicting; early work suggested that low compensation rates (on average $1.60/h) did not affect the quality of completed tasks.…”

Section: Introductionmentioning

confidence: 99%

“…Evidence regarding the influence of compensation rates is conflicting; early work suggested that low compensation rates (on average $1.60/h) did not affect the quality of completed tasks. However, a recent study found that although compensation did not influence quality for US turkers, turkers from India produced higher quality data for higher compensation (20). Turkers have been used in health-related studies and can be useful in research given their low pricing and speed of service (21).…”

Section: Introductionmentioning

confidence: 99%

Diabetes Topics Associated With Engagement on Twitter

Harris¹,

Mart²,

Moreland-Russell³

et al. 2015

Prev. Chronic Dis.

View full text Add to dashboard Cite

IntroductionSocial media are widely used by the general public and by public health and health care professionals. Emerging evidence suggests engagement with public health information on social media may influence health behavior. However, the volume of data accumulating daily on Twitter and other social media is a challenge for researchers with limited resources to further examine how social media influence health. To address this challenge, we used crowdsourcing to facilitate the examination of topics associated with engagement with diabetes information on Twitter.MethodsWe took a random sample of 100 tweets that included the hashtag “#diabetes” from each day during a constructed week in May and June 2014. Crowdsourcing through Amazon’s Mechanical Turk platform was used to classify tweets into 9 topic categories and their senders into 3 Twitter user categories. Descriptive statistics and Tweedie regression were used to identify tweet and Twitter user characteristics associated with 2 measures of engagement, “favoriting” and “retweeting.”ResultsClassification was reliable for tweet topics and Twitter user type. The most common tweet topics were medical and nonmedical resources for diabetes. Tweets that included information about diabetes-related health problems were positively and significantly associated with engagement. Tweets about diabetes prevalence, nonmedical resources for diabetes, and jokes or sarcasm about diabetes were significantly negatively associated with engagement.ConclusionCrowdsourcing is a reliable, quick, and economical option for classifying tweets. Public health practitioners aiming to engage constituents around diabetes may want to focus on topics positively associated with engagement.

show abstract

The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk

Cited by 239 publications

References 22 publications

Incentivizing high quality crowdwork

Incentivizing high quality crowdwork

Is sadness blue? The problem of using figurative language for emotions on psychological tests

Diabetes Topics Associated With Engagement on Twitter

Contact Info

Product

Resources

About