MTurk participants have substantially lower evaluative subjective well-being than other survey participants

Abstract

Amazon's MTurk platform has become a popular site for obtaining relatively inexpensive and convenient adult samples for use in behavioral research. Concerns have been raised about selection issues, because MTurk workers chose to participate in the platform and select the tasks they perform (of many offered to them). Prior studies have documented demographic and psychological differences with national samples. In this paper we studied evaluative subjective well-being (the Cantril Ladder) in an MTurk sample, a national Internet panel sample, and a national telephone survey conducted by Gallup-Sharecare. A surprising finding was that MTurk participants' Ladder scores were substantial lower than the other two samples. Analyses controlling for six demographic differences among the samples only slightly reduced the mean differences. However, patterns of demographic-well-being associations were similar within the samples. To corroborate these results, we conducted a secondary analysis on another three samples, one MTurk sample and two Internet panel samples. The same group differences in Ladder scores were observed. These findings add to the growing literature documenting the characteristics of MTurk samples and we discuss the implications for future research with such samples.

Data Collection via Online Platforms: Challenges and Recommendations for Future Research

“…A central concern that arises with engaging participants through online platforms, therefore, is whether the researchers (and the platform itself) compensate participants ethically (Bohannon, 2016; Lovett et al, 2018; Pittman & Sheehan, 2016). This is especially a concern in light of research that has shown that participants recruited through online platforms tend to have lower socio‐economic status, lower levels of well‐being (Stone, Walentynowicz, Schneider, Junghaenel, & Wen, 2019), and higher rates of clinical depression (Ophir, Sisso, Asterhan, Tikochinski, & Reichart, 2020) than the general population.…”

Robo-Tipping: Are Customers Game?

“…A convenience sample of participants who were located in the United States, 21 years of age or older and who had patronized a bar was recruited using Amazon's Mechanical Turk (MTurk) platform. To increase the quality of responses, participants were also required to have an approval rate of previous requests of 95% or above [16]. In addition to that, the survey contained a combination of multiple-choice and openended questions.…”

A meta-analysis of financial self-control strategies: Comparing empirical findings with online media and lay person perspectives on what helps individuals curb spending and start saving

“…Note that this sample of lay person reports was a convenience sample recruited through MTurk. Thus our findings are based on a specific subset of the population (younger, more educated, and more white; [ 73 ]) and may not generalize to other populations. Also, individual differences such as age might play a role in strategy effectiveness (e.g., older people might prefer paying by cash more than younger people).…”