Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations

Keith, Melissa G.; Tay, Louis; Harms, Peter D.

doi:10.3389/fpsyg.2017.01359

Cited by 270 publications

(233 citation statements)

References 135 publications

Supporting

Mentioning

222

Contrasting

Order By: Relevance

“…We recruited a sample of 1,030 U.S. employed individuals using the Prolific Academic crowd‐sourcing platform for a web‐based survey (https://prolific.ac). Prolific is similar to the Amazon Mechanical Turk (MTurk) platform, which has seen widespread use in organizational research (Keith, Tay, & Harms, ). However, Prolific participants have been found to be more naïve, less dishonest, more diverse, and to provide data of similar quality compared to MTurk participants (Peer, Brandimarte, Sama, & Acquisti, ).…”

Section: Methodsmentioning

confidence: 99%

Factors affecting compliance with reference check requests

Hedricks

Rupayana

Fisher

et al. 2019

Int J Selection Assessment

View full text Add to dashboard Cite

Structured reference checks have been demonstrated to be a reliable and valid predictor of job performance. However, the reference check is a unique assessment method for personnel selection in that a third party, the reference provider, is the source of the critical information on the candidate. If a reference provider is unwilling to either partially or fully comply with a reference check request, the usefulness of the reference check is likely to be compromised. This study examined the factors affecting compliance of the potential reference provider with a reference check request. The sample consisted of 905 U.S. adults who were recruited through the Prolific online crowdsourcing platform (https://prolific.ac). We asked the participants a series of questions related to their actual experiences over the past year in responding to reference check requests. We also included a between‐subjects scenario that examined whether job candidate performance, relationship to job candidate, and method of providing the employment reference would affect hypothetical compliance. The results of the study can be used to more deeply understand the factors that are related to the compliance of the reference provider, and as a result, more fully understand the value of the reference check for selection.

show abstract

Section: Methodsmentioning

confidence: 99%

Factors affecting compliance with reference check requests

Hedricks

Rupayana

Fisher

et al. 2019

Int J Selection Assessment

View full text Add to dashboard Cite

show abstract

“…We recruited a convenience sample of gamblers by using MTurk. Some research has shown that MTurk samples are not representative of the American general population-these samples tend to be younger, more educated, and lower on the income scale (for a review, see Keith, Tay, & Harms, 2017). That said, samples drawn from MTurk are more demographically diverse and representative of the general population than student-based samples are (Buhrmester, Kwang, & Gosling, 2011).…”

Section: Limitationsmentioning

confidence: 99%

Perfectionism Predicts Disordered Gambling Via Financially Focused Self-Concept

Tabri

Werner

Milyavskaya

et al. 2018

JGI

View full text Add to dashboard Cite

Perfectionism has been implicated in several psychiatric disorders, including eating disorders, anxiety disorders, and depression. In the current research, we extended the analysis of perfectionism to understand disordered gambling. Unlike other life domains in which people with perfectionistic tendencies can objectively control outcomes (e.g., dieting to control one's body shape or weight in eating disorders), perfectionism in the gambling context is unique because there is little to no objective control over gambling outcomes (i.e., winning money). We hypothesized that gamblers with perfectionistic tendencies may set themselves a high standard within the financial success domain, which would manifest in more severe disordered gambling symptoms. We also hypothesized that having a self-concept that is focused on financial success would mediate the relation between perfectionistic tendencies and disordered gambling severity. To test this mediation model, we asked a community sample of gamblers (n = 258) to complete measures that assessed perfectionistic tendencies, financially focused self-concept, and disordered gambling severity. In line with expectations, there was a moderate positive relation between perfectionistic tendencies and disordered gambling severity, which was further mediated by financially focused self-concept. These findings suggest that perfectionistic tendencies among gamblers are associated with disordered gambling because such tendencies result in a self-concept that is focused on financial success. The findings also suggest that targeting gamblers' perfectionistic tendencies in prevention and treatment interventions may be instrumental in alleviating their financial focus, which would help curtail the development and maintenance of disordered gambling.Keywords: perfectionism, self-concept, disordered gambling, money RésuméOn a établi que le perfectionnisme entrait en ligne de compte dans plusieurs troubles psychiatriques, entre autres les troubles de l'alimentation, les troubles anxieux et la

show abstract

“…If prescreening is not possible, but genuine eligibility can be ensured using data managed by the online platform being utilized, it should be considered worthwhile, even at extra cost. Also, even though the current study focused on the possible harms of explicitly listing eligibility criteria, other complementary steps should be taken to maximize the validity of data collected on MTurk (e.g., attention checks; see Keith et al, for a discussion).…”

Section: Discussionmentioning

confidence: 99%

“…The popularity of using Amazon’s Mechanical Turk (MTurk) for data collection continues to grow within academic communities (e.g., Bowen, Daniel, Williams, & Baird, ; Chan & Holosko, ; Goodman, Cryder, & Cheema, ; Gosling & Mason, ; Mason & Suri, ; see Sheehan, for a review). Accordingly, there have been numerous reviews of MTurk, its utility, and ways of maximizing data integrity (e.g., Jia, Steelman, & Reich, ; Keith, Tay, & Harms, ). These reviews note several benefits, including the often‐described utility of allowing scholars to access hard to reach populations (Smith, Sabat, Martinez, Weaver, & Xu, ; see Chandler & Paolacci, for a list of various examples).…”

Section: Introductionmentioning

confidence: 99%

A conceptual replication examining the risk of overtly listing eligibility criteria on Amazon’s Mechanical Turk

Siegel

Navarro

2019

J Applied Social Pyschol

View full text Add to dashboard Cite

Recent scholarship indicates that explicitly listing eligibility requirements on Amazon’s Mechanical Turk can lead to eligibility falsification. Offering a conceptual replication of prior studies, we assessed the prevalence of eligibility falsification and its impact on data integrity. A screener survey collected the summer before the 2016 presidential election assessed political affiliation. Participants were then randomly assigned to be exposed to a second survey link for which they were eligible or ineligible. There was a significant interaction such that the differences between self‐reported Republicans and Democrats on outcome measures (e.g., attitudes toward Hillary Clinton), were smaller among participants that were falsifying eligibility (i.e., imposters) than those that were not (i.e., genuine participants). Moreover, for most outcomes, imposters put forth responses that were significantly different from the responses put forth by those in the political party with which imposters were pretending to be affiliated. Imposters’ responses were also significantly different from participants in the political party with which imposters initially claimed to genuinely belong. For example, those who self‐reported themselves as Democrats on the screener survey but responded to a survey for “only Republicans” (i.e., imposter Republicans), reported more favorable attitudes toward Donald Trump than genuine Democrats, but indicated less favorable attitudes toward Donald Trump than genuine Republicans. These results highlight the potential harms of explicitly listing eligibility requirements and emphasize the importance of minimizing imposter participation.

show abstract

Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations

Cited by 270 publications

References 135 publications

Factors affecting compliance with reference check requests

Factors affecting compliance with reference check requests

Perfectionism Predicts Disordered Gambling Via Financially Focused Self-Concept

A conceptual replication examining the risk of overtly listing eligibility criteria on Amazon’s Mechanical Turk

Contact Info

Product

Resources

About