Amazon Mechanical Turk for Industrial and Organizational Psychology: Advantages, Challenges, and Practical Recommendations

Woo, Sang Eun; Keith, Melissa G.; Thornton, Meghan A.

doi:10.1017/iop.2015.21

Cited by 67 publications

(45 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Taken together, our conclusions echo the conclusions of other researchers (cf., Behrend et al, 2011 ; Buhrmester et al, 2011 ; Berinsky et al, 2012 ; Casler et al, 2013 ; Brandon et al, 2014 ; Woo et al, 2015 ): MTurk samples are not representations of the general population. We should carefully consider whether representativeness of national data should be the criterion, as organizational researchers have traditionally collected data from class settings (e.g., MBA or undergraduate), specific organizations/communities, and even other online panel data from survey companies where participant solicitation processes are not transparent.…”

Section: A Systems Perspective For Evaluating Mechanical Turkmentioning

confidence: 99%

Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations

2017

View full text Add to dashboard Cite

Amazon Mechanical Turk (MTurk) is becoming a prevalent source of quick and cost effective data for organizational research, but there are questions about the appropriateness of the platform for organizational research. To answer these questions, we conducted an integrative review based on 75 papers evaluating the MTurk platform and 250 MTurk samples used in organizational research. This integrative review provides four contributions: (1) we analyze the trends associated with the use of MTurk samples in organizational research; (2) we develop a systems perspective (recruitment system, selection system, and work management system) to synthesize and organize the key factors influencing data collected on MTurk that may affect generalizability and data quality; (3) within each factor, we also use available MTurk samples from the organizational literature to analyze key issues (e.g., sample characteristics, use of attention checks, payment); and (4) based on our review, we provide specific recommendations and a checklist for data reporting in order to improve data transparency and enable further research on this issue.

show abstract

Section: A Systems Perspective For Evaluating Mechanical Turkmentioning

confidence: 99%

Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations

2017

View full text Add to dashboard Cite

show abstract

“…Studies can be conducted simultaneously in a large number of languages and cultures, a strategy not scalable in other settings. MTurk enables researchers to apply multiple levels of screeners in order to target specific respondents, including on geography, demographics, and language-an approach that is simply not possible in other research settings (Rouse, 2015;Steelman et al, 2014;Woo et al, 2015). Generally, no.…”

Section: The Convenience Critique: the Myth Of The Organizational Stumentioning

confidence: 99%

“Cargo Cult” science in traditional organization and information systems survey research: A case for using nontraditional methods of data collection, including Mechanical Turk and online panels

Lowry

D’Arcy

Hammer

et al. 2016

The Journal of Strategic Information Systems

230

113

View full text Add to dashboard Cite

show abstract

“…Nonetheless, the results of the current study and prior studies offer further evidence that this is a problem faced by scholars who collect data through MTurk. As noted by Woo, Keith, and Thornton (), “…researchers should avoid using MTurk when the participant anonymity creates a strong possibility for dishonest responses or when the entire study would be made invalid if the participants’ self‐reported identities were false…” (p. 176). The problem of imposters should not be interpreted as a reason to avoid MTurk, as there are many advantages to its use (e.g., diverse samples; Krupnikov & Levine, ).…”

Section: Discussionmentioning

confidence: 99%

A conceptual replication examining the risk of overtly listing eligibility criteria on Amazon’s Mechanical Turk

Siegel

Navarro

2019

J Applied Social Pyschol

View full text Add to dashboard Cite

Recent scholarship indicates that explicitly listing eligibility requirements on Amazon’s Mechanical Turk can lead to eligibility falsification. Offering a conceptual replication of prior studies, we assessed the prevalence of eligibility falsification and its impact on data integrity. A screener survey collected the summer before the 2016 presidential election assessed political affiliation. Participants were then randomly assigned to be exposed to a second survey link for which they were eligible or ineligible. There was a significant interaction such that the differences between self‐reported Republicans and Democrats on outcome measures (e.g., attitudes toward Hillary Clinton), were smaller among participants that were falsifying eligibility (i.e., imposters) than those that were not (i.e., genuine participants). Moreover, for most outcomes, imposters put forth responses that were significantly different from the responses put forth by those in the political party with which imposters were pretending to be affiliated. Imposters’ responses were also significantly different from participants in the political party with which imposters initially claimed to genuinely belong. For example, those who self‐reported themselves as Democrats on the screener survey but responded to a survey for “only Republicans” (i.e., imposter Republicans), reported more favorable attitudes toward Donald Trump than genuine Democrats, but indicated less favorable attitudes toward Donald Trump than genuine Republicans. These results highlight the potential harms of explicitly listing eligibility requirements and emphasize the importance of minimizing imposter participation.

show abstract

Amazon Mechanical Turk for Industrial and Organizational Psychology: Advantages, Challenges, and Practical Recommendations

Cited by 67 publications

References 24 publications

Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations

Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations

“Cargo Cult” science in traditional organization and information systems survey research: A case for using nontraditional methods of data collection, including Mechanical Turk and online panels

A conceptual replication examining the risk of overtly listing eligibility criteria on Amazon’s Mechanical Turk

Contact Info

Product

Resources

About