Ethical concerns arising from recruiting workers from Amazon's Mechanical Turk as research participants: Commentary on Burnette et al. (2021)

Abstract

In this commentary, we respond to Burnette et al.'s (2021) paper, which gives significant practical recommendations to improve data quality and validity while gathering data via Amazon's Mechanical Turk (MTurk). We argue that it is also important to acknowledge and review the specific ethical issues that might arise when recruiting MTurk workers as participants. We particularly raise three main ethical concerns that need to be addressed when recruiting research participants from participant recruitment platforms: participants' economic vulnerability, participants' sensitivity, and power dynamics between participants and researchers. We elaborate on these issues by discussing the ways in which they may appear and be responded to. We conclude that considering the ethical aspects of data collection and the potential impacts of data collection on those involved would complement Burnette et al.'s recommendations. Consequently, data collection processes should be transparent as well, in addition to data screening processes.

Brain-computer interfaces, disability, and the stigma of refusal: A factorial vignette study

“…It also a strength of our study that we collected samples from a platform with more demographically diverse samples as compared with standard Internet and American college samples ( Buhrmester et al, 2011 ; Clifford et al, 2015 ). Still, such samples are not representative of the population and, as a technology, AMT may pose its own ethical challenges and potential for inequity ( Gleibs and Albayrak-Aydemir, 2022 ; Heen et al, 2014 ; Ipeirotis, 2010 ). For example, we found a larger number of men participating in our experiments.…”

The Relationship of Self-identified Weight Status With Perceived Mental and Physical Health

“…We also used Amazon's Mechanical Turk website to recruit participants for the current study. There is debate over the ethics of using this site as a source of participants, including questions about the validity of data, [32][33][34][35][36] although survey data from participants recruited through Mechanical Turk do not appear notably worse than those obtained through other internet surveys. [37][38][39] To protect our data from potential invalid responses, we included 3 formal validity checks throughout the survey and examined response patterns during the data cleaning process.…”

Rationale and Study Checklist for Ethical Rejection of Participants on Crowdsourcing Research Platforms

“…The combination of poor median compensation for crowdsourced workers and the inconsistent availability of higher-paying work through research studies raises questions of economic vulnerability in understanding worker behavior. 14 In the aforementioned RCT 15 on MTurk data quality, the four specific issues that were addressed through control questions, rejection, and resampling were: (1) the use of virtual private networks (VPNs) to mimic U.S.-based internet protocol (IP) addresses, 16 (2) the use of automation to complete surveys ("bots"), 17 (3) dishonest responding, 18 and (4) inattention, 19 though other threats to data quality may also exist. In each case, framing behavior from a "work" rather than a "research" perspective demonstrates how poor-quality data might reasonably arise from attempts to maximize compensation.…”