The original sin of crowd work for human subjects research

Abstract

Purpose Academic scholars have leveraged crowd work platforms such as Amazon Mechanical Turk for human subjects research for almost two decades. However, few scholars have reflected or questioned this mode of academic research. This paper aims to examine three fundamental problems of crowd work and elaborates on their lasting effects on impacting the validity and quality of human subjects research on crowd work. Design/methodology/approach` A critical analysis is conducted on the characteristics of crowd work, and three fundamental problems of crowd work since its origin were identified, namely, the position of “Human-as-a-service,” the confusion of terminology and crowd work platforms’ abdication of responsibilities. Findings This paper explains that the three identified fundamental problems of crowd work render at least two lasting problems in crowd work-based research: first, the negligence of the teleological difference between crowd work and academic research; second, the ontological schism between scholars and institutional review boards (IRBs) in their ethical concerns and practices. Originality/value This paper critiques the foundation of crowd work-based research that has become growingly popular, extolled and taken for granted. Such a critique is deficient in literature and may seem a bit peculiar. However, we hold that it is time to take research ethics seriously in crowd work because we need to introspect and question ourselves as scholars: What is our motive or ethical stance in using crowd work for human subjects research? Is it for advancing scientific knowledge, promoting crowd workers’ welfare, or predominantly for benefiting ourselves from the fast, cheap and “good” data via crowd work?

What scholars and IRBs talk when they talk about the Belmont principles in crowd work‐based research

“…First, there is a dilemma between the dehumanization nature of crowd work and the expected autonomy in crowd work‐based research. On the one hand, crowd work platforms like MTurk treat crowd workers as a commodity or API call, which is regarded as the origin of various ethical issues in crowd work‐based research (Xia, 2022a). On the other, IRBs and scholars expect crowd workers to be autonomous human beings.…”

“…However, the Belmont principles may face particular challenges in crowd work‐based research. Crowd work‐based research depends on crowd workers for data collection, while crowd workers have been dehumanized as a mere service and data reservoir since MTurk's launch with its slogan “Human‐as‐a‐Service” (Bezos, 2006; Gray & Suri, 2019; Xia, 2022a). This fact triggers two contrasting approaches to the Belmont principle of RFP among scholars.…”

“…The other group of scholars, primarily those using crowd work for data annotation, seem to ignore the Belmont principles altogether and avoid the IRB's approval because they do not regard crowd work‐based research as human subjects research (Ramírez et al, 2021; Shmueli et al, 2021). Meanwhile, crowd workers are unlike volunteers in conventional sociological surveys since they are primarily motivated by money, not research (Crowston, 2012; Litman et al, 2015; Xia, 2022a). Hence, whether the Belmont principle of beneficence equates to a minimum wage payment to crowd workers is questionable.…”

“…Third, how to position crowd workers in academic research is still an open question. Treating them as wage‐earners may require paying them a minimum wage for respect, but treating them as citizen scientists or co‐researchers would not require such specific payment criteria since many citizen science projects, such as Zooniverse, are without any payment (Bassi et al, 2020; Xia, 2022a).…”