Crowdsourcing as a Tool for Research: Methodological, Fair, and Political Considerations

Abstract

Crowdsourcing platforms are powerful tools for academic researchers. Proponents claim that crowdsourcing helps researchers quickly and affordably recruit enough human subjects with diverse backgrounds to generate significant statistical power, while critics raise concerns about unreliable data quality, labor exploitation, and unequal power dynamics between researchers and workers. We examine these concerns along three dimensions: methods, fairness, and politics. We find that researchers offer vastly different compensation rates for crowdsourced tasks, and address potential concerns about data validity by using platform-specific tools and user verification methods. Additionally, workers depend upon crowdsourcing platforms for a significant portion of their income, are motivated more by fear of losing access to work than by specific compensation rates, and are frustrated by a lack of transparency and occasional unfair treatment from job requesters. Finally, we discuss critical computing scholars’ proposals to address crowdsourcing’s problems, challenges with implementing these resolutions, and potential avenues for future research.

Careless responding in crowdsourced alcohol research: A systematic review and meta-analysis of practices and prevalence.

“…Therefore, we are unable to make robust generalizations to, or comparisons with, other research fields. Finally, we were unable to directly address the issue of inauthentic users ("bots") which have been thought to proliferate crowdsourcing platforms and were first identified as an issue in research studies circa 2018 (Rea et al, 2020). Many platforms block repeat internet protocol (IP) addresses in attempts to circumvent this issue; however, this can be overcome with Virtual Private Networks.…”

Representation and Partisanship: What Determines the Topics That Members of Parliament Prioritize in Communications with Their Constituents?

“…We are confident that employing these multiple attempts over 11 combined months of fieldwork exhausted all reasonable avenues of collecting these documents 1 . Future research should consider crowdsourcing as a method of obtaining similar documents (Rea et al, 2020).…”

How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets

“…Further, through breaking tasks down, and dispersing their completion, crowd work has enabled some firms to move away from reliance on in-house employment (Berg 2015), exacerbating the ethical considerations surrounding such modes of employment (Vallas and Schor 2020). These considerations are compounded by the power dynamics that frame this work, with task requesters able to reject and refuse payment for any tasks not completed to their satisfaction (Rea et al 2020;Alkhatib, Bernstein, and Levi 2017). Qualitative research has been critical in highlighting these dynamics, particularly in drawing attention to how platforms such as MTurk "rely not only on the calculative mechanisms of control that metrics afford but also on normative mechanisms in the form of [...] inducements that strengthen user attachment" (Vallas and Schor 2020)(p. 279).…”