On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices

Abstract

Crowdsourcing is being increasingly adopted as a platform to run studies with human subjects. Running a crowdsourcing experiment involves several choices and strategies to successfully port an experimental design into an otherwise uncontrolled research environment, e.g., sampling crowd workers, mapping experimental conditions to micro-tasks, or ensure quality contributions. While several guidelines inform researchers in these choices, guidance of how and what to report from crowdsourcing experiments has been largely overlooked. If under-reported, implementation choices constitute variability sources that can affect the experiment's reproducibility and prevent a fair assessment of research outcomes. In this paper, we examine the current state of reporting of crowdsourcing experiments and offer guidance to address associated reporting issues. We start by identifying sensible implementation choices, relying on existing literature and interviews with experts, to then extensively analyze the reporting of 171 crowdsourcing experiments. Informed by this process, we propose a checklist for reporting crowdsourcing experiments.

Careless responding in crowdsourced alcohol research: A systematic review and meta-analysis of practices and prevalence.

“…In both cases above, it is impossible to distinguish whether pre-screening and eligibility checks took place but were simply not reported (and therefore studies were coded as not including these steps). Future research may benefit from clear reporting guidelines for crowdsourced studies (see Ramírez et al, 2021).…”

A Checklist to Combat Cognitive Biases in Crowdsourcing

“…Initial steps have been taken towards defining a taxonomy of relevant attributes to report on crowdsourcing studies, such as the employed crowd, the task shown to the workers, the applied quality control mechanisms, and the experimental design (Ramírez et al 2020;Ramírez et al 2021). We believe that cognitive biases are an additional factor to consider in reports on crowdsourcing studies.…”

“…Specifically for crowdsourcing annotations, Ramírez et al (2020) proposed a set of guidelines for reporting crowdsourcing experiments to better account for reproducibility purposes. Ramírez et al (2021) then followed up on this work by proposing a checklist that requesters can use to comprehensively report on their crowdsourced data sets. This body of research aligns with and facilitates current efforts towards more trustworthy artificial intelligence through better documentation (Arnold et al 2019;Stoyanovich and Howe 2019).…”

WDV: A Broad Data Verbalisation Dataset Built from Wikidata

“…The first, by Gebru et al [14], pertains to the effective documentation of machine learning datasets, supporting the transparency and reproducibility of their creation process. The second, by Ramirez et al [31], pertains to the detailing of crowdsourcing experiments to guarantee clarity and repeatability. It ensures the impact of task design, data processing, and other factors on our conclusions, as well as their validity, can be assessed.…”