Crowdsourcing in health and medical research: a systematic review

Abstract

Background: Crowdsourcing is used increasingly in health and medical research. Crowdsourcing is the process of aggregating crowd wisdom to solve a problem. The purpose of this systematic review is to summarize quantitative evidence on crowdsourcing to improve health. Methods: We followed Cochrane systematic review guidance and systematically searched seven databases up to September 4th 2019. Studies were included if they reported on crowdsourcing and related to health or medicine. Studies were excluded if recruitment was the only use of crowdsourcing. We determined the level of evidence associated with review findings using the GRADE approach. Results: We screened 3508 citations, accessed 362 articles, and included 188 studies. Ninety-six studies examined effectiveness, 127 examined feasibility, and 37 examined cost. The most common purposes were to evaluate surgical skills (17 studies), to create sexual health messages (seven studies), and to provide layperson cardiopulmonary resuscitation (CPR) out-of-hospital (six studies). Seventeen observational studies used crowdsourcing to evaluate surgical skills, finding that crowdsourcing evaluation was as effective as expert evaluation (low quality). Four studies used a challenge contest to solicit human immunodeficiency virus (HIV) testing promotion materials and increase HIV testing rates (moderate quality), and two of the four studies found this approach saved money. Three studies suggested that an interactive technology system increased rates of layperson initiated CPR out-ofhospital (moderate quality). However, studies analyzing crowdsourcing to evaluate surgical skills and laypersoninitiated CPR were only from high-income countries. Five studies examined crowdsourcing to inform artificial intelligence projects, most often related to annotation of medical data. Crowdsourcing was evaluated using different outcomes, limiting the extent to which studies could be pooled. Conclusions: Crowdsourcing has been used to improve health in many settings. Although crowdsourcing is effective at improving behavioral outcomes, more research is needed to understand effects on clinical outcomes and costs. More research is needed on crowdsourcing as a tool to develop artificial intelligence systems in medicine.

Crowdsourcing in medical research: concepts and applications

“…Our observations about using crowdsourcing in medical research have several important limitations. First, we did not focus our analysis based on different categories of crowdsourcing because other systematic reviews have covered this territory (Crequit et al, 2018; Wang et al, 2018a). Second, although there is a growing literature on crowdsourcing in medical research, (Pan et al, 2017) the number of randomized controlled trials and related studies is still limited (Wang et al, 2018a).…”

Peer Review #1 of "Crowdsourcing in medical research: concepts and applications (v0.1)"

“…First, we did not focus our analysis based on different categories of crowdsourcing because other systematic reviews have covered this territory. (Crequit et al 2018;Wang et al 2018) Second, although there is a growing literature on crowdsourcing in medical research, (Pan et al 2017) the number of randomized controlled trials and related studies is still limited. (Wang et al 2018) Third, we have not included a list of areas which problems may be more amenable to crowdsourcing because this has been partially covered in a previous review (Wazny 2017) and is difficult to infer from the existing literature.…”

Crowdsourcing to identify social innovation initiatives in health in low- and middle-income countries

“…The value of crowdsourcing as an approach is that aligns with the bottom-up, participatory and inclusive nature of SI. It has been well described in the literature and its potential contribution to public health has been shared [8,15,19,40]. However, its application has been limited mainly in HICs settings with limited application in LMIC settings.…”

“…The study involved a descriptive analysis of the crowdsourcing process conducted by the authors in several LMICs, making a case for using crowdsourcing as a novel approach to identify SI and other health initiatives. However, while the crowdsourcing process described in this paper offers detailed guidance on its implementation, there are debates regarding the advantages of different methodological steps [40], and it may not be generalizable to all LMICs contexts. Moreover, the study was retrospective, which may have introduced a degree of recall bias.…”