Crowdsourcing—Harnessing the Masses to Advance Health and Medicine, a Systematic Review

Abstract

OBJECTIVE:Crowdsourcing research allows investigators to engage thousands of people to provide either data or data analysis. However, prior work has not documented the use of crowdsourcing in health and medical research. We sought to systematically review the literature to describe the scope of crowdsourcing in health research and to create a taxonomy to characterize past uses of this methodology for health and medical research. DATA SOURCES: PubMed, Embase, and CINAHL through March 2013. STUDY ELIGIBILITY CRITERIA: Primary peerreviewed literature that used crowdsourcing for health research. STUDY APPRAISAL AND SYNTHESIS METHODS: Two authors independently screened studies and abstracted data, including demographics of the crowd engaged and approaches to crowdsourcing. RESULTS: Twenty-one health-related studies utilizing crowdsourcing met eligibility criteria. Four distinct types of crowdsourcing tasks were identified: problem solving, data processing, surveillance/monitoring, and surveying. These studies collectively engaged a crowd of >136,395 people, yet few studies reported demographics of the crowd. Only one (5 %) reported age, sex, and race statistics, and seven (33 %) reported at least one of these descriptors. Most reports included data on crowdsourcing logistics such as the length of crowdsourcing (n=18, 86 %) and time to complete crowdsourcing task (n=15, 71 %). All articles (n=21, 100 %) reported employing some method for validating or improving the quality of data reported from the crowd. LIMITATIONS: Gray literature not searched and only a sample of online survey articles included. CONCLUSIONS AND IMPLICATIONS OF KEY FINDINGS:Utilizing crowdsourcing can improve the quality, cost, and speed of a research project while engaging large segments of the public and creating novel science. Standardized guidelines are needed on crowdsourcing metrics that should be collected and reported to provide clarity and comparability in methods. INTRODUCTIONCrowdsourcing is an approach to accomplishing a task by opening up its completion to broad sections of the public. Innovation tournaments, prizes for solving an engineering problem, or paying online participants for categorizing images are examples of crowdsourcing. What ties these approaches together is that the task is outsourced with little restriction on who might participate. Despite the potential of crowdsourcing, little is known about the applications and feasibility of this approach for collecting or analyzing health and medical research data where the stakes are high for data quality and validity.One of the most celebrated crowdsourcing tasks was the prize established in 1714 by Britain's Parliament in the Longitude Act, offered to anyone who could solve the problem of identifying a ship's longitudinal position.1 The Audubon Society's Christmas Bird Count began in 1900 and continues to this day as a way for "citizen scientists" to provide data that can be used for studying bird population trends.2 However, today the world has 2.3 billion Internet users an...

Crowdsourcing’s ten years in: A review

“…With regards to the demographics of the crowd, Ranard et al’s review found that few articles reported on demographics and for those that did, the level of demographics reported varied [27]. However, Khare et al state that the crowd should be poorly defined and diverse [3].…”

“…In AMT and CrowdFlower, the initiator (or crowdsourcer) posts a task and the ‘crowd’ responds and are paid in small quantities for completing small HITs. Other online platforms use distributed online processes to compete for the best solution, such as InnoCentive or CrowdMed [27,32]. Advances in mHealth, such as wearable technologies and sensors, could enable real–time data collection and monitoring from mass amounts of people [38].…”

“…Possible solutions for these issues were proposed, including having multi–level reviews. Here, there are multiple stages to each crowdsourcing task and each task is reviewed multiple times and aggregated [6] having objectifiable tasks to ‘weed out’ malicious workers or having standards by which workers must fulfil prior to be considered for the task [6,27]. For example, in AMT, workers may have obtained certain scores in previous tasks.…”

“…As previous authors had reported few search results in indexed journals [10,27,39] and crowdsourcing is a new method, semantic searches in Google Scholar were used to retrieve both peer–reviewed and grey literature published on crowdsourcing. “Crowdsourcing” as well as ‘crowdsourcing’ joint with health terms, such as genetics, diagnosis, epidemiology, surveillance, public health and disease were searched in August, 2015.…”

“…Crowdsourcing has considerable benefits in research, as it has the potential to substantially lower costs while massively increasing the sample size and researchers can receive the data in real–time [7,16,19,27–29]. Because of these qualities, crowdsourcing has potential to improve global health research.…”