Enhancing Big Data in the Social Sciences with Crowdsourcing: Data Augmentation Practices, Techniques, and Opportunities

Abstract

The importance of big data is a contested topic among social scientists. Proponents claim it will fuel a research revolution, but skeptics challenge it as unreliably measured and decontextualized, with limited utility for accurately answering social science research questions. We argue that social scientists need effective tools to quantify big data's measurement error and expand the contextual information associated with it. Standard research efforts in many fields already pursue these goals through data augmentation, the systematic assessment of measurement against known quantities and expansion of extant data by adding new information. Traditionally, these tasks are accomplished using trained research assistants or specialized algorithms. However, such approaches may not be scalable to big data or appease its skeptics. We consider a third alternative that may increase the validity and value of big data: data augmentation with online crowdsourcing. We present three empirical cases to illustrate the strengths and limits of crowdsourcing for academic research, with a particular eye to how they can be applied to data augmentation tasks that will accelerate acceptance of big data among social scientists. The cases use Amazon Mechanical Turk to (1) verify automated coding of the academic discipline of dissertation committee members, (2) link online product pages to a book database, and (3) gather data on mental health resources at colleges. In light of these cases, we consider the costs and benefits of augmenting big data with crowdsourcing marketplaces and provide guidelines on best practices. We also offer a standardized reporting template that will enhance reproducibility. Enhancing Big Data in the Social Sciences with Crowdsourcing2

The Acoustic Dissection of Cough: Diving into Machine Listening-based COVID-19 Analysis and Detection

“…In Limitations. The classification performance reported in our study needs to be interpreted in the light of the well-known challenges of data collection via crowdsourcing, including data validity, data quality, and participant selection bias (Afshinnekoo et al, 2016;Khare et al, 2015;Porter et al, 2020). The COUGHVID database does not allow to verify the COVID-19 status of the participants, as the participants were not asked to provide a copy or confirmation of their positive or negative COVID-19 test.…”

Audit Studies: Behind the Scenes with Theory, Method, and Nuance

“…The deception of the audit study may allow us to document discrimination but a similar scenario presented as a survey experiment may allow us to explore potential mechanisms with the right questions. Moreover, the rise of Amazon's Mechanical Turk (MTurk) makes collecting survey experiment data relatively quick and cheap (Campbell and Gaddis forthcoming;Porter, Verdery, and Gaddis 2017). In ongoing work combing an audit with a survey experiment, I find that roommate discrimination against many different racial and ethnic groups is driven by issues of cultural fit.…”

How many people microwork in France? Estimating the size of a new labor force

“…These tasks are often part of data-intensive processes and lengthy supply chains, feeding activities as varied as digitization of archives, market research, management of back-office operations and most importantly, development of artificial intelligence [Gray and Suri 2017]. Recent applications of machine learning -on which smart technologies, autonomous vehicles and virtual assistants are all based -rely on the creation and maintenance of large databases that need to be annotated, refined, labeled and more generally, augmented [Porter et al 2017]. Microwork is used, first, to prepare, categorize and qualify information for automatic learning algorithms; and second, to assess their performance and if necessary, to make corrections.…”