Foundations of Crowd Data Sourcing

Abstract

Crowdsourcing techniques are very powerful when harnessed for the purpose of collecting and managing data. In order to provide sound scientific foundations for crowdsourcing and support the development of efficient crowdsourcing processes, adequate formal models must be defined. In particular, the models must formalize unique characteristics of crowd-based settings, such as the knowledge of the crowd and crowd-provided data; the interaction with crowd members; the inherent inaccuracies and disagreements in crowd answers; and evaluation metrics that capture the cost and effort of the crowd. In this paper, we review the foundational challenges in modeling crowd-based data sourcing, for its two main tasks, namely, harvesting data and processing it with the help of the crowd. For each of the two task types, we dive into the details of one foundational line of work, analyzing its model and reviewing the theoretical results established using this model, such as complexity bounds and efficient algorithms. We also overview a broader spectrum of work on crowd data sourcing, and highlight directions for further research.

A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective

“…A recent survey [88] provides an extensive discussion on the challenges for data crowdsourcing. Another survey [89] touches on the theoretical foundations of data crowdsourcing. According to both surveys, data generation using crowdsourcing can be divided into two steps: gathering data and preprocessing data.…”

Crowd Science: Measurements, Models, and Methods

“…From this research beginning, the body of Crowdsourcing research has expanded rapidly, supplying numerous taxonomies, typologies, and findings related to the various processes, attributes, and outcomes of engaging IT-mediated crowds, including some of the following themes; task complexity [25], crowdsourcing models [19], crowdsourcing processes [18], crowd ability [26], solution quality [27], crowdsourced data [28] and data processing [29], enterprise crowdsourcing [30], crowdsourcing as a lens for human-computer interaction [31], organizational crowdsourcing intentions [23,32], value creation [33], crowdsourcing multiple tasks [34], crowdsourcing for behavioral science purposes [35], crowdsourcing and algorithms [36], crowdsourcing for innovation [37], crowdsourcing labor law [38], crowdsourcing workers with disabilities [39], health care crowdsourcing [38], crowdsourcing for policy assessment [40], the geography of crowdsourcing participation [62], and cultural factors in crowdsourcing [41]. The universal characteristics of all Crowdsourcing applications listed above are useful and important, since they allow researchers and practitioners alike to understand the stable, relative differences between the forms of Crowdsourcing, while vividly displaying the inherent trade-offs that organizations face when considering Crowdsourcing initiatives.…”

Introduction to Crowd Science Minitrack

“…In the purview of this mini-track, IT-mediated crowd phenomena can be found in these areas of research; Crowdsourcing [11-14, 18, 64], Crowd Finance (Crowdfunding, Blockchains, Distributed Ledgers) [8,21,50], Prediction Markets [6,23], Citizen Science [17,71], Open Innovation & Tournament platforms [5,9,15,16,27,53], Social Media for resource creation [30][31][32], Wikis & Wikipedia [39,40,72,75], Big Data from Crowds [3], Spatial Crowdsourcing (Sharing/Gig Economy) [57], Situated/IoT Crowdsourcing [57], Wearables Crowdsourcing [57], IT-mediated Collective Intelligence [37,42,57] We encourage new empirical and theoretical submissions from social, economic, technical and organizational scholars, investigating these phenomena in a variety of contexts, including: Health Care [49,52], law [74], Education [4,19,22,38,47,48,54,62]  IT-mediated crowds and law/intellectual property [74].…”