Expecting the Unexpected: Effects of Data Collection Design Choices on the Quality of Crowdsourced User-Generated Content

Abstract

This appendix describes our applicability check in more detail. The purpose of the applicability check (Rosemann and Vessey 2008) was to determine whether attribute data could be transformed to a form (in this case, species level classification) useful to data consumers (in this case, biologists). We also used the applicability check to explore perceptions that biologists in a university setting held about the potential uses and usefulness of data collected using an instance-based approach (versus a class-based approach). The applicability check is discussed briefly in the main manuscript; here, we provide details about the method we used to collect data, and the feedback we received from participants. Method We tested the applicability of an attribute-based data collection approach to users of UGC via an interactive seminar presentation. We made the presentation as part of a seminar series in the Department of Geography at Memorial University of Newfoundland, as geography is a field in which there is considerable interest in crowdsourced UGC (referred to by geographers as volunteered geographic information) and none of the authors are affiliated with the department. We developed a questionnaire, which was distributed in paper form, on the tables where audience members sat. The questionnaire asked six open-ended questions about the perceived benefits and limitations of both instance-based versus class-based approaches, as well as about potential applications of the instance-based approach to the respondent's own research. In addition, there were two questions asking respondents to rank (on a seven-point Likert scale) their agreement with two statements, one about the relevance and applicability of the instance-based data collection approach and the other about the relevance and applicability of the experimental findings we presented. The questionnaire also included some biographical questions (gender, position, research field, highest degree obtained).

Citizen Science: An Information Quality Research Frontier

“…For example, participants who know the objective of the project may overinflate or exaggerate information (Galloway et al 2006;Miller et al 2012). Some have explored the use of tasks which specifically do not require training (Eveleigh et al 2014;Lukyanenko et al 2019), while others have investigated the possibility of attracting diverse crowds so that training induced biases are mitigated due to the diversity of the participants (Ogunseye et al 2017;Ogunseye and Parsons 2016). All these are novel ideas for traditional information quality research.…”

“…Thus, holding data contributors to data consumer standards may curtail their ability to provide high quality content as defined by data consumers, or suggest a need to refine instructions, procedures, or expectations. Guided by this definition, a series of laboratory and fields experiments demonstrated that accuracy and completeness of citizen science data can indeed be improved by relaxing the requirements to comply with data consumer needs (Lukyanenko et al 2014a(Lukyanenko et al , 2014bLukyanenko et al 2019), which is a standard project design strategy for achieving data quality targets in citizen science. Further, for projects focusing on new, emerging phenomena, it may be challenging to anticipate the optimal structure of citizen science data due to the different needs of diverse data consumers, so traditional solutions for storage premised on a priori structures (e.g., relational databases) may be inadequate in this setting (Sheppard et al 2014).…”

Citizen Science in Information Systems Research

“…Lukyanenko et al ( 2019a , p. 7) state that despite the importance for the society and the relatedness to our discipline IS “[…] continues to lag behind such disciplines as biology and education in working with citizen science as a context for research.” Disciplines like biology, conservation, and physics are much more active here (demonstrated clearly by Lukyanenko et al 2019b ). Although there are some academic articles in IS literature on citizen science there is still a lack of citizen science projects with clear IS research questions.…”

Capturing the Forest or the Trees: Designing for Granularity in Data Crowdsourcing

“…Such a technique might be used as an automatic reconciliation system that treats every new contribution of sets of attributes as raw data and, simultaneously, as training data for an instance. A recent study, for example, demonstrates the potential of machine learning classification by classifying finegrained crowdsourced data into more useful coarsegrained data with reasonable accuracy [47]. Further explorations of how to use similar artificial intelligence tools to enhance the utility of crowdsourced data is a potent area for future research.…”