Content Analysis by the Crowd: Assessing the Usability of Crowdsourcing for Coding Latent Constructs

Abstract

Crowdsourcing platforms are commonly used for research in the humanities, social sciences and informatics, including the use of crowdworkers to annotate textual material or visuals. Utilizing two empirical studies, this article systematically assesses the potential of crowdcoding for less manifest contents of news texts, here focusing on political actor evaluations. Specifically, Study 1 compares the reliability and validity of crowdcoded data to that of manual content analyses; Study 2 proceeds to investigate the effects of material presentation, different types of coding instructions and answer option formats on data quality. We find that the performance of the crowd recommends crowdcoded data as a reliable and valid alternative to manually coded data, also for less manifest contents. While scale manipulations affected the results, minor modifications of the coding instructions or material presentation did not significantly influence data quality. In sum, crowdcoding appears a robust instrument to collect quantitative content data.

In Validations We Trust? The Impact of Imperfect Human Annotations as a Gold Standard on the Quality of Validation of Automated Content Analysis

“…Given that coders are treated as interchangeable, any (potentially) remaining coder idiosyncrasies (either coder-specific systematic errors or random measurement errors) are in effect no longer considered, neither in the analyses nor in the interpretations of the findings (see , for a detailed discussion on this issue). When there is a sufficiently large number of coders, or each materials are coded by multiple coders ("duplicated coding" as in some SML applications or in crowdcoding: see Lind, Gruber, & Boomgaarden, 2017;Scharkow, 2013), the impact of coder idiosyncrasiesespecially random errorswould diminish, as they will cancel each other out as long as the number of coders/ duplicated coding instances increases. Nevertheless, remaining systematic errors in coder idiosyncrasies may still introduce bias in gold standard materials with respect to the target of inference, especially for data with a higher level of intercoder reliability (i.e., a systematic deviation from the true target).…”

Does Suffering Suffice? An Experimental Assessment of Desert Retributivism

“…Crowd-coding is both hailed as a useful strategy but also viewed critically (Snow et al 2008, Benoit et al 2016, Lind et al 2017, Dreyfuss 2018. Because Krippendorf's alpha was not higher for certain categories we carried out additonal analyses to see whether our results remain robust to the exclusion of certain workers.…”

Can the online crowd match real expert judgments? How task complexity and coder location affect the validity of crowd‐coded data

“…This simple but powerful idea that good collective decisions can emanate from various averaged independent judgements of non-experts is long discussed in academia, business and popular science (see Surowiecki 2004;Lehman & Zobel 2017). Yet, notwithstanding instructive earlier studies with positive conclusions regarding the validity of crowd-coded data (e.g., Berinsky et al 2014;Haselmayer & Jenny 2016;Lind et al 2017), it seems fair to say that crowd-coding is only starting to gain traction in political science at large since Benoit et al (2016) have convincingly argued that the results of expert judgements -still considered the gold standard by many (e.g., when it comes to the location of parties) -can be matched with crowd-coding, at least for simple coding tasks. This is significant, since experts are expensive and in short supply and automated (coding) methods are not yet good enough at extracting meaning (Benoit et al 2016: 280).…”

“…; Haselmayer & Jenny ; Lind et al. ), it seems fair to say that crowd‐coding is only starting to gain traction in political science at large since Benoit et al. () have convincingly argued that the results of expert judgements – still considered the gold standard by many (e.g., when it comes to the location of parties) – can be matched with crowd‐coding, at least for simple coding tasks.…”

“…This averaged confidence can also be conceived of as inter‐coder reliability within the crowd (see Lind et al. ). Everyone can use the above formula to arrive at the confidence score for a given statement, so it is very transparent.…”