In Validations We Trust? The Impact of Imperfect Human Annotations as a Gold Standard on the Quality of Validation of Automated Content Analysis

Abstract

Political communication has become one of the central arenas of innovation in the application of automated analysis approaches to ever-growing quantities of digitized texts. However, although researchers routinely and conveniently resort to certain forms of human coding to validate the results derived from automated procedures, in practice the actual "quality assurance" of such a "gold standard" often goes unchecked. Contemporary practices of validation via manual annotations are far from being acknowledged as best practices in the literature, and the reporting and interpretation of validation procedures differ greatly. We systematically assess the connection between the quality of human judgment in manual annotations and the relative performance evaluations of automated procedures against true standards by relying on large-scale Monte Carlo simulations. The results from the simulations confirm that there is a substantially greater risk of a researcher reaching an incorrect conclusion regarding the performance of automated procedures when the quality of manual annotations used for validation is not properly ensured. Our contribution should therefore be regarded as a call for the systematic application of high-quality manual validation materials in any political communication study, drawing on automated text analysis procedures.

Kitchen Hygiene in the Spotlight: How Cooking Shows Influence Viewers’ Hygiene Practices

“…The coefficient was based on the agreement between both coders’ ratings in the pretest. The value obtained for Krippendorff's alpha (0.794) confirmed that the ratings were reliable (Song et al., 2020).…”

oolong: An R package for validating automated content analysis tools

“…Dictionary-based methods such as AFINN (Nielsen, 2011) can be validated by creating a gold standard dataset (Song et al, 2020). Oolong provides a workflow for generating such gold standard dataset.…”

“…Printing the summary shows Krippendorff's Alpha, an indicator of interrater reliability. The validity metrics of a text analytic method can be tinted by poor interrater reliability of manual annotations (Song et al, 2020). It is important to ensure high interrater reliability first.…”

“…Oolong focuses on semantic validation. The package provides the "human-in-the-loop" semantic validation procedures suggested by Chang et al (2009) and Song et al (2020). The procedure proposed by Chang et al (2009) has been adopted in subsequent social science studies as the gold standard to validate topic models, e.g.…”

“…Bohr (2020), Chuang et al (2015), and Miller (2017). The procedure proposed by Song et al (2020) emphasizes both criterion validity and interrater reliability.…”