oolong: An R package for validating  automated content analysis tools

Chan, Chung-hong; Sältzer, Marius

doi:10.21105/joss.02461

Cited by 15 publications

(5 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This was accomplished using the structural topic model package in R, and captured the top 30 topics across the data set, which we built to contextualize our findings with regard to the third research question. The topic model was validated with a word intrusion test using R's oolong package (C.-h. Chan & Sältzer, 2020) and achieved an acceptable accuracy of 80%, as all topics reported specifically in this article could be successfully identified in word intrusion test.…”

Section: Methodsmentioning

confidence: 73%

Close to Beijing: Geographic Biases in People’s Daily

Weston

Rauchfleisch

2021

MaC

View full text Add to dashboard Cite

Inequities in China are reflected within state-run media coverage due to its specific role “guiding public opinion,” and with our study we contribute to the geographic turn in the Chinese context with regard to media and journalism. As a subject of a spatial study, China is unique due to several factors: geographic diversity, authoritarian control, and centralized media. By analyzing text from 53,000 articles published in <em>People’s Daily</em> (rénmín rìbào, 人民日報) from January 2016 to August 2020, we examine how the amount of news coverage varies by region within China, how topics and sentiments manifest in different places, and how coverage varies with regard to foreign countries. Automated methods were used to detect place names from the articles and geoparse them to specific locations, combining spatial analysis, topic modeling and sentiment analysis to identify geographic biases in news coverage in an authoritarian context. We found remarkably uniform and positive coverage domestically, but substantial differences towards coverage of different foreign countries.

show abstract

Section: Methodsmentioning

confidence: 73%

Close to Beijing: Geographic Biases in People’s Daily

Weston

Rauchfleisch

2021

MaC

View full text Add to dashboard Cite

show abstract

“…Furthermore, one can also distinguish between validation steps that are universally applicable, and validation steps that are only eligible for specific types of methods (Birkenmaier et al, 2023;Grimmer & Stewart, 2013). The literature on unsupervised methods, for instance, proposes a great variety of metrics and validation steps to demonstrate the consistency of topics for specific variants of topic models (Chan & Sältzer, 2020;Chang et al, 2009;Ying et al, 2022). On the other hand, there are generic validation steps that can be universally applied to different types of CTAM.…”

Section: Generic or Method-specific Validationmentioning

confidence: 99%

“…Often, these method-specific contributions propose valuable workflows, targeted at the unique challenges of validating specific types of CTAM. Recent examples of such method-specific frameworks and guidelines target in particular unsupervised methods (Bernhard et al, 2023;Chan & Sältzer, 2020;Maier et al, 2022;Terragni et al, 2021;Ying et al, 2022) or supervised methods (see Chapter 20 on validation in Grimmer et al, 2022;Park & Montgomery, 2023). While they tend to be geared towards specific methodspecific workflows, these approaches still furnish researchers with practical and valuable guidance.…”

Section: An Outlook On More Unified Validation Frameworkmentioning

confidence: 99%

The search for solid ground in text as data: A systematic review and practical recommendations for validation

Birkenmaier¹,

Lechner²,

Wagner³

2023

Preprint

View full text Add to dashboard Cite

Nowadays, text-based methods are widely applied to detect and measure a wide range of social science constructs, such as emotions or political attitudes. However, the validity of text-based measures can be difficult to assess. This is because text data is typically unstructured, complex, and noisy. Additionally, practical and conceptual guidance on how to validate text-based measures is scarce and inconsistent. But what are the consequences of this reality for applied research? Our work aims to answer this research question by providing a fundament for the systematic engagement with validity for text-based research in the social sciences. Based on a systematic review and qualitative expert interviews, we describe and identify differences and common themes in validation practices. Our results show that scholars applied a great variety of validation steps, which, however, were rarely selected on a conceptual understanding of validity. These findings are also supported by our qualitative results as we recorded considerable confusion about what actions ultimately represent valid validation steps and uncertainty on how these steps should then be reported. Overall, we call for more systematic efforts to develop a commonly shared language of validation practices to effectively document and evaluate validation steps for text as data methods.

show abstract

“…[6][7][8][9][10][11] for details of the model robustness checks. We validated our model choice with the use of manual topic intrusion validation using the R package oolong 90 . We created a Shiny app with an intrusion test which was completed by 6 third-party testers who were unaware of the aims of the research models.…”

Section: Latent Dirichlet Analysis Topic Modelmentioning

confidence: 99%

Science governs the future of the mesopelagic zone

et al. 2023

View full text Add to dashboard Cite

The potential of the mesopelagic zone (200–1000 m depth) to provide natural resources and ecosystem services is of increasing interest to a broad range of societal stakeholders. As this interest grows, divergent ideas about its current and future role in supporting human life are being expressed in scientific and public discourse. We use automated content analysis to identify 13 key topics about the mesopelagic zone from 2226 scientific abstracts and 4066 tweets. Two dominant ideas emerge: (1) the exploitation of fish resources and (2) the role of the mesopelagic zone as a carbon sink. We find that, in the absence of significant mesopelagic zone policy, science plays a de facto governing role by shaping how the ecosystem is known to the public and policymakers alike. These results indicate the need and potential for new knowledge and governance arrangements that can broaden the range of meanings and knowledge needed to anticipate future uses of deeply uncertain ecosystems and resources.

show abstract

oolong: An R package for validating automated content analysis tools

Cited by 15 publications

References 24 publications

Close to Beijing: Geographic Biases in People’s Daily

Close to Beijing: Geographic Biases in People’s Daily

The search for solid ground in text as data: A systematic review and practical recommendations for validation

Science governs the future of the mesopelagic zone

Contact Info

Product

Resources

About