Capturing a News Frame – Comparing Machine-Learning Approaches to Frame Analysis with Different Degrees of Supervision

Developing a synthetic news corpus to validate generic frame detection methods

“…But the less-than-superb performance cannot be explained by intercoder variations alone because the multiverse analysis has considered both the single-and double-coder scenarios. This low performance is not what we expected and also thought-provoking: Given the fact that the "gold standard" can only detect 50% of frames correctly and the state-of-the-art supervised classifiers classify frames at around 60% accuracy, should we trust the supervised frame classifiers trained on the so-called "gold standard" data (e.g., Kroon et al, 2022;Eisele et al, 2023;Kwak et al, 2020;Liu et al, 2019)? We do not have an empirical answer to this question, because we did not study supervised methods in this paper.…”

“…A relatively new approach is to apply unsupervised machine learning techniques to find frames through induction. As of writing, we are able to find several methods papers suggesting that these unsupervised machine learning techniques can be used to find frames (Burscher et al, 2016;DiMaggio et al, 2013;Eisele et al, 2023;Greussing & Boomgaarden, 2017;Nicholls & Culpepper, 2021;Walter & Ophir, 2019). Not surprisingly, all, except DiMaggio et al (2013), have given the "obligatory nod" -in Reese ( 2007)'s sense -to Entman (1993).…”

“…But both methods are claimed to be able to measure theoretical constructs through the provision of theorydriven dictionaries. Eisele et al (2023) applied KeyATM to detect frames but found it unfit for the task. Therefore, semi-supervised methods were included in this study for exploratory purposes only.…”

Framing Social Movements on Social Media

“…They find that gun rights organizations emphasized the inefficacy of gun restrictions and highlighted law enforcement failure as the primary problem, while gun control groups identified easy access to guns as the main problem and emphasized mobilization. However, scholars argue that unsupervised approaches like topic modeling do not capture theoretically-grounded frames, in contrast to supervised classification with existing taxonomies (Nicholls and Culpepper, 2021;Eisele et al, 2023). Supervised frame detection involves first manually coding texts based on a preexisting frame taxonomy, and then using this labeled data to train machine learning classification models.…”

“…Supervised frame detection involves first manually coding texts based on a preexisting frame taxonomy, and then using this labeled data to train machine learning classification models. Prior work has implemented a wide variety of supervised classification models to detect frames, including support vector machines, random forest classifiers, neural networks, and fine-tuning pretrained language models such as RoBERTa (Khanehzar et al, 2019(Khanehzar et al, , 2021Ali and Hassan, 2022;Eisele et al, 2023). A recent but quickly-growing body of literature is also exploring the potential of prompting large language models (e.g., ChatGPT) for both supervised and unsupervised frame analysis (Guo et al, 2022;Mou et al, 2022;Roy et al, 2022;Ziems et al, 2023).…”