Extracting Latent Moral Information from Text Narratives: Relevance, Challenges, and Solutions

Weber, René; Mangus, J. Michael; Huskey, Richard; Hopp, Frederic R.; Amir, Ori; Swanson, Reid; Gordon, Andrew S.; Khooshabeh, Peter; Hahn, Lindsay; Tamborini, Ron

doi:10.1080/19312458.2018.1447656

Cited by 46 publications

(43 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For crowd coding, in contrast, there is no reason to assume why the outcome will be structurally different if another researcher launches the same job, guaranteeing replicability and validity of the reliability measure. This is in line with the findings and recommendations of Weber et al (2018), who advocate crowd coding after having difficulty replicating their own previous codings of moral claims.…”

Section: Discussionsupporting

confidence: 87%

“…Traditionally, sentiment is measured using manual annotators and a codebook (for examples, see Aday, 2010;Cho, 2013;Dunaway et al, 2015;Elenbaas & De Vreese, 2008;Kleinnijenhuis et al, 2007Kleinnijenhuis et al, , 2019Martins et al, 2013;McCombes et al, 2000;Meijer & Kleinnijenhuis, 2006;Muddiman & Stroud, 2017;Nagel et al, 2012;Natarajan & Xiaoming, 2003;Rodgers & Thorson, 2003;Shah et al, 2007). Manual coding is expensive, however, and even when using extensive training programs high levels of reliability are not always achieved (Weber et al, 2018).…”

Section: Existing Methods Of Sentiment Analysismentioning

confidence: 99%

See 1 more Smart Citation

The Validity of Sentiment Analysis: Comparing Manual Annotation, Crowd-Coding, Dictionary Approaches, and Machine Learning Algorithms

Atteveldt

Velden

Boukes

2021

Communication Methods and Measures

218

View full text Add to dashboard Cite

Sentiment is central to many studies of communication science, from negativity and polarization in political communication to analyzing product reviews and social media comments in other sub-fields. This study provides an exhaustive comparison of sentiment analysis methods, using a validation set of Dutch economic headlines to compare the performance of manual annotation, crowd coding, numerous dictionaries and machine learning using both traditional and deep learning algorithms. The three main conclusions of this article are that: (1) The best performance is still attained with trained human or crowd coding; (2) None of the used dictionaries come close to acceptable levels of validity; and (3) machine learning, especially deep learning, substantially outperforms dictionary-based methods but falls short of human performance. From these findings, we stress the importance of always validating automatic text analysis methods before usage. Moreover, we provide a recommended step-bystep approach for (automated) text analysis projects to ensure both efficiency and validity.

show abstract

Section: Discussionsupporting

confidence: 87%

Section: Existing Methods Of Sentiment Analysismentioning

confidence: 99%

The Validity of Sentiment Analysis: Comparing Manual Annotation, Crowd-Coding, Dictionary Approaches, and Machine Learning Algorithms

Atteveldt

Velden

Boukes

2021

Communication Methods and Measures

218

View full text Add to dashboard Cite

show abstract

“…Our work adds to the growing literature concerned with distilling reactions to situations (Vu et al, 2014;Ding and Riloff, 2016) as well as so-cial and moral dynamics in language (Van Hee et al, 2015). Commonly used for coarse-grained analyses of morality in text (Fulgoni et al, 2016;Volkova et al, 2017;Weber et al, 2018), Graham et al (2009) introduce the Moral Foundations lexicon, a dictionary of morality-evoking words (later extended by Rezapour et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

Social Chemistry 101: Learning to Reason about Social and Moral Norms

Forbes¹,

Hwang²,

Shwartz³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Social norms-the unspoken commonsense rules about acceptable social behavior-are crucial in understanding the underlying causes and intents of people's actions in narratives. For example, underlying an action such as "wanting to call cops on my neighbor" are social norms that inform our conduct, such as "It is expected that you report crimes." We present SOCIAL CHEMISTRY, a new conceptual formalism to study people's everyday social norms and moral judgments over a rich spectrum of real life situations described in natural language. We introduce SOCIAL-CHEM-101, a large-scale corpus that catalogs 292k rules-of-thumb such as "It is rude to run a blender at 5am" as the basic conceptual units. Each rule-of-thumb is further broken down with 12 different dimensions of people's judgments, including social judgments of good and bad, moral foundations, expected cultural pressure, and assumed legality, which together amount to over 4.5 million annotations of categorical labels and free-text descriptions. Comprehensive empirical results based on state-of-the-art neural models demonstrate that computational modeling of social norms is a promising research direction. Our model framework, NEURAL NORM TRANSFORMER, learns and generalizes SOCIAL-CHEM-101 to successfully reason about previously unseen situations, generating relevant (and potentially novel) attribute-aware social rules-of-thumb. 1 Note that the social identities of the participants of situations would further inform which social norms are most

show abstract

“…Lind et al (2017) show that the crowd equals traditional coders in identifying the presence and valence of target evaluations in text sentences. Weber et al (2018) demonstrate that ordinary citizens can draw on “moral intuitions” to code text. And Budak et al (2016) show that online workers can classify the partisan slant of newspaper articles.…”

Section: Online Coding Interface Coders and Coding Tasksmentioning

confidence: 99%

Online coders, open codebooks: New opportunities for content analysis of political communication

Winter

Hughes

Sanders

2019

PSRM

View full text Add to dashboard Cite

Analyzing audiovisual communication is challenging because its content is highly symbolic and less rule-governed than verbal material. But audiovisual messages are important to understand: they amplify, enrich, and complicate the meaning of textual information. We describe a fully-reproducible approach to analyzing video content using minimally—but systematically—trained online workers. By aggregating the work of multiple coders, we achieve reliability, validity, and costs that equal those of traditional, intensively trained research assistants, with much greater speed, transparency, and replicability. We argue that measurement strategies relying on the “wisdom of the crowd” provide unique advantages for researchers analyzing complex and intricate audiovisual political content.

show abstract

Extracting Latent Moral Information from Text Narratives: Relevance, Challenges, and Solutions

Cited by 46 publications

References 39 publications

The Validity of Sentiment Analysis: Comparing Manual Annotation, Crowd-Coding, Dictionary Approaches, and Machine Learning Algorithms

The Validity of Sentiment Analysis: Comparing Manual Annotation, Crowd-Coding, Dictionary Approaches, and Machine Learning Algorithms

Social Chemistry 101: Learning to Reason about Social and Moral Norms

Online coders, open codebooks: New opportunities for content analysis of political communication

Contact Info

Product

Resources

About