Argumentation Quality Assessment: Theory vs. Practice

Wachsmuth, Henning; Naderi, Nona; Habernal, Ivan; Hou, Yufang; Hirst, Graeme; Gurevych, Iryna; Stein, Benno

doi:10.18653/v1/p17-2039

Cited by 44 publications

(48 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Krippendorff's α has been shown to be often low in such cases (Di Eugenio and Glass, 2004). Indeed, the values are in line with those obtained for similar tasks in other studies (Wachsmuth et al, 2017a). Table 5 shows the distribution of news editorials over their combined effect on the two opposing belief groups, ignoring which group is liberal and which is conservative.…”

Section: Inter-annotator Agreementsupporting

confidence: 88%

Challenge or Empower: Revisiting Argumentation Quality in a News Editorial Corpus

Baff¹,

Wachsmuth

Khatib

et al. 2018

Proceedings of the 22nd Conference on Computational Natural Language Learning

Self Cite

View full text Add to dashboard Cite

News editorials are said to shape public opinion, which makes them a powerful tool and an important source of political argumentation. However, rarely do editorials change anyone's stance on an issue completely, nor do they tend to argue explicitly (but rather follow a subtle rhetorical strategy). So, what does argumentation quality mean for editorials then? We develop the notion that an effective editorial challenges readers with opposing stance, and at the same time empowers the arguing skills of readers that share the editorial's stance-or even challenges both sides. To study argumentation quality based on this notion, we introduce a new corpus with 1000 editorials from the New York Times, annotated for their perceived effect along with the annotators' political orientations. Analyzing the corpus, we find that annotators with different orientation disagree on the effect significantly. While only 1% of all editorials changed anyone's stance, more than 5% meet our notion. We conclude that our corpus serves as a suitable resource for studying the argumentation quality of news editorials.

show abstract

Section: Inter-annotator Agreementsupporting

confidence: 88%

Challenge or Empower: Revisiting Argumentation Quality in a News Editorial Corpus

Baff¹,

Wachsmuth

Khatib

et al. 2018

Proceedings of the 22nd Conference on Computational Natural Language Learning

Self Cite

View full text Add to dashboard Cite

show abstract

“…To achieve reliable performance in these complex tasks, modern systems rely on the analysis of the underlying linguistic structures that characterize successful argumentation, rhetoric, and persuasion. Consequently, to distill the building blocks of argumentation from a text corpus, it is not sufficient to employ off-the-shelf Natural Language Processing techniques [65], which are typically developed for coarser analytical tasks (see [42] for an overview), such as with the high-level tasks of topic modeling [19] or sentiment analysis [5]. * e-mail: firstname.lastname@uni-konstanz.de Hence, to master the challenge of identifying argumentative substructures in large text corpora, computational linguistic researchers are actively developing techniques for the extraction of argumentative fragments of text and the relations between them [41].…”

Section: Introductionmentioning

confidence: 99%

VIANA: Visual Interactive Annotation of Argumentation

Sperrle

Sevastjanova

Kehlbeck

et al. 2019

2019 IEEE Conference on Visual Analytics Science and Technology (VAST)

View full text Add to dashboard Cite

Figure 1: VIANA is a system for interactive annotation of argumentation. It offers five different analysis layers, each represented by a different view and tailored to a specific task. The layers are connected with semantic transitions. With increasing progress of the analysis, users can abstract away from the text representation and seamlessly transition towards distant reading interfaces. ABSTRACTArgumentation Mining addresses the challenging tasks of identifying boundaries of argumentative text fragments and extracting their relationships. Fully automated solutions do not reach satisfactory accuracy due to their insufficient incorporation of semantics and domain knowledge. Therefore, experts currently rely on time-consuming manual annotations. In this paper, we present a visual analytics system that augments the manual annotation process by automatically suggesting which text fragments to annotate next. The accuracy of those suggestions is improved over time by incorporating linguistic knowledge and language modeling to learn a measure of argument similarity from user interactions. Based on a long-term collaboration with domain experts, we identify and model five high-level analysis tasks. We enable close reading and note-taking, annotation of arguments, argument reconstruction, extraction of argument relations, and exploration of argument graphs. To avoid context switches, we transition between all views through seamless morphing, visually anchoring all text-and graph-based layers. We evaluate our system with a two-stage expert user study based on a corpus of presidential debates. The results show that experts prefer our system over existing solutions due to the speedup provided by the automatic suggestions and the tight integration between text and graph views.

show abstract

“…For details, see the source code we provide. We did not check any quality criteria of arguments, as this was not our focus; see, e.g.,(Wachsmuth et al, 2017) for argumentation quality.…”

mentioning

confidence: 99%

The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants

Habernal¹,

Wachsmuth²,

Gurevych³

et al. 2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

Self Cite

123

View full text Add to dashboard Cite

Reasoning is a crucial part of natural language argumentation. To comprehend an argument, one must analyze its warrant, which explains why its claim follows from its premises. As arguments are highly contextualized, warrants are usually presupposed and left implicit. Thus, the comprehension does not only require language understanding and logic skills, but also depends on common sense. In this paper we develop a methodology for reconstructing warrants systematically. We operationalize it in a scalable crowdsourcing process, resulting in a freely licensed dataset with warrants for 2k authentic arguments from news comments. 1 On this basis, we present a new challenging task, the argument reasoning comprehension task. Given an argument with a claim and a premise, the goal is to choose the correct implicit warrant from two options. Both warrants are plausible and lexically close, but lead to contradicting claims. A solution to this task will define a substantial step towards automatic warrant reconstruction. However, experiments with several neural attention and language models reveal that current approaches do not suffice.

show abstract

Argumentation Quality Assessment: Theory vs. Practice

Cited by 44 publications

References 12 publications

Challenge or Empower: Revisiting Argumentation Quality in a News Editorial Corpus

Challenge or Empower: Revisiting Argumentation Quality in a News Editorial Corpus

VIANA: Visual Interactive Annotation of Argumentation

The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants

Contact Info

Product

Resources

About