Challenge or Empower: Revisiting Argumentation Quality in a News Editorial Corpus

Baff, Roxanne El; Wachsmuth, Henning; Khatib, Khalid Al; Stein, Benno

doi:10.18653/v1/k18-1044

Cited by 13 publications

(15 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other approaches take into account properties of the source, i.e., the author (Durmus and Cardie, 2019) or the audience (El Baff et al, 2018;Durmus and Cardie, 2018). In contrast, we assume that a system may not have much knowledge about the authors or audience and thus our models operate solely on the text.…”

Section: Related Workmentioning

confidence: 99%

“…Providing relevant and sufficient justifications for a claim and using clear language to express reasoning are important features of everyday writing. These are components of Argument Quality (AQ), which has been studied in many domains, such as student essays (Wachsmuth et al, 2016), news editorials (El Baff et al, 2018), and debate forums (Lukin et al, 2017).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Rhetoric, Logic, and Dialectic: Advancing Theory-based Argument Quality Assessment in Natural Language Processing

Lauscher¹,

Ng²,

Napoles³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Though preceding work in computational argument quality (AQ) mostly focuses on assessing overall AQ, researchers agree that writers would benefit from feedback targeting individual dimensions of argumentation theory. However, a large-scale theory-based corpus and corresponding computational models are missing. We fill this gap by conducting an extensive analysis covering three diverse domains of online argumentative writing and presenting GAQCorpus: the first largescale English multi-domain (community Q&A forums, debate forums, review forums) corpus annotated with theory-based AQ scores. We then propose the first computational approaches to theory-based assessment, which can serve as strong baselines for future work. We demonstrate the feasibility of large-scale AQ annotation, show that exploiting relations between dimensions yields performance improvements, and explore the synergies between theory-based prediction and practical AQ assessment.RQ1: Can we develop a large-scale theory-based AQ corpus? We conduct an extensive annotation study with trained linguists and crowd workers on 5, 295 arguments from three domains to create the Grammarly Argument Quality Corpus (GAQCorpus), the first large-scale multi-domain English corpus annotated with theory-based AQ scores.RQ2: Are we able to develop computational models that can do theory-based AQ assessment in varying domains? Based on GAQCorpus, we are the first to propose computational approaches to theory-based AQ assessment and show that it is possible to develop models for this task. Our models can serve as strong baselines for future research and enable the field to investigate follow-up research questions.RQ3: Can the interrelations between the different AQ dimensions be exploited in a computational setup? Inspired by the hierarchical structure of the taxonomy, we explore whether the relationships between dimensions can be computationally exploited. In addition to simple single-task learning approaches, we study the effect of jointly predicting AQ dimensions in two variants (flat vs. hierarchical) and find that combining the training signals of all four aspects benefits theory-based AQ assessment.RQ4: Does the corpus support training a single unified model for multi-domain evaluation? When enough data from a single domain is available, training on in-domain data is typically preferred over multi-domain. However, larger amounts of data are especially useful for complex model architectures currently prominent in NLP (e.g., BERT (Devlin et al., 2019), GPT2 (Radford et al., 2019). We study these two mutually opposing effects on GAQCorpus and show that our corpus supports training a single unified model across all three domains, with improved performances in individual domains.RQ5: Can we empirically substantiate the idea that theory-based and practical AQ assessment can learn from each other? Wachsmuth et al. (2017a) suggest that both the practical and the theory-based views can learn from each other, but so far, this has been only tested manually. Employing our mode...

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Rhetoric, Logic, and Dialectic: Advancing Theory-based Argument Quality Assessment in Natural Language Processing

Lauscher¹,

Ng²,

Napoles³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Ultimately, such judgments remain subjective (Lukin et al, 2017). To alleviate this, El Baff et al (2018) encode the reader's ideology and personality, but such information is often not given in practice.…”

Section: Related Workmentioning

confidence: 99%

Intrinsic Quality Assessment of Arguments

Wachsmuth¹,

Werner²

2020

Proceedings of the 28th International Conference on Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Several quality dimensions of natural language arguments have been investigated. Some are likely to be reflected in linguistic features (e.g., an argument's arrangement), whereas others depend on context (e.g., relevance) or topic knowledge (e.g., acceptability). In this paper, we study the intrinsic computational assessment of 15 dimensions, i.e., only learning from an argument's text.In systematic experiments with eight feature types on an existing corpus, we observe moderate but significant learning success for most dimensions. Rhetorical quality seems hardest to assess, and subjectivity features turn out strong, although length bias in the corpus impedes full validity. We also find that human assessors differ more clearly to each other than to our approach.

show abstract

“…Understanding the properties of persuasive arguments and identifying them in text have been widely studied in the NLP community (Tan et al, 2016;Habernal and Gurevych, 2016b;Wei et al, 2016;Persing and Ng, 2017;Stab and Gurevych, 2017;Hidey et al, 2017;Lukin et al, 2017;El Baff et al, 2018;Durmus and Cardie, 2018;Gleize et al, 2019;Yang et al, 2019;Xiao and Khazaei, 2019).…”

Section: Related Workmentioning

confidence: 99%

Predicting Stance Change Using Modular Architectures

Porco

Goldwasser

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

The ability to change a person's mind on a given issue depends both on the arguments they are presented with and on their underlying perspectives and biases on that issue. Predicting stance changes requires characterizing both aspects and the interaction between them, especially in realistic settings in which stance changes are very rare.In this paper, we suggest a modular learning approach, which decomposes the task into multiple modules, focusing on different aspects of the interaction between users, their beliefs, and the arguments they are exposed to. Our experiments show that our modular approach archives significantly better results compared to the end-to-end approach using BERT over the same inputs.

show abstract

Challenge or Empower: Revisiting Argumentation Quality in a News Editorial Corpus

Cited by 13 publications

References 22 publications

Rhetoric, Logic, and Dialectic: Advancing Theory-based Argument Quality Assessment in Natural Language Processing

Rhetoric, Logic, and Dialectic: Advancing Theory-based Argument Quality Assessment in Natural Language Processing

Intrinsic Quality Assessment of Arguments

Predicting Stance Change Using Modular Architectures

Contact Info

Product

Resources

About