Incorporating Coherence of Topics as a Criterion in Automatic Response-to-Text Assessment of the Organization of Writing

Abstract

This paper presents an investigation of score prediction for the Organization dimension of an assessment of analytical writing in response to text. With the long-term goal of producing feedback for students and teachers, we designed a task-dependent model that aligns with the scoring rubric and makes use of the source material. Our experimental results show that our rubric-based model performs as well as baselines on datasets from grades 6-8. On shorter and noisier essays from grades 5-6, the rubric-based model performs better than the baselines. Further, we show that the baseline model (lexical chaining) can be improved if we extend it with information from the source text for shorter and noisier data.

Computational Argumentation Quality Assessment in Natural Language

“…In their experiments, sequences of paragraph discourse functions (e.g., introduction or rebuttal) turn out to be most effective. Organization is also analyzed by Rahimi et al (2015) on the same dataset used for the evidence Not only linguistic quality, but also effectiveness is assessed in recent work: Persing and Ng (2015) score the argument strength of essays, which they define rhetorically in terms of how many readers would be persuaded. Although potentially subjective, their manual 7-point score annotations of 1000 ICLE essays differ by at most 1 in 67% of the studied cases.…”

“…Also, we omit approaches that classify argumentation schemes (Feng and Hirst, 2011), evidence types (Rinott et al, 2015), ethosrelated statements (Duthie et al, 2016), and myside bias ; their output may help assess quality assessment, but they do not actually assess it. The same holds for argument mining, Level of support Braunstain et al (2016) Evidence Rahimi et al (2014) Sufficiency Stab and Gurevych (2017) Thesis clarity Persing and Ng (2013) Prompt adherence Persing and Ng (2014) Global coherence Feng et al (2014) Evaluability Park et al (2015) Acceptability Cabrio and Villata (2012) Organization Persing et al (2010), Rahimi et al (2015) Argument strength Persing et al (2015) Persuasiveness Tan et al (2016), Wei et al (2016) Winning side Zhang et al (2016) Convincingness Habernal et al (2016) Prominence Boltužic and Šnajder (2015) Relevance Wachsmuth et al (2017) Figure 1: The proposed taxonomy of argumentation quality as well as the mapping of existing assessment approaches to the covered quality dimensions. Arrows show main dependencies between the dimensions.…”

“…In their experiments, sequences of paragraph discourse functions (e.g., introduction or rebuttal) turn out to be most effective. Organization is also analyzed by Rahimi et al (2015) on the same dataset used for the evidence Park et al (2015) develop an argumentation model that prescribes what information users should give to achieve evaluability (e.g., testimony evidence or references to resources).…”

Analysing Rhetorical Structure as a Key Feature of Summary Coherence

“…Automatic assessment of text quality can include content, language accuracy, sophistication and style as well as sometimes overlapping features such as topic similarity, focus, coherence, cohesion, readability, or text organisation and development, e.g., (Pitler et al, 2010;Yannakoudakis and Briscoe, 2012;Guo et al, 2013;Rahimi et al, 2015;Gao et al, 2018). Coherence is a broad concept assessed by different automatic tools, e.g., (Higgins et al, 2004;Yannakoudakis and Briscoe, 2012;.…”

EDTC: A Corpus for Discourse-Level Topic Chain Parsing

“…Similarly, researches on text-tilling achieve topic segments through lexical cohesion modeling (Hearst, 1997;Purver et al, 2006). Instead of lexical cohesion measuring, Rahimi et al (2015) put their attention on evaluating the organization and cohesion of pieces of evidence and build topic chains on related text units. Besides, recent studies on argument mining explore to build links or clusters for topic-dependent arguments (Wachsmuth et al, 2018;Shnarch et al, 2018;Reimers et al, 2019).…”