Toward the Automated Scoring of Written Arguments: Developing an Innovative Approach for Annotation

Abstract

This project focuses on laying the foundations for automated analysis of argumentation schemes, supporting identification and classification of the arguments being made in a text, for the purpose of scoring the quality of written analyses of arguments. We developed annotation protocols for 20 argument prompts from a college‐level test under the framework of the theory of argumentation schemes, which defines reasoning patterns in argumentation. The annotation protocols listed critical questions associated with each argumentation scheme, which makes the argument structure in a text explicit and classifiable. Furthermore, we annotated 200 student essays across four selected argument prompts to test if the annotation protocols can be applied reliably by human annotators. Preliminary results indicate that this method of analyzing argument structure is reliable and promising.

Detecting Good Arguments in a Non-Topic-Specific Way: An Oxymoron?

“…We used Song et al (2014) annotation protocol, adapting as needed to cover additional argumentation schemes. Song et al (2017) provides a detailed exposition of the argumentation-schemebased analysis of a number of prompts and of the annotation process. For the current study, we used a simplified version of the annotation where sentences are labeled as non-generic (namely, containing a good critique according to some argumentation scheme), or generic (all the rest of the sentences in the essay).…”

Detecting Attackable Sentences in Arguments

“…Unlike the general argument quality described in the previous paragraph, CQs serve as an evaluation tool that specify local attackable points in an argument. They have been adopted for grading essays (Song et al, 2017) and teaching argumentation skills (Nussbaum et al, 2018). Some of the sentence characteristics in our work are informed by argumentation schemes and CQs.…”