Annotating Claims in the Vaccination Debate

Abstract

In this paper we present annotation experiments with three different annotation schemes for the identification of argument components in texts related to the vaccination debate. Identifying claims about vaccinations made by participants in the debate is of great societal interest, as the decision to vaccinate or not has impact in public health and safety. Since most corpora that have been annotated with argumentation information contain texts that belong to a specific genre and have a well defined argumentation structure, we needed to adjust the annotation schemes to our corpus, which contains heterogeneous texts from the Web. We started with a complex annotation scheme that had to be simplified due to low IAA. In our final experiment, which focused on annotating claims, annotators reached 57.3% IAA.

Annotation for computational argumentation analysis: Issues and perspectives

“…For example, the scheme from the essay corpus is used by Miller et al (2019) on product reviews, but they reach lower IAA than for the essays. Another example of this is presented by Torsi and Morante (2018), who try different annotation schemes for argumentation components in their corpus consisting of documents from the vaccination debate, collected from blog posts, editorials and news articles. They get fair to moderate agreement for both a claim-premise scheme as in Stab and Gurevych (2017) and a Toulmin-inspired scheme (Habernal & Gurevych, 2017) (described in Section 3.4).…”

Beyond facts – a survey and conceptualisation of claims in online discourse analysis

“…of argumentation mining, depends on the domain or task and is somewhat arbitrary. Also, as Torsi et al [186] show, related annotation categories are often not well defined. With the use case of scientific articles in mind, Mayer et al [111] define a claim as a concluding statement made by the author about the outcome of the study.…”

“…However, both the used terminology and the underlying conceptual models are still strongly diverging, within and across the academic literature and the involved applications [35,186]. For example, "Animals should have lawful rights" is considered a claim in Chen et al [29] and according to many definitions from the argumentation mining community which define claims as the conclusive parts of an argument.…”