Detecting Narrative Elements in Informational Text

Abstract

Automatic extraction of narrative elements from text, combining narrative theories with computational models, has been receiving increasing attention over the last few years. Previous works have utilized the oral narrative theory by Labov and Waletzky to identify various narrative elements in personal stories texts. Instead, we direct our focus to informational texts, specifically news stories.We introduce NEAT (Narrative Elements An-noTation) -a novel NLP task for detecting narrative elements in raw text. For this purpose, we designed a new multi-label narrative annotation scheme, better suited for informational text (e.g. news media), by adapting elements from the narrative theory of Labov and Waletzky (Complication and Resolution) and adding a new narrative element of our own (Success). We then used this scheme to annotate a new dataset of 2,209 sentences, compiled from 46 news articles from various category domains 1 . We trained a number of supervised models in several different setups over the annotated dataset to identify the different narrative elements, achieving an average F 1 score of up to 0.77. The results demonstrate the holistic nature of our annotation scheme as well as its robustness to domain category.

A Decomposition-Based Approach for Evaluating Inter-Annotator Disagreement in Narrative Analysis

“…We analyzed the dataset created by Levi et al (2022) for narrative analysis, consisting of 2,209 sentences (taken from news articles). The authors employed a total of three annotators (henceforth denoted as A 1 , A 2 and A 3 ).…”

“…All the sentences were annotated by A 1 , while each sentence was additionally annotated by either A 2 or A 3 (statistics are given in Table 1). Further details on the annotation scheme, guidelines and dataset can be found in Levi et al (2022)…”

“…In the field of NLP, several works have constructed and annotated datasets for capturing various narrative aspects in texts (Baiamonte et al, 2016;Delmonte and Marchesini, 2017;Papalampidi et al, 2019;Volpetti et al, 2020). A more specific line of work is inspired by the works of Labov and Waletzky (1967) and Labov (2013) on characterizing narratives by identifying different narrative element types (Swanson et al, 2014;Ouyang and McKeown, 2014;Li et al, 2017;Saldias and Roy, 2020;Levi et al, 2022).…”

“…In this work, we evaluate and discuss sources of interannotator disagreement in narrative analysis in light of this very question. For this purpose, we examine an existing dataset of sentences annotated with three different narrative plot elements: Complication, Resolution and Success (Levi et al, 2022). Together, these three elements capture a typical tension-release plot type, varied by the different possible elements combinations.…”