TellMeWhy: A Dataset for Answering Why-Questions in Narratives

Abstract

Answering questions about why characters perform certain actions is central to understanding and reasoning about narratives. Despite recent progress in QA, it is not clear if existing models have the ability to answer "why" questions that may require commonsense knowledge external to the input narrative. In this work, we introduce TellMeWhy, a new crowd-sourced dataset that consists of more than 30k questions and free-form answers concerning why characters in short narratives perform the actions described. For a third of this dataset, the answers are not present within the narrative. Given the limitations of automated evaluation for this task, we also present a systematized human evaluation interface for this dataset. Our evaluation of state-of-the-art models show that they are far below human performance on answering such questions. They are especially worse on questions whose answers are external to the narrative, thus providing a challenge for future QA and narrative understanding research.

A Survey of Machine Narrative Reading Comprehension Assessments

“…Event Relation The majority of new event-centric tasks in the NLP field focus on the prediction of a relation between two events, with both events provided and described in the narrative texts. The relationships considered include the causal and conditional relationships [Mirza and Tonelli, 2014;Lal et al, 2021]; temporal relationships, which define the relationship between two events in terms of time, or between one event and a specific time point, such as tomorrow, as covered by the recent TORQUE [Ning et al, 2020] dataset.…”

“…Narrative Source Targeted Story Elements Event Character Setting Functional Structure MCTest [Richardson et al, 2013] multi-choice children stories CBT [Hill et al, 2015] cloze test children stories LAMBADA [Paperno et al, 2016] language model literature literary events [Sims et al, 2019] event trigger detection literature HiEve [Glavaš et al, 2014] event relation detection news stories TORQUE [Ning et al, 2020] event relation detection news stories TellMeWhy [Lal et al, 2021] multi-choice short fictions MCScript [Ostermann et al, 2018] multi-choice daily narratives ROCStories [Mostafazadeh et al, 2016] multi-choice short stories…”

A Survey of Machine Narrative Reading Comprehension Assessments

“…Event Relation The majority of new event-centric tasks in the NLP field focus on the prediction of a relation between two events, with both events provided and described in the narrative texts. The relationships considered include the causal and conditional relationships [Lal et al, 2021]; temporal relationships, which define the relationship between two events in terms of time, or between one event and a specific time point, such as tomorrow, as covered by the recent TORQUE [Ning et al, 2020] dataset.…”

“…Narrative Source Targeted Story Elements Event Character Setting Functional Structure MCTest [Richardson et al, 2013] multi-choice children stories ✓ ✓ CBT [Hill et al, 2015] cloze test children stories ✓ LAMBADA [Paperno et al, 2016] language model literature ✓ literary events [Sims et al, 2019] event trigger detection literature ✓ HiEve [Glavaš et al, 2014] event relation detection news stories ✓ TORQUE [Ning et al, 2020] event relation detection news stories ✓ TellMeWhy [Lal et al, 2021] multi-choice short fictions ✓ MCScript [Ostermann et al, 2018] multi-choice daily narratives ✓ ROCStories [Mostafazadeh et al, 2016] multi-choice short stories ✓ NarrativeQA [Kočiskỳ and others, 2018] free-answering QA movie scripts, literature ✓ ✓ ✓ FriendsQA [Yang and Choi, 2019] extractive QA TV show scripts ✓ ✓ ✓ NovelChapters [Ladhak et al, 2020] / BookSum [Kryściński et al, 2021] summarization literature ✓ SumScreen [Chen et al, 2021] summarization correct ending for a story from the given endings.…”